Senior Associate _Azure DE + Databricks Developer _Data &Analytics

Job Description & Summary

A career within Data and Analytics services will provide you with the opportunity to help organisations uncover enterprise insights and drive business results using smarter data analytics. We focus on a collection of organisational technology capabilities, including business intelligence, data management, and data assurance that help our clients drive innovation, growth, and change within their organisations in order to keep up with the changing nature of customers and technology. We make impactful decisions by mixing mind and machine to leverage data, understand and navigate risk, and help our clients gain a competitive edge.

Creating business intelligence from data requires an understanding of the business, the data, and the technology used to store and analyse that data. Using our Rapid Business Intelligence Solutions, data visualisation and integrated reporting dashboards, we can deliver agile, highly interactive reporting and analytics that help our clients to more effectively run their business and understand what business questions can be answered and how to unlock the answers

Responsibilities

1. Data Lake and Lakehouse Implementation
o Design, implement, and manage Data Lake and Lakehouse architectures. (Must have)
o Develop and maintain scalable data pipelines and workflows. (Must have)
o Utilize Azure Data Lake Services (ADLS) for data storage and management. (Must have)
o Knowledge on Medalion Architecture, Delta Format. (Must have)
2. Data Processing and Transformation
o Use PySpark for data processing and transformations. (Must have)
o Implement Delta Live Tables for real-time data processing and analytics. (Good to have)
o Ensure data quality and consistency across all stages of the data lifecycle. (Must have)
3. Data Management and Governance
o Employ Unity Catalog for data governance and metadata management. (Good to have)
o Ensure robust data security and compliance with industry standards. (Must have)
4. Data Integration
o Extract, transform, and load (ETL) data from multiple sources (Must have) including SAP (Good to have), Dynamics 365 (Good to have), and other systems.
o Utilize Azure Data Factory (ADF) and Synapse Analytics for data integration and orchestration. (Must have)
o Performance Optimization of the Jobs. (Must have)
5. Data Storage and Access
o Implement and manage Azure Data Lake Storage (ADLS) for large-scale data storage. (Must have)
o Optimize data storage and retrieval processes for performance and cost-efficiency. (Must have)
6. Collaboration and Communication
o Work closely with data scientists, analysts, and other stakeholders to understand data requirements. (Must have)
o Provide technical guidance and mentorship to junior team members. (Good to have)
7. Continuous Improvement

o Stay updated with the latest industry trends and technologies in data engineering and cloud computing. (Good to have)
o Continuously improve data processes and infrastructure for efficiency and scalability. (Must have)

Required Skills and Qualifications
Technical Skills
o Proficient in PySpark and Python for data processing and analysis.
o Strong experience with Azure Data Lake Services (ADLS) and Data Lake architecture.
o Hands-on experience with Databricks for data engineering and analytics.
o Knowledge of Unity Catalog for data governance.
o Expertise in Delta Live Tables for real-time data processing.
o Familiarity with Azure Fabric for data integration and orchestration.
o Proficient in Azure Data Factory (ADF) and Synapse Analytics for ETL and data warehousing.
o Experience in pulling data from multiple sources like SAP, Dynamics 365, and others.

Soft Skills
o Excellent problem-solving and analytical skills.
o Strong communication and collaboration abilities.
o Ability to work independently and as part of a team.
o Attention to detail and commitment to data accuracy and quality.
Certifications required
Certification in Azure Data Engineering or relevant Azure certifications.
o DP203 (Must have)
Certification in Databricks.
o Databricks certified Data Engineer Associate (Must have)
o Databricks certified Data Engineer Professional (Good Have)

Mandatory skill sets Azure DE, Pyspark, Databricks

Preferred skill sets Azure DE, Pyspark, Databricks

Years of experience required 5-10 Years

Educational Qualification BE, B.Tech, MCA, M.Tech

Education (if blank, degree and/or field of study not specified)

Degrees/Field of Study required Master of Engineering, Bachelor of Engineering

Degrees/Field of Study preferred