Azure Data Engineer - Data Factory/Databricks (5-10 yrs)

We are seeking an experienced Azure Data Engineer with 5-10 years of expertise in Azure Data Factory (ADF), Databricks, and PySpark to join our team. This role requires a strong background in data engineering, cloud resources, and ETL processes, with a focus on delivering scalable and efficient data solutions. You will work on designing, developing, and maintaining data pipelines and transformations within the Azure ecosystem. Ideal candidates will be self-motivated, technically skilled, and capable of managing client deliverables independently.

Key Responsibilities :

- Develop and maintain data pipelines in Azure using ADF and Databricks to facilitate seamless data ingestion, transformation, and loading.

- Design and implement robust ETL processes for data extraction from various sources, transforming it as required, and loading it into target databases or data lakes.

- Utilize Microsoft Azure cloud resources to ensure the scalability, security, and reliability of data solutions.

- Leverage Azure tools for monitoring, managing, and optimizing data workflows and pipeline performance.

- Write efficient PySpark scripts for data transformations within Databricks, ensuring code quality and reusability.

- Implement best practices in data processing and distributed computing to handle large datasets effectively.

- Develop complex SQL queries for data extraction and transformation, ensuring data quality and accuracy.

- Work with relational databases and implement data modeling concepts to structure data for optimal querying and storage.

- Own and manage client deliverables, from understanding requirements to providing solutions, with minimal supervision.

- Communicate effectively with clients and stakeholders, offering technical insights and progress updates as required.

- Document data pipelines, ETL processes, and data engineering workflows to ensure maintainability and knowledge sharing within the team.

- Follow data engineering best practices, including code reviews, version control, and process automation where applicable.

Required Skills & Qualifications :

- 5-10 years of hands-on experience in Azure Data Engineering.

- Mandatory Skills: Strong experience with Databricks, ADF, and SQL.

- Proficiency in ETL tools and techniques, with a focus on Azure-based solutions.

- Solid working knowledge of Microsoft Azure cloud resources and cloud-based data integration.

- Excellent communication skills to effectively collaborate with both technical and non-technical stakeholders.

- Ability to work independently and take ownership of client deliverables.

- Demonstrated ability to troubleshoot and resolve complex data engineering issues.

- Strong analytical skills to optimize data processes and ensure data quality.