Azure Data Engineer - Python/PySpark (4-7 yrs)

Job Summary :

We are seeking a skilled and experienced Azure Data Engineer to join our dynamic team. The ideal candidate will have a strong background in Azure technologies and data engineering practices, with expertise in Azure Databricks, Azure Data Factory, Data Modeling, PySpark, and SQL. You will play a critical role in designing, developing, and maintaining scalable data solutions to meet business needs.

Key Responsibilities :

Data Solution Design & Development :

- Develop and optimize scalable data pipelines using Azure Databricks and Azure Data Factory.

- Design, implement, and maintain effective data models to support analytics and reporting needs.

Data Transformation & Integration :

- Utilize PySpark and SQL to transform and process large datasets.

- Build efficient data integration workflows to connect disparate data sources into a centralized data ecosystem.

Performance Optimization :

- Optimize data processing pipelines for performance and scalability.

- Ensure high availability and performance of data solutions in cloud environments.

Collaboration & Documentation :

- Work closely with stakeholders to understand data requirements and provide actionable insights.

- Document data solutions, workflows, and best practices.

Quality Assurance :

- Implement robust data quality and validation checks in all data workflows.

- Conduct regular reviews to ensure data integrity and accuracy.

Required Skills & Qualifications :

- Proficient in Azure Databricks and Azure Data Factory for building data solutions.

- Strong knowledge of data modeling concepts and practices.

- Hands-on experience with PySpark for data transformation and analysis.

- Advanced SQL skills, including query optimization and complex data manipulation.

- Experience working with large-scale datasets and distributed computing.

- Knowledge of CI/CD processes and version control systems is a plus.

- Excellent problem-solving skills and the ability to work in a collaborative team environment.