Lead Data Engineer - Microsoft Fabric (10-12 yrs)

Lead Data Engineer-MS Fabric.

Exp : 10+ years.

Type : Contract(remote).

Shift Time : 3PM to 12AM IST.

Description :

We are looking for a Lead Data Engineer with extensive experience in developing ETL processes using PySpark Notebooks and Microsoft Fabric and supporting existing legacy SQL Server environments.

The ideal candidate will possess a strong background in Spark-based development, demonstrate a high proficiency in SQL, and be comfortable working independently, collaboratively within a team, or leading other developers when required.

- Design, develop, and maintain ETL pipelines using PySpark Notebooks and Microsoft Fabric.

- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver efficient data solutions.

- Migrate and integrate data from legacy SQL Server environments into modern data platforms.

- Optimize data pipelines and workflows for scalability, efficiency, and reliability.

- Provide technical leadership and mentorship to junior developers and other team members.

- Troubleshoot and resolve complex data engineering issues related to performance, data quality, and system scalability.

- Develop, maintain, and enforce data engineering best practices, coding standards, and documentation.

- Conduct code reviews and provide constructive feedback to improve team productivity and code quality.

- Support data-driven decision-making processes by ensuring data integrity, availability, and consistency across different platforms.

Requirements :

- Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related field.

- Experience with Microsoft Fabric or similar cloud-based data integration platforms is a must.

- 10+ years of experience in data engineering, with a strong focus on ETL development using PySpark or other Spark-based tools.

- Proficiency in SQL with extensive experience in complex queries, performance tuning, and data modeling.

- Strong knowledge of data warehousing concepts, ETL frameworks, and big data processing.

- Familiarity with other data processing technologies (e.g, Hadoop, Hive, Kafka) is an advantage.

- Experience working with both structured and unstructured data sources.

- Excellent problem-solving skills and the ability to troubleshoot complex data engineering issues.

- Proven ability to work independently, as part of a team, and in leadership roles.

- Strong communication skills with the ability to translate complex technical concepts into business terms.

Mandate skills :

- Experience with Data lake, Data warehouse, Delta lake.

- Experience with Azure Data Services, including Azure Data Factory, Azure Synapse, or similar tools.

- Knowledge of scripting languages (e.g, Python, Scala) for data manipulation and automation.

- Familiarity with DevOps practices, CI/CD pipelines, and containerization (Docker, Kubernetes) is a plus.