Senior/Lead Data Engineer - Synapse Analytics (6-14 yrs)

Key Responsibilities :

Technical Leadership :

- Provide technical leadership and guidance to the data engineering team.

- Design and implement robust and scalable data pipelines using Azure Data Factory, Azure Synapse Analytics, Databricks, and other relevant tools.

- Develop data ingestion, transformation, and loading processes to extract value from diverse data sources.

- Optimize data pipelines for performance and cost-efficiency.

- Ensure data quality, integrity, and security throughout the data lifecycle.

- Collaborate with data scientists, analysts, and business stakeholders to understand their data needs and translate them into technical solutions.

Team Management :

- Lead and mentor a team of data engineers.

- Assign tasks, monitor progress, and provide timely feedback.

- Foster a collaborative and innovative team culture.

- Recruit and onboard new team members.

Cloud Architecture :

- Design and implement cloud-native data architectures on Azure.

- Leverage Azure Data Lake Storage for efficient data storage and retrieval.

- Utilize Azure Synapse Analytics for unified data ingestion, analytics, and ETL.

- Explore the capabilities of Microsoft Fabric for a unified data and analytics platform.

Data Governance :

- Establish and enforce data governance policies and standards.

- Implement data lineage and impact analysis to ensure data traceability.o Monitor data quality and identify potential issues

- Monitor data quality and identify potential issues

Senior Role :

Responsibilities :

- Design, develop, and implement data pipelines using Azure Data Factory (ADF) or other orchestration tools.

- Write efficient SQL queries to extract, transform, and load (ETL) data from various sources into Azure Synapse Analytics.

- Utilize PySpark and Python for complex data processing tasks on large datasets within Azure Databricks.

- Collaborate with data analysts to understand data requirements and ensure data quality.

- Hands-on experience in designing and developing Datalakes and Warehouses

- Implement data governance practices to ensure data security and compliance.

- Monitor and maintain data pipelines for optimal performance and troubleshoot any issues.

- Develop and maintain unit tests for data pipeline code.

- Work collaboratively with other engineers and data professionals in an Agile development environment.