India Infoline - Lead Data Engineer - Azure/Databricks (10-15 yrs)

Key Responsibilities :

- Data Lakehouse Development : Lead the design and implementation of our data lakehouse architecture using Azure and Databricks.

- Data Modeling : Develop conceptual, logical, and physical data models utilizing star schema methodologies.

- Pipeline Optimization : Build and optimize data pipelines using PySpark and Databricks, ensuring high performance and scalability.

- Data Governance : Implement data governance frameworks and ensure compliance with data quality standards.

- Tool Integration : Leverage Azure Data Factory (ADF) and Change Data Capture (CDC) mechanisms for efficient data ingestion and processing.

- Unity Catalog Management : Utilize Unity Catalog for metadata management and data cataloging.

- Team Leadership : Mentor junior engineers and collaborate with cross-functional teams to drive data initiatives.

- Best Practices : Stay updated with the latest industry trends and integrate best practices into the data engineering processes.

Qualifications & Experience : Minimum of 7+ years in data engineering with at least 4 years focused on building data lakehouse.

- Azure & Databricks Expertise : 3-4 years of hands-on experience with Azure services and Databricks platform.

Technical Skills : Proficient in ADF, CDC, PySpark, and Unity Catalog.

- Data Modeling : Strong knowledge in creating conceptual, logical, and physical data models, specifically using star schema.

- Data Governance & Quality : Demonstrated experience in implementing data governance frameworks and ensuring data quality.

- Pipeline Development : Expertise in building optimized data pipelines on Databricks.

- Leadership Skills : Proven ability to lead projects and mentor team members.

Education : Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.

Preferred Qualifications : Certifications : Azure or Databricks certifications.

Soft Skills : Excellent communication, problem-solving abilities, and attention to detail.