Senior Lead Engineer Data Engineering

Senior Lead Engineer Data Engineering

Exp:7+ years (Databricks certification preferred)

Qualifications

• 5+ years of experience in data engineering with significant expertise in Databricks and Apache Spark.
• Proficient in Unity Catalog for managing data lineage, security, and governance within the Databricks ecosystem.
• Experience of estimating and migrating legacy data warehouse workloads to Azure/Hybrid Cloud.
• Proficient in Unity Catalog for managing data lineage, security, and governance within the Databricks ecosystem.
• Experience building and optimizing ETL pipelines using tools like Azure Data Factory, Informatica, or similar.
• Strong understanding of CI/CD practices with experience in Git for version control and integration with Databricks.
• Expertise in SQL development and performance tuning for large-scale datasets.
• Knowledge of the Azure ecosystem, including data services like Azure Data Factory, Azure Data Lake and Azure Storage.
• Ability to work with both batch and streaming data processing pipelines.
• Experience with data modeling and dimensional design (e.g., star schema).
• Good understanding of data governance, compliance, and security best practices.
• Excellent communication and problem-solving skills, with the ability to manage multiple priorities.
• Ability to stay current on Databricks innovations and proactively introduce new features and capabilities to the team.