Senior Data Engineer (Leadership role)

We are looking for a highly experienced Senior Data Engineer with 12+ years of experience to lead and drive the implementation of advanced data solutions on Databricks. This role requires a deep understanding of data architecture, complex data transformations, and the ability to mentor junior team members. As a senior member, you will collaborate closely with stakeholders, lead data strategy discussions, and ensure the successful deployment of robust data solutions.

Key Responsibilities:

• Lead the design and implementation of complex data engineering solutions.
• Mentor and guide junior data engineers.
• Collaborate with business stakeholders to understand data requirements and translate them into technical solutions.
• Optimize data pipelines for performance, cost-effectiveness, and scalability.
• Implement data governance and security best practices.
• Stay up-to-date with emerging technologies and industry trends.
• Contribute to the development of data engineering standards and methodologies.
• Implement and monitor CI/CD pipelines in Databricks, ensuring efficient deployment and operation of data solutions.
• Evaluate and optimize the performance of Databricks clusters, ensuring cost-effective use of cloud resources.

Required Skills and Experience:

• Bachelors or Masters degree in Computer Science, Data Science, or a related field.
• 12+ years of data engineering experience, with a minimum of 5 years on Databricks and Spark.
• Strong proficiency in Python or Scala programming languages.
• Advanced knowledge of Databricks and Apache Spark.
• Expertise in data warehousing and data modeling concepts.
• Experience with cloud platforms (AWS, Azure, GCP).
• Strong understanding of data governance and security best practices.
• Excellent leadership and communication skills.
• Proven leadership and mentorship skills, with experience managing junior engineers.

Bonus Skills:

• Knowledge of ML model integration and deployment within Databricks.
• Experience with data orchestration tools, such as Apache Airflow or Azure Data Factory.
• Familiarity with infrastructure as code (IaC) tools like Terraform or ARM templates.