Databricks Data Engineer

We are seeking a skilled Data bricks Data Engineer to join our dynamic team. The ideal candidate will have strong expertise in PySpark, Spark SQL, and a comprehensive understanding of the Databricks data engineering ecosystem. You will play a critical role in designing and implementing data pipelines, optimizing data workflows, and ensuring data integrity.

Key Responsibilities:

Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.

Utilize PySpark and Spark SQL to process large datasets and perform data transformations.

Collaborate with data scientists and analysts to understand data requirements and deliver efficient solutions.

Optimize existing data workflows for performance and reliability.

Monitor data pipeline performance and troubleshoot issues as they arise.

Implement best practices for data governance and security.

Participate in code reviews and contribute to team knowledge sharing.

Key Skills and Qualifications:

Proven experience as a Data Engineer with a focus on Databricks.

Strong proficiency in PySpark and Spark SQL.

Deep understanding of the Data bricks ecosystem, including Delta Lake, ML flow, and Databricks SQL.

Experience with data modeling, ETL processes, and data warehousing concepts.

Familiarity with cloud platforms such as AWS, Azure, or Google Cloud.

Knowledge of data visualization tools and frameworks is a plus.

Strong problem-solving skills and the ability to work independently.

Excellent communication skills and ability to collaborate with cross-functional teams.