Senior Data Engineer - Databricks Expert

We are seeking a Senior Data Engineer with in-depth knowledge of Databricks and Unity Catalog to serve as the subject matter expert for all things Databricks within our organization. This role requires deep expertise in Databricks, including CI/CD setup for data bricks, data lineage through Unity Catalog, and strong proficiency in ETL, SQL, and modern data engineering practices. You will be the go-to person for designing, implementing, and optimizing data solutions with Databricks.

Key Responsibilities:

Serve as the point of contact and subject matter expert for all Databricks-related activities, including architecture, development, and operational best practices.

Design, develop, and manage ETL/ELT pipelines in Databricks using Python (PySpark), integrating various data sources to support business operations.

Leverage Unity Catalog to ensure data lineage, security, and governance are properly managed across the Databricks environment.

Implement and maintain CI/CD pipelines for Databricks, ensuring smooth deployments, version control, and automation using Git and other DevOps tools.

Build scalable data architectures, including Data Lakes, Lakehouses, and Data Warehouses, ensuring efficient data management and accessibility.

Configure and optimize Databricks clusters, jobs, and workflows for both batch and streaming data processing to handle large-scale datasets.

Stay up-to-date with the latest Databricks features and advancements, continuously enhancing our data engineering practices.

Collaborate with cross-functional teams to implement data governance and ensure compliance with security and industry regulations.

Monitor and tune Databricks workloads to ensure high performance and scalability, adapting to business needs as required.

Provide guidance and mentorship to junior engineers, ensuring adherence to best practices and fostering a collaborative environment.