Bay Area Technology Solutions - Senior/Lead Engineer - Data Engineering (6-9 yrs)

Qualifications :

- 5+ years of experience in data engineering with significant expertise in Databricks and Apache Spark.

- Proficient in Unity Catalog for managing data lineage, security, and governance within the Databricks ecosystem.

- Experience of estimating and migrating legacy data warehouse workloads to Azure/Hybrid Cloud.

- Proficient in Unity Catalog for managing data lineage, security, and governance within the Databricks ecosystem.

- Experience building and optimizing ETL pipelines using tools like Azure Data Factory, Informatica, or similar.

- Strong understanding of CI/CD practices with experience in Git for version control and integration with Databricks.

- Expertise in SQL development and performance tuning for large-scale datasets.

- Knowledge of the Azure ecosystem, including data services like Azure Data Factory, Azure Data Lake and Azure Storage.

- Ability to work with both batch and streaming data processing pipelines.

- Experience with data modeling and dimensional design (e.g., star schema).

- Good understanding of data governance, compliance, and security best practices.

- Excellent communication and problem-solving skills, with the ability to manage multiple priorities.

- Ability to stay current on Databricks innovations and proactively introduce new features and capabilities to the team.