Azure Databricks Architect - Spark/DataLake (5-8 yrs)

Responsibilities :

- Design, develop, and deploy data solutions on Azure Databricks, leveraging its capabilities for data engineering, machine learning, and business intelligence.

- Architect and implement data pipelines using Spark, Delta Lake, and other relevant technologies within the Databricks ecosystem.

- Optimize Databricks clusters and jobs for performance, scalability, and cost-effectiveness.

- Integrate Databricks with other Azure services, such as Azure Data Lake Storage, Azure Synapse Analytics, and Azure Machine Learning.

- Collaborate with data scientists, analysts, and business stakeholders to understand requirements and translate them into technical solutions.

- Lead and mentor junior Databricks engineers.

- Stay up-to-date with the latest Databricks features and best practices.

Qualifications :

- 5-8 years of experience in data engineering and architecture.

- Strong understanding of Azure Databricks and its components (Spark, Delta Lake, etc.)

- Proficiency in programming languages like Python, Scala, or Java.

- Experience with data warehousing and ETL concepts.

- Knowledge of cloud computing concepts and Azure services.

- Excellent communication and collaboration skills.

- Bachelor's degree in Computer Science or a related field.

Additional Skills (Preferred) :

- Experience with machine learning frameworks (e.g, TensorFlow, PyTorch).

- Knowledge of DevOps practices and tools.

- Experience with data governance and security.

- Azure certifications (e.g, Azure Data Engineer Associate, Azure Solutions Architect Expert)