Azure Databricks Engineer - ETL/Python (6-11 yrs)

We have Immediate Openings on Azure Databricks Engineer

Job Description :

- Design, develop, and maintain scalable data processing solutions using Azure Databricks and Azure Data Factory.

- Build and optimize end-to-end data pipelines for batch and real-time data ingestion, transformation, and loading.

- Develop complex ETL processes using PySpark on Databricks, ensuring high performance, scalability, and reliability.

- Utilize Python to script and automate data workflows and transformations.

- Write optimized SQL queries for querying and managing large datasets in Databricks and other data storage solutions.

- Integrate Azure Data Factory pipelines with Databricks and other Azure services to orchestrate data workflows.

- Collaborate with data engineers, data scientists, and analysts to understand data requirements and deliver effective data solutions.

- Implement data validation, quality checks, and error-handling mechanisms to ensure data accuracy and integrity.

- Monitor, troubleshoot, and optimize Databricks jobs and pipelines for performance and cost efficiency.

- Stay up to date with the latest developments in Azure data services and best practices in cloud data engineering.