Azure Data Engineer - ETL/PySpark (3-9 yrs)

Job Title : Azure Data Engineer | Senior Data Engineer | Technical Lead

Location : Bangalore (Whitefield) / Hyderabad (Knowledge City) / Pune

Working Model : Hybrid

No. of Positions : 20

Experience : 3 to 9 years in Cloud Data Engineering

Primary Skills : Azure Databricks, PySpark, SQL

Secondary Skills : Azure Data Factory (ADF)

Notice Period : Immediate to 30 days (Preferred)

Job Summary :

We are seeking highly skilled Azure Data Engineers with expertise in Azure Databricks, PySpark, and SQL to design, develop, and maintain scalable data solutions. The ideal candidate will have experience working with large datasets, complex ETL pipelines, and Azure cloud technologies, ensuring optimal performance and reliability.

Key Responsibilities :

- Design, develop, and maintain ETL pipelines using Databricks, PySpark, and Azure Data Factory (ADF) to process structured and unstructured data.

- Optimize and fine-tune SQL queries for performance, scalability, and efficiency, handling complex joins, subqueries, functions, and stored procedures.

- Implement and manage large-scale data processing solutions using Azure technologies, including Azure Data Lake Storage, ADF, and Databricks.

- Ensure data reliability, security, and compliance while building scalable data pipelines.

- Work with batch and streaming data processing frameworks (preferred but not mandatory).

- Collaborate with cross-functional teams to support data analytics, reporting, and business intelligence initiatives.

- Troubleshoot performance issues and enhance existing data workflows for efficiency.

- Work independently with minimal supervision, taking ownership of tasks and delivering high-quality solutions.

Required Skills & Experience :

- 3 to 9 years of experience in Cloud Data Engineering, with a minimum of 3 years in Azure Databricks & PySpark.

- Strong expertise in SQL, with hands-on experience in query optimization and performance tuning.

- Proven experience in building large-scale ETL pipelines using Databricks, PySpark, and Azure Data Factory (ADF).

- Experience handling large datasets and optimizing ETL processes for performance.

- Familiarity with batch and streaming data processing (good to have).

- Ability to work independently, troubleshoot issues, and deliver scalable data solutions.

- Experience with Azure Data Lake Storage and other Azure cloud services.

- Strong problem-solving skills and an analytical mindset.

- Excellent communication and collaboration skills.

Preferred Certifications :

- Microsoft Certified: Azure Data Engineer Associate (DP-203)

- Databricks Certified Data Engineer Associate

- Databricks Certified Data Engineer Professional

- Microsoft Certified: Azure Fundamentals (AZ-900) (Optional but beneficial)