Azure Databricks/PySpark Engineer - ETL (3-12 yrs)

Job Description : Data Engineer (Azure Databricks & PySpark)

Position : Data Engineer

Experience : 3 to 12 years

Primary Skills : Azure Databricks, PySpark, SQL

Secondary Skills : ADF (Azure Data Factory)

Location : Bengaluru/Hyderabad

Mode of Work : Hybrid

Notice Period : 0 to 30 Days

Key Responsibilities :

ETL Development & Data Processing :

- Design, develop, and maintain ETL pipelines using Azure Databricks, PySpark, and ADF.

- Extract, transform, and load data from multiple sources into Azure Data Lake Storage.

- Optimize and fine-tune ETL workflows for performance and scalability.

SQL & Query Optimization :

- Write complex SQL queries, joins, subqueries, and functions for data processing.

- Perform SQL tuning and query optimization for improved efficiency.

- Work with large datasets and ensure optimal data handling.

Azure Cloud Integration :

- Implement data solutions using Azure technologies (Databricks, ADF, ADLS).

- Work with batch and streaming data processing for real-time analytics (preferred).

- Ensure data security, compliance, and reliability in cloud-based systems.

Collaboration & Problem-Solving :

- Work closely with cross-functional teams to support data analytics and business intelligence initiatives.

- Identify and resolve data-related issues proactively.

- Provide insights and recommendations for improving ETL performance.

Required Skills & Qualifications :

- 3 to 5 years of experience in Azure Databricks, PySpark, and SQL.

- Strong SQL skills complex joins, functions, procedures, query tuning.

- Hands-on experience in ADF (Azure Data Factory) and Azure Data Lake Storage.

- Ability to handle large datasets efficiently.

- Experience in performance optimization of ETL workflows.

- Good understanding of batch and streaming data processing (preferred).

- Strong analytical and problem-solving skills.

- Ability to work independently and take ownership of tasks.