Data Engineer - PySpark/Azure Databricks (4-5 yrs)

Job Description :

We are seeking a skilled Data Engineer with expertise in PySpark, Databricks, and SQL to design, build, and maintain scalable data pipelines and infrastructure.

The ideal candidate will have experience in handling large datasets, optimizing performance, and working with cloud-based data platforms.

Key Responsibilities :

- Develop and maintain scalable ETL pipelines using PySpark and Databricks.

- Optimize and manage data workflows, ensuring high performance and reliability.

- Work with structured and unstructured data to enable analytics and reporting.

- Implement and enforce data governance and security best practices.

- Collaborate with data scientists, analysts, and business stakeholders to understand data needs.

- Monitor and troubleshoot data pipeline issues and improve data quality.

- Develop and optimize complex SQL queries for data extraction and transformation.

- Work with cloud platforms such as AWS, Azure, or GCP for data storage and processing.

- Automate data workflows and deployments using CI/CD practices.