Data Engineer (Python, PySpark) - Vadodara

We are seeking a skilled Data Engineer with minimum 2 years of hands-on experience in Python, PySpark, and core data engineering practices. The ideal candidate will have strong problem-solving skills, expertise in managing data pipelines, and the ability to work closely with IT teams and business users to deliver insights. The candidate should also be familiar with Azure cloud technologies, including Fabric services, and have experience implementing CI/CD pipelines.

Key Responsibilities:

• Design, build, and maintain scalable data pipelines using Python and PySpark.
• Implement streaming pipelines, incremental data refresh processes, and MERGE operations for Delta Lake.
• Understand and follow best practices for writing unit tests and managing CI/CD pipelines using Azure DevOps or other tools, with knowledge of Git.
• Collaborate with IT leads and meet with business users to understand requirements and apply data engineering principles to deliver insights.
• Work with Microsoft Azure services, including Azure Storage and Azure Fabric (preferred).
• Manage incremental data loads and develop processes for real-time and batch data streaming.
• Monitor data pipelines, ensuring lineage tracking, performance optimization, and troubleshooting.
• Apply data governance principles where applicable.

Technical Skills:

• Proficiency in Python and PySpark, with hands-on experience in core data engineering practices.
• Familiarity with PySpark libraries, streaming frameworks, and incremental refresh techniques.
• Experience with Azure Storage, RDBMS systems such as SQL Server and SAP HANA databases.
• Knowledge of data modeling concepts and ability to write complex SQL queries efficiently.
• Understanding of CI/CD practices using Azure DevOps or similar tools, along with version control using Git.
• Knowledge of data governance, lineage tracking, and monitoring tools (preferred).
• Good to have: Understanding of SAP ERP data collaboration and experience working with business data systems.

Interested candidates can send their resume at itcv@alembic.co.in with the subject line "Data Engineer "