Data Engineer - Python/PySpark (4-8 yrs)

Position : Data Engineer

Work Location : Bangalore

Experience : 4+ Years

Job Description :

Key Responsibilities :

- 5+ years of data management experience.

- Technically strong in Python, PySpark, Hadoop, Hive

- Experienced in data integration projects working for large clients using the above tech stacks.

- Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teams and stakeholders.

- Replicate and optimize SQL Server stored procedures in Snowflake, ensuring functional accuracy and performance in the cloud environment.

- Implement and manage automated workflows using Prefect Scheduler to streamline ETL/ELT processes, ensuring reliability, monitoring and alerting for critical workflows.

- Design and maintain CI/CD pipelines using tools such as Jenkins or GitLab, automating testing, deployment and version control.

- Collaborate with cross-functional teams, including data scientists, analysts and business stakeholders, to understand data requirements and deliver impactful solutions, simplifying technical concepts for non-technical stakeholders.

- Use version control systems like GitLab to manage collaborative code development, branching and versioning while fostering best practices in code management.

- Continuously monitor, troubleshoot and optimize data pipelines for performance, scalability and compliance with SLAs.

- Develop reconciliation frameworks and implement automated migration validation processes to ensure data consistency and accuracy during cloud migrations.

- Maintain comprehensive documentation for data pipelines, architectures and processes to enable effective knowledge sharing.