S P A Enterprise Services - Data Engineer - Python/Spark (3-5 yrs)

Job Description :

Responsibilities :

- Design, develop, and maintain efficient and reliable data pipelines using PySpark, Python, and AWS services (S3, EC2, Lambda, Redshift, CloudFormation, etc.).

- Implement data transformation logic, data cleansing, and data enrichment processes to ensure data quality and consistency.

- Optimize data pipelines and queries using Spark SQL and PySpark to improve performance and scalability.

- Refactor and modernize legacy codebases, focusing on clean code, improved readability, and maintainability.

- Utilize CloudFormation to provision and manage AWS infrastructure in an automated and repeatable manner.

- Implement and adhere to best practices for code versioning using Git and artifact management using JFrog Artifactory.

- Develop and execute comprehensive unit tests and implement

- Test-Driven Development (TDD) methodologies to ensure code quality and reliability.

- Diagnose and resolve complex bugs and performance issues in data pipelines and applications.

- Collaborate effectively with data scientists, analysts, and other engineers to understand data requirements and deliver solutions.

- Maintain a strong understanding of AWS architecture and best practices, and apply them to design and implement efficient and scalable data solutions.