Software Engineer - Pyspark

Roles and Responsibilities :

• Design, develop, test, deploy and maintain large-scale data processing pipelines using AWS services such as S3, Lambda, Step Functions.
• Collaborate with cross-functional teams to identify business requirements and design solutions that meet those needs.
• Develop high-quality code in Python using PySpark to process large datasets stored in SQL databases.
• Troubleshoot issues related to pipeline failures and optimize system performance.

Job Requirements :

• 5-12 years of experience in software development with a focus on big data processing.
• Strong proficiency in AWS services including S3, Lambda, Step Functions.
• Experience working with Python software developer tools like PySpark.