AWS PySpark Data Engineer

This is a permanent work from home position from anywhere in India

Notice period - less than 30 Days (Immediate joiners preferred)

We are seeking an AWS PySpark Data Engineer to design, develop, and optimize data pipelines in a cloud-based environment. The ideal candidate should have expertise in PySpark, AWS services, and big data processing.

Key Responsibilities:

• Develop and optimize ETL/ELT data pipelines using PySpark on AWS.
• Work with AWS services like Glue, Lambda, S3, EMR, Redshift, Athena, and Step Functions.
• Implement data transformations, aggregations, and validations for structured and unstructured data.
• Optimize PySpark jobs for performance and scalability.
• Design and maintain data lake and data warehouse solutions on AWS.
• Work with structured (SQL, Parquet) and unstructured data formats.
• Collaborate with data scientists, analysts, and stakeholders to ensure data integrity and availability.
• Troubleshoot performance bottlenecks and resolve data processing issues.

Required Skills:

• 5+ years of experience in PySpark and big data processing.
• Strong expertise in AWS services (Glue, S3, Redshift, EMR, Lambda, Step Functions, Athena).
• Hands-on experience in SQL and NoSQL databases.
• Experience with data lake and warehouse architectures.
• Proficiency in Python, Spark, and SQL for data processing.
• Strong understanding of ETL workflows, data modeling, and data governance.
• Experience in performance tuning of Spark jobs.

Preferred Skills:

• Experience with CI/CD pipelines for data engineering.
• Knowledge of Apache Airflow for workflow orchestration.
• Familiarity with Terraform or CloudFormation for infrastructure automation.
• Exposure to Kafka or streaming data processing is a plus.