Data Engineer - Python/PySpark (6-9 yrs)

Data Engineer

Job Title : Data Engineer

About the Role :

- This role involves designing, developing, and maintaining data pipelines and solutions within an AWS environment.

- The ideal candidate will have strong expertise in ETL processes, data warehousing, and big data technologies.

Key Responsibilities :

- Develop and maintain ETL pipelines using AWS services like Glue, EMR, and S3.

- Write and optimize complex SQL queries for data extraction, transformation, and loading.

- Develop and maintain data pipelines using Python and PySpark.

- Work closely with cross-functional teams (e.g , data analysts, data scientists, business stakeholders) to understand data requirements and translate them into technical solutions.

- Troubleshoot and resolve data quality issues.

- Implement and maintain data security and privacy best practices.

- Contribute to the development and improvement of data infrastructure.

- Participate in the entire software development lifecycle, including design, development, testing, deployment, and maintenance.

- Experience with version control systems (Git) and CI/CD pipelines (Bamboo).

- Understanding of industry standards and best practices in data engineering.

Required Skills :

- 6+ years of experience with AWS ETLs, Glue, EMR, S3, and Athena.

- Expert proficiency in Python and PySpark.

- Advanced knowledge of SQL and database management systems.

- Excellent problem-solving and analytical skills.

- Strong communication and collaboration skills.

- Experience with version control practices and CI/CD processes (Git, Bamboo, etc.

Preferred Skills :

- Experience with Redshift, Aurora, and DynamoDB.

- Experience in the Financial Services Industry, particularly with IMRE (Investment Management Regulatory Excellence).