Data Engineer - AWS (7-10 yrs)

About the job :

We are seeking a skilled AWS Data Engineer with expertise in various AWS services.

The an ideal candidate will have hands-on experience with Lambda, Glue, SNS, SQS, Step Functions, PySpark, Python, Athena, CloudWatch, S3, and more.

The successful candidate should also have working experience with various data file formats such as JSON, XML, CSV, Parquet.

Proficiency in SQL, and visualization tool experience with Looker or Power BI.

Responsibilities :

- Develop and maintain robust data pipelines using AWS Glue for efficient ETL processes.

- Implement serverless computing solutions with AWS Lambda to automate tasks and processes.

- Utilize SNS and SQS for efficient messaging and event-driven architecture.

- Design and orchestrate data workflows using AWS Step Functions.

- Leverage PySpark, Python, and SQL for data processing, analysis, and transformation.

- Implement and optimize queries using AWS Athena for efficient querying of large datasets.

- Monitor and manage resources and applications using AWS CloudWatch.

- Manage data storage and retrieval using AWS S3.

- Work with various data file formats, including JSON, XML, CSV, TSV, Parquet, and execute SQL queries as needed.

- Utilize visualization tools such as Looker or Power BI for effective data representation.

- Build end-to-end data pipelines, from conception to implementation, ensuring scalability and efficiency.

- Hands-on experience with CI/CD tools such as Jenkins, GitLab/Github, Jira, Confluence, and other related tools.

- Experience working with Delta Lake for efficient version control and data management.

Qualifications :

- 7+ years experience as a Data Engineer in consumer finance or equivalent industry

- Proven experience as a Data Engineer with a strong focus on AWS services.

- Proficiency in Python, PySpark, SQL, and AWS services for data processing and analysis.

- Hands-on experience with AWS Lambda, Glue, SNS, SQS, Step Functions, Athena, CloudWatch, and S3.

- Practical experience working with JSON, XML, CSV, TSV, Parquet file formats.

- Experience with visualization tools such as Looker or Power BI is a significant plus.

- Good understanding of serverless architecture and event-driven design.

- Hands-on experience with CI/CD tools, including Jenkins, GitLab/Github, Jira, Confluence, and other related tools.

- Comfortable learning about and deploying new technologies and tools.

- Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines.

- Good written and oral communication skills and the ability to present results to non- technical audiences.

- Knowledge of business intelligence and analytical tools, technologies, and techniques.

- Experience with Terraform is a plus