Exponentia.ai - AWS Data Engineer - ETL/DataLake (4-5 yrs)

Job Overview :

As an AWS Data Engineer, you will be responsible for designing, building, and maintaining data pipelines and solutions on the AWS platform. You will work closely with data scientists, analysts, and other teams to ensure the seamless integration of data into analytical platforms and support key business decisions through the effective use of data infrastructure.

Key Responsibilities :

- Data Pipeline Development : Design, implement, and maintain end-to-end data pipelines using AWS services like AWS Glue, AWS Lambda, and Amazon Kinesis.

- Data Storage Management : Manage and optimize data storage solutions using AWS services such as Amazon S3, Amazon Redshift, and DynamoDB.

- ETL Processes : Develop, test, and maintain efficient ETL (Extract, Transform, Load) processes to move data from source systems to data lakes, warehouses, or databases.

- Automation : Automate data-related tasks and processes to ensure scalability and efficiency using tools such as AWS CloudFormation, Terraform, and AWS Step Functions.

- Cloud Infrastructure : Manage and optimize cloud infrastructure for data workloads on AWS, ensuring security, scalability, and performance.

- Collaboration : Work closely with data scientists, business analysts, and other stakeholders to understand data needs and provide support for data-driven decision-making.

- Data Quality : Monitor data quality, troubleshoot data issues, and implement solutions to improve data accuracy and consistency.

- Performance Tuning : Optimize and fine-tune AWS data solutions for performance, cost-efficiency, and reliability.

- Documentation : Document data pipelines, architecture, and processes to ensure reproducibility and knowledge sharing across teams.

Key Skills & Qualifications :

- Experience : 4-5 years of experience in data engineering, with hands-on experience in AWS technologies and cloud-based data solutions.

- AWS Services : Strong knowledge of AWS services such as Amazon S3, AWS Glue, Amazon Redshift, AWS Lambda, AWS Kinesis, and AWS RDS.

- Data Warehousing : Experience working with data warehouses (e.g., Amazon Redshift, Snowflake).

- ETL/ELT Tools : Proficiency in building and managing ETL pipelines using AWS Glue, Apache Spark, or other related tools.

- Programming : Strong programming skills in languages such as Python, SQL, or Java for data processing and scripting.

- Database Knowledge : Hands-on experience with relational and NoSQL databases, including MySQL, PostgreSQL, DynamoDB, etc.

- Cloud Infrastructure & Automation : Familiarity with cloud infrastructure management and automation tools like AWS CloudFormation, Terraform, or AWS Step Functions.

- Data Modeling : Experience with data modeling, schema design, and optimizing data storage and retrieval for performance.

- Problem Solving : Strong troubleshooting and debugging skills for identifying and resolving data issues quickly.

- Communication : Excellent communication skills, with the ability to collaborate effectively across teams and with stakeholders.

Preferred Qualifications :

- Certifications : AWS Certified Data Analytics - Specialty or any relevant AWS certifications.

- Big Data Technologies : Experience with big data frameworks such as Apache Hadoop, Apache Spark, or Apache Kafka.

- Data Governance : Knowledge of data governance practices, including data security, privacy, and compliance.