AWS Data Engineer - Python/Scala (4-6 yrs)

About the Company

- They provide insights into AI performance, enabling insurers to effectively drive value and evaluate risks.

Position : AWS Data Engineer.

Position Overview :

- As an AWS Data Engineer focused on our SaaS application, you will play a key role in designing,. implementing, and maintaining data pipelines and infrastructure on the AWS platform.

You will collaborate closely with cross-functional teams, including software engineers and data scientists, to gather requirements, design data solutions, and ensure the availability of high-quality datasets for machine-learning analysis.

The ideal candidate will have a strong background in AWS services and data engineering.

Key Responsibilities :

- Design, develop, and deploy scalable data pipelines on AWS to support the creation of machine-learning ready datasets for our SaaS application.

- Collaborate with software engineers to integrate data pipelines into our SaaS application architecture, ensuring seamless data ingestion and processing.

- Implement efficient ETL processes to extract, transform, and load data from various sources, including internal databases, third-party APIs, and streaming sources.

- Optimize data storage and retrieval for performance, scalability, and cost-effectiveness,.leveraging AWS services such as S3, Glue, Redshift, Athena, etc.

- Work closely with data scientists to understand modeling requirements and ensure the availability of clean, curated datasets for training and evaluation.

- Monitor and troubleshoot data pipelines to ensure reliability, availability, and data quality, implementing automated testing and alerting mechanisms.

- Implement security best practices for data storage, transmission, and access control, ensuring compliance with relevant regulations and standards.

- Document design specifications, deployment procedures, and operational processes, facilitating knowledge sharing and collaboration within the team.

Qualifications :

- Bachelor's degree in Computer Science, Engineering, or a related field.

- Proven experience as a Data Engineer or similar role, with at least 4 years of experience.

- Experience with Test-Driven Development (TDD), writing unit and integration tests to ensure data quality, reliability, and maintainability.

- Strong proficiency in AWS services such as S3, Glue, Redshift, Athena, EMR, Lambda, etc.

- Hands-on experience with data modeling, ETL development, and SQL.

- Excellent problem-solving skills and attention to detail.

- Strong communication and problem-solving skills, with the ability to collaborate effectively in an.
agile, fast-paced environment.

Preferred Qualifications :

AWS certification (e.g , AWS Certified Data Analytics Specialty, AWS Certified Solutions. Architect Associate).

- Experience with programming languages such as Python, Scala, or Java.

- Experience with containerization and orchestration tools such as Docker and Kubernetes.

- Familiarity with DevOps practices and tools for CI/CD.

- Familiarity with machine learning concepts and workflows, including data preprocessing, feature. engineering, and model evaluation.

- Experience with real-time data processing and streaming technologies.

- Solid knowledge of trunk-based development practices, including frequent integration with the main codebase, feature toggles, and continuous delivery workflows.