Senior Cloud Data Engineer - AWS (7-12 yrs)

Cloud Data Engineer

You will have the following responsibilities :

Design :

- Analyse relevant internally and externally sourced data (raw data) to generate BI and Advanced Analytics datasets based on your stakeholders' requirements

- Design data pipelines to curate sourced data into the inhouse data warehouse

- Design data marts to facilitate dataset consumption out of the inhouse data warehouse by business and IT internal stakeholders

- Design data model changes that align with the inhouse data warehouse standards

- Define migration execution activities to move data from existing database solutions to the inhouse data warehouse

Engineer :

- Regular housekeeping of raw data and data stored in the inhouse data warehouse

- Build and maintenance of data pipelines and data platforms

- Build data solution prototypes

- Explore ways to enhance data quality and reliability

- Identify and realize opportunities to acquire better data (raw data)

- Develop analytical tooling to better support BI and Advanced Data Analytics activities

- Execute data migration from existing databases to the inhouse data warehouse

- Promote and champion data engineering standards and best-in-class methodology

Responsibilities :

• Develop, test, and deploy software to generate data assets (relational, non-relational) for use by downstream BI engineers and data scientists
• Work with big data and cloud technologies such as EC2, Lambda, AWS Glue, Airflow, dbt, Redshift etc.
• Work closely with stakeholders to ensure successful data asset design and development
• Create software artifacts and patterns for reuse within the Data Engineering team.
• Ensure data pipelines are scalable, resilient and produced with the highest quality standards, metadata and validated for completeness and accuracy
• Work on a cross-functional Agile team responsible for end-to-end delivery of business needs
• Help improve data management processes - acquiring, transforming and storing massive volumes of structured and unstructured data
• Work closely with development teams to learn about needs, current processes and to promote best practices.

• University degree in Computer Science, Mathematics, Engineering, or related field.
• 5+ years of experience in software engineering with strong focus on data.
• Experience working with Cloud data platforms, preferably AWS (Lambda, Step Functions, S3, AWS, Glue, Athena, Redshift).
• An expert in Python and SQL, including query optimization for relational, NoSQL and columnar databases.
• Sound knowledge of CI workflows and build/test/deploy automation.
• Strong understanding of data modelling concepts and best practices.
• Relevant experience with IaC (Terraform, CloudFormation)
• Relevant experience with modern big data processing and orchestration tools such as dbt and Airflow.
• A great teammate and self-starter, strong detail orientation is critical in this role.