AWS Data Engineer - Big Data (4-8 yrs)

Overview /Objective :

We is looking for a Data Engineer with experience in building modern data platforms from the ground up.

The successful candidate will build and maintain cloud-centric data processing capabilities that unleash the value of the League's data assets to gain competitive advantage in the marketplace. They will be a hands-on contributor in the design and implementation of our cloud data platform that powers advanced analytics workloads.

The Data Engineer will work in an agile environment and will be responsible for building and maintaining data integration, ingestion, curation and pipeline orchestration capabilities. They are comfortable challenging assumptions to improve existing solutions and ensure the team is building the best scalable and cost-efficient product.

Job Description

Responsibilities :

- Develop, test, and deploy software to generate data assets (relational, non-relational) for use by downstream BI engineers and data scientists

- Work with big data and cloud technologies such as EC2, Lambda, AWS Glue, Airflow, dbt, Redshift etc.

- Work closely with stakeholders to ensure successful data asset design and development

- Create software artifacts and patterns for reuse within the Data Engineering team.

- Ensure data pipelines are scalable, resilient and produced with the highest quality standards, metadata and validated for completeness and accuracy

- Work on a cross-functional Agile team responsible for end-to-end delivery of business needs

- Help improve data management processes - acquiring, transforming and storing massive volumes of structured and unstructured data

- Work closely with development teams to learn about needs, current processes and to promote best practices.

Required Qualifications :

- University degree in Computer Science, Mathematics, Engineering, or related field.

- 5+ years of experience in software engineering with strong focus on data.

- Experience working with Cloud data platforms, preferably AWS (Lambda, Step Functions, S3, AWS Glue, Athena, Redshift).

- An expert in Python and SQL, including query optimization for relational, NoSQL and columnar databases.

- Sound knowledge of CI workflows and build/test/deploy automation.

- Strong understanding of data modelling concepts and best practices.

- Relevant experience with IaC (Terraform, CloudFormation)

- Relevant experience with modern big data processing and orchestration tools such as dbt and Airflow.

- A great teammate and self-starter, strong detail orientation is critical in this role.