Associate Data Engineer

In this role as Associate Data engineer on the Global Data and Analytics (GDA) team, you will be working on our new strategic visibility initiative to develop recommendation solutions facing internal and external users. The overall objective is to develop actionable recommendations which enable unprecedented flexibility during supply chain execution and strategic planning and unlock new types of services for our internal users and our customers. Building on top of a best-in-industry visibility data foundation, you will be partnering with product managers and engineering teams to develop scalable solutions following industry best-practices. There is a lot of exciting challenges ahead of us, and the ideal candidate will have a passion for working on industry-transforming products and creating impact from the ground up in a fast-paced environment.

You should have demonstrated ability to make sense out of large, integrated datasets, and build data pipelines to serve machine learning (ML) models. For this role it is crucial to have prior hands-on experience developing and deploying end to end data pipelines with feature engineering to production in collaboration with data engineering, software engineering, ML engineering teams.

Key Responsibilities

We are seeking a strong Data Engineer with expertise in design and build scalable solutions using Spark, Python, Kafka, Delta, Docker, and K8s

The ideal candidate should have an expertise in data architecture, system design, building and deploying data and ML pipelines using CICD

Collaborate with data scientists and product owners to translate business problems and/or technical ML problems into scalable performant data pipelines.

Good problem-solving and troubleshooting skills.

Key Skills

Deep technical knowledge and expert coding skills in python and pyspark

Experience building data pipeline with Spark, Kafka, Structured streaming, etc.

Develop, test, and deploy end-to-end data pipelines together with the team.

Hands-on experience working with complex datasets (e.g. nested Jsons, sensor data,etc)

Build data and feature engineering pipelines for train/test machine learning models.

Ensure data quality and consistency through robust data validation and monitoring

Strong analytical skills with expert level competency in SQL

Optimize and fine-tune Spark jobs for maximum efficiency and resource utilization.

Qualifications

Bachelors degree in B.E/BTech, preferably in computer science

2+ years of years of relevant experience in the field of Data Engineering

2+ years of hands-on experience with Apache Spark, Python and SQL

Experience with collaborative development workflow: IDE (Integrated Development Environment), Version control(github), CI/CD (e.g. automated tests in github actions)

Communicate effectively with technical and non-technical audiences with experience in stakeholder management

A good team player

Preferred qualifications

Experience working with large datasets and big data technologies to train and evaluate machine learning models.

Experience with containerization: Kubernetes Docker

Expertise in building cloud native applications and data pipelines (Azure, Databricks, AWS, GCP) C

Experience with common dashboarding and API technologies (PowerBI, Superset, Flask, FastAPI, etc