Associate Data Engineer

Job Description

Maersk, the world s largest shipping company responsible for moving 20 % of global trade, is on a mission to become the Global Integrator of Container Logistics. To achieve this, we are transforming into an industrial digital giant by combining our assets across air, land, ocean, and ports with our growing portfolio of digital assets to connect and simplify our customer s supply chain through global end-to-end solutions, all the while rethinking the way we engage with customers and partners.

In this role as Data engineer on the Global Data and Analytics (GDA) team, you will be working on our new strategic visibility initiative to develop recommendation solutions facing internal and external users. The overall objective is to develop actionable recommendations which enable unprecedented flexibility during supply chain execution and strategic planning and unlock new types of services for our internal users and our customers. Building on top of a best-in-industry visibility data foundation, you will be partnering with product managers and engineering teams to develop scalable solutions following industry best-practices. There is a lot of exciting challenges ahead of us, and the ideal candidate will have a passion for working on industry-transforming products and creating impact from the ground up in a fast-paced environment.

You should have demonstrated ability to make sense out of large, integrated datasets, and build data pipelines to serve machine learning (ML) models. For this role it is crucial to have prior hands-on experience developing and deploying end to end data pipelines with feature engineering to production in collaboration with data engineering, software engineering, ML engineering teams.

Key Responsibilities
We are seeking a strong Data Engineer with expertise in design and build scalable solutions using Spark, Python, Kafka, Delta, Docker, and K8s
The ideal candidate should have an expertise in data architecture, system design, building and deploying data and ML pipelines using CICD
Collaborate with data scientists and product owners to translate business problems and/or technical ML problems into scalable performant data pipelines.
Good problem-solving and troubleshooting skills.

Key Skills
Deep technical knowledge and expert coding skills in python and pyspark
Experience building data pipeline with Spark, Kafka, Structured streaming, etc.
Develop, test, and deploy end-to-end data pipelines together with the team.
Hands-on experience working with complex datasets (e. g. nested Jsons, sensor data, etc)
Build data and feature engineering pipelines for train/test machine learning models.
Ensure data quality and consistency through robust data validation and monitoring
Strong analytical skills with expert level competency in SQL
Optimize and fine-tune Spark jobs for maximum efficiency and resource utilization.

Qualifications
Bachelors degree in B. E/BTech, preferably in computer science
3+ years of years of relevant experience in the field of Data Engineering
2+ years of hands-on experience with Apache Spark, Python and SQL
Experience with collaborative development workflow: IDE (Integrated Development Environment), Version control(github), CI/CD (e. g. automated tests in github actions)
Communicate effectively with technical and non-technical audiences with experience in stakeholder management
A good team player

Preferred qualifications
Experience working with large datasets and big data technologies to train and evaluate machine learning models.
Experience with containerization: Kubernetes & Docker
Expertise in building cloud native applications and data pipelines (Azure, Databricks, AWS, GCP) C
Experience with common dashboarding and API technologies (PowerBI, Superset, Flask, FastAPI, etc

Maersk is committed to a diverse and inclusive workplace, and we embrace different styles of thinking. .

We are happy to support your need for any adjustments during the application and hiring process. If you need special assistance or an accommodation to use our website, apply for a position, or to perform a job, please contact us by emailing .