Data Pipelines Engineering Lead

General Summary:

We are looking for an experiencedhands-ondataengineering lead with 10+ years of experience to lead the definition, design and deployment of bigdatapipelines systems. Responsibilities include high-level software architecture and design for the whole system, advancing and advocating for software quality as a means of maintaining long-term productivity, and coordinating development teams to ensure consistency and efficiency.

Relevant technologies and skills include AWS Cloud, Kubernetes, Argo Workflows, Observability/Monitoring tools, Bazel, Python and familiarity with modern C++. You will also be coordinating and collaborating with the cloud infrastructure and with product and research teams.

Our teams are primarily in India, Sweden, Romania, Germany, and the US.

Minimum Qualifications:

Bachelor's degree in Engineering, Information Systems, Computer Science, or related field and 4+ years of Software Engineering or related work experience.
OR
Master's degree in Engineering, Information Systems, Computer Science, or related field and 3+ years of Software Engineering or related work experience.
OR
PhD in Engineering, Information Systems, Computer Science, or related field and 2+ years of Software Engineering or related work experience.

2+ years of work experience with Programming Language such as C, C++, Java, Python, etc.

Responsibilities:

Architect, develop, and maintain the Autonomous DrivingDataplatform, both on-premises and AWS Cloud

Lead a DistributedDataSystems team while remaining hands-on

Translate Product Management and Department requirements into work packages.

Collaborate withdatascientists and software engineers to define requirements and ensure seamless integration ofdatajobs into the platform.

Optimize the platform's performance, scalability and costs, considering factors such as resources utilization, throughput requirements anddatamovement.

Monitor and troubleshoot system performance, identifying and resolving issues to ensure the availability and reliability of thedataplatform.

Implement and maintain CIpipelines for application containers, and deployment using technologies like Argo Workflow.

Implement and maintain monitoring stack to ensure the health and performance of thedataplatform.

Provide technical guidance and mentorship to junior team members.

What are we looking for:

• Proven experience asDataOpsEngineeringLeador similar role, with a focus on large-scaleDatainfrastructure.
• Strong expertise in configuring and optimizingKubernetesclusters fordataworkloads.
• Experience with the ELK stack, in particular ElasticSearch and Kibana
• Proficient in using the Kubernetes platform, including technologies like Helm, ArgoCD, Argo Workflow, Prometheus, and Grafana.
• Solidprogrammingskills in languages likePython, C++ and experience Bazel build systems.
• Familiarity with containerization technologies such as Docker and orchestration tools.
• Experience with CI/CDpipelines and automation tools (e.g., Jenkins, GitHub, ArgoCD).
• Experience withAWSservices such as EKS, EC2, VPC, IAM, S3, EFS, Batch
• Experience with AWS logging and monitoring tools.
• Strong problem-solving skills and the ability to troubleshoot complex technical issues.
• Excellent communication and collaboration skills to work effectively within a cross-functional team.

We would love to see:

• Experience withDataPlatforms for Automated Driving.
• Familiarity withdatastorage and retrieval systems.
• Understanding of security and compliance requirements inDatainfrastructure.
• Proven track record of creating software systems that emphasize software quality
• Experience with both system level architecture and class level design
• In-depth understanding of distributed computing and parallel computing.
• Extensive experience with different system architectural concepts like event driven architecture, workflows-based architecture, domain driven design