Data Engineering Tech Lead

Job Description

Major accountabilities:

• Responsible for the detailed design, development, and delivery of pipeline solutions to address business needs. Define solution architecture and develop detailed process designs for day-to-day operation of the preclinical pipelines
• Ensure solution designs align with the overall data42 architecture. Facilitate peer reviews and secure sign-off from the preclinical lead and business stakeholders
• Coordinate and oversee the data engineering teams day-to-day activities, review and merge pull requests, manage the product delivery roadmap, and track progress on user stories in the JIRA board.
• Review pipeline releases and migration activities, conduct code reviews, serve as technical SME for the team. Lead and actively contribute to documentation efforts and audit readiness activities.
• Proactively manage release notes and communicate pipeline updates to stakeholders.
• Initiate and review architecture changes in the pipeline, ensuring regular optimization and continuous improvement.

Key performance indicators:

• Achieve high level of quality and timeliness of delivering preclinical pipeline deliverables as assessed by the Preclinical pipeline lead
• Ensure that technical documentation and pipeline management are aligned with standards and effectively maintained
• Collaboration with other data42 product teams and product technical leads
• Ability and effectiveness in training, mentoring and coordinating internal and external analysts assigned to the same project as assessed by the functional/operational manager.

Job Dimensions:

The role involves close collaboration with the Preclinical pipeline lead to define the roadmap and delivery plans. Work with the team to ensure timely execution of deliverables, maintain technical documentation, and manage release communications to end users.

Minimum Requirements:

Education: Bachelor’s/Master's degree in Computer Science, Applied Mathematics, Engineering, or any other technology related field; equivalent of the same in working experience may also be accepted

Work Experience:

• 8+ years IT experience, with 6+ years in Data Engineering on Big Data platforms.
• Ability to work and lead cross-functional team in a matrix organization. Led and mentored technical teams for 2+ years.
• Must have experience managing pipeline development activities, with strong project management skills.
• Strong communication skills with the ability to effectively collaborate with cross functional teams and stakeholders.
• Proficient in working with Git workflow for project execution; a strong understanding of DevOps (CI/CD framework) is essential.
• Had actively participated in agile work practices and coordinated with team members to ensure smooth project execution.
• Expertise in Python, PySpark and Spark.
• Hands-on experience with JIRA and Confluence for technical documentation. Responsible for technical documentation, audit readiness, and release management.
• Strong Analytical thinking and problem-solving skills.
• Expertise on Palantir Foundry Platform using Code Repository, Code Workbook, Data Connection, etc. components to develop data pipelines.
• Strong knowledge of AI/ML concepts with hands-on experience.
• Knowledge of preclinical in-vivo study data e.g. CDSIC SEND standard will be desirable.

Skills:

• Back-End Development.
• Code Analysis.
• OOD (Object-Oriented Design).
• Data Wrangling.
• Software Documentation.
• Software/Data Engineering.
• Software/Data Testing.
• Analytical thinking.
• CDISC SEND.
• Palantir Foundry.
• Unit Testing.

Desired Skills:

OOD (Object-Oriented Design), REST (Representational State Transfer), Software Design, Software Documentation, Software Engineering, Software Testing, Palantir Foundry, Unit Testing, CDISC SEND, Analytical thinking.

Languages :

• Fluent English (Oral and Written)

Skills Desired

Code Analysis, Hardware Architecture, MVVN (Model View Model), MySQL, Net Framework, Nosql Database, OOD (Object-Oriented Design), REST (Representational State Transfer), RTOS (Real-Time Operating System), Software Architecture, Software Design, Software Documentation, Structured Programming, Systems Architecture, Ubuntu Linux, Unit Testing, WPF