Lead Data Engineer

Purpose of your role

This role sits within the ISS Data Platform Team. The Data Platform team is responsible for building and maintaining the platform that enables the ISS business to operate.

This role is appropriate for a Lead Data Engineer capable of taking ownership and a delivering a subsection of the wider data platform.

Key Responsibilities

Design, develop and maintain scalable data pipelines and architectures to support data ingestion, integration and analytics.
Be accountable for technical delivery and take ownership of solutions.
Lead a team of senior and junior developers providing mentorship and guidance.
Collaborate with enterprise architects, business analysts and stakeholders to understand data requirements, validate designs and communicate progress.
Drive technical innovation within the department to increase code reusability, code quality and developer productivity.
Challenge the status quo by bringing the very latest data engineering practices and techniques.

Essential Skills and Experience

Core Technical Skills

Expert in leveraging cloud-based data platform (Snowflake, Databricks) capabilities to create an enterprise lake house.
Advanced expertise with AWS ecosystem and experience in using a variety of core AWS data services like Lambda, EMR, MSK, Glue, S3.
Experience designing event-based or streaming data architectures using Kafka.
Advanced expertise in Python and SQL. Open to expertise in Java/Scala but require enterprise experience of Python.
Expert in designing, building and using CI/CD pipelines to deploy infrastructure (Terraform) and pipelines with test automation.
Data Security & Performance Optimization: Experience implementing data access controls to meet regulatory requirements.
Experience using both RDBMS (Oracle, Postgres, MSSQL) and NOSQL (Dynamo, OpenSearch, Redis) offerings.
Experience implementing CDC ingestion.
Experience using orchestration tools (Airflow, Control-M, etc..)

Bonus technical Skills:

Strong experience in containerisation and experience deploying applications to Kubernetes.
Strong experience in API development using Python based frameworks like FastAPI.

Key Soft Skills:

Problem-Solving: Leadership experience in problem-solving and technical decision-making.
Communication: Strong in strategic communication and stakeholder engagement.
Project Management: Experienced in overseeing project lifecycles working with Project Managers to manage resources.