Sr. Data Engineer

Role & responsibilities

• Work with data product managers, analysts, and data scientists to architect, build and maintain data processing pipelines in SQL or Python.
• Build and maintain a data warehouse / data lake-house for analytics, reporting and ML predictions.
• Implement DataOps and related DevOps focused on creating ETL pipelines for data analytics / reporting, and ELT pipelines for model training.
• Support, optimise and transition our current processes to ensure well architected implementations and best practices.
• Work in an agile environment within a collaborative agile product team using Kanban
• Collaborate across departments and work closely with data science teams and with business (economists/data) analysts in refining their data requirements for various initiatives and data consumption requirements.
• Educate and train colleagues such as data scientists, analysts, and stakeholders in data pipelining and preparation techniques, which make it easier for them to integrate and consume the data they need for their own use cases.
• Participate in ensuring compliance and governance during data use, to ensure that the data users and consumers use the data provisioned to them responsibly through data governance and compliance initiatives.
• Become a data and analytics evangelist, and promote the available data and analytics capabilities and expertise to business unit leaders, and educate them in leveraging these.

Preferred candidate profile

What you'll need to be successful

• 8+ years of professional experience with data processing environments used in large scale digital applications.
• Extensive experience with programming in Python, Spark( SparkSQL) and SQL
• Experience with warehouse technologies such as Snowflake, and data modelling, lineage and data governance tools such as Alation.
• Professional experience of designing, building and managing bespoke data pipelines (including ETL, ELT and lambda architectures), using technologies such as Apache Airflow, Snowflake, Amazon Athena, AWS Glue, Amazon EMR, or other equivalent.
• Strong, fundamental technical expertise in cloud-native technologies, such as serverless functions, API gateway, relational and NoSQL databases, and caching.
• Experience in leading / mentoring data engineering teams.
• Experience in working in teams with data scientists and ML engineers, for building automated pipelines for data pre-processing and feature extraction.
• An advanced degree in software / data engineering, computer / information science, or a related quantitative field or equivalent work experience.
• Strong verbal and written communication skills and ability to work well with a wide range of stakeholders.
• Strong ownership, scrappy and biassed for action.

Perks and benefits