The primary role of a Data Engineer position is to participate in the planning, building and operating of data platforms that make data available to other software services, data scientists, and data analysts. The position has the following responsibilities:
Participate in the planning process with senior engineers to align data platform architecture with business requirements
Provision the collection of data from various sources as required
Structure and configure data pipelines, databases, and data warehouses
Structure and configure cloud services in the AWS ecosystem that provide data engineering functionalities
Work with data scientists and analysts to optimise data structures with ETL/ELT tasks based on the modelling requirements of the scientific and analytical processes
Automate data collection, transformation, life cycling, and disposal tasks
Work to ensure well-executed data governance and compliance with data privacy and security standards as required
Work with application developers to integrate data platforms with business applications
Knowledge and Technical Skill Requirements Mandatory
Proficiency in programming with Python at an advanced level
Experience working with Jupyter Notebooks and/or Databricks
Experience with and advanced working knowledge of relational and non-relational databases, SQL scripting and associated query languages
Proficiency in using Linux operating systems programmatically
Working knowledge of big data technologies such as Apache Hadoop, Apache Spark, Hive, Presto and Apache Kafka
Preferred
Working with big data cloud services such as AWS EMR, AWS Glue and AWS MSK for Apache Kafka, AWS Lambda and the various AWS storage solutions
Working with Elasticsearch and Kibana
Knowledge of data collection, transmission, and ingestion processes with MQTT, Sparkplug and SCADA systems in the context of Industrial Internet of Things (IIoT) technologies
Working knowledge of message queues, stream processing and event stores
Working knowledge of mathematical modelling, data science and data analytics applications and processes
Experience and Qualification Requirements
Degree or certification in a mathematical, scientific, engineering, or associated interdisciplinary field or equivalent experience
2 years of professional data engineering experience