Senior Data Engineer

This role is accountable for running day to day operations of Data Platform in Azure / AWS Databricks, Data Engineer is accountable for ongoing Development, Enhancement support and maintenance data availability and data quality, performance enhancement and stability of the system.

1. Designing and implementing data ingestion pipelines from multiple sources using Azure Databricks

2. Ensure data pipelines run smoothly and efficiently

3. Adherence to security, regulatory and audit control guidelines

4. Drive optimization, continuous improvement and efficiency

SECTION B: KEY RESPONSIBILITIES AND RESULTS

Indicate key responsibilities and performance indicators of this role.

1 Designing and implementing data ingestion pipelines from multiple sources using Azure Databricks

2 Ensure data pipelines run smoothly and efficiently

3 Developing scalable and re-usable frameworks for ingesting of data sets

4 Integrating the end to end data pipeline - to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times

5 Working with event based / streaming technologies to ingest and process data

6 Working with other members of the project team to support delivery of additional project components (API interfaces, Search)

7 Evaluating the performance and applicability of multiple tools against customer requirements

8 Technically competent in Cloud and Databricks to provide technical advice to the team and to be involved in issue resolution

9 Provide on-call support and afterhours/weekend support as needed

10 Fulfill Service Requests related to Data Analytics platform

11 Lead and drive optimization and continuous improvement initiatives

12 Play gate-keeping role and conduct technical review of the changes as part of release management

13 Understand various data security standards and adhere to the required data security controls in the platform

SECTION C: QUALIFICATIONS / EXPERIENCE / KNOWLEDGE REQUIRED

Indicate key knowledge and skills required for this role to perform the tasks to a satisfactory level. To also specify a suitable level of qualification required (i.e. basic, advanced, or professional), where applicable.

Category Essential for this role Good to have

Education and Qualifications • Bachelors degree in IT, Computer Science, Software Engineering, Business Analytics or equivalent. • Relevant accredited certifications for Azure, AWS, Cloud Engineering and/or Databricks

Work Experience • Minimum 8 years of experience in data analytics field

• Experience with Azure/AWS Databricks
• Experience in building and optimizing data pipelines, architectures and data sets
• Excellent experience in Scala or Python, PySpark and SQL
• Ability to troubleshoot and optimize complex queries on the Spark platform
• Knowledgeable on structured and unstructured data design / modelling, data access and data storage techniques
• Expertise in designing and deploying data applications on cloud solutions, such as Azure or AWS
• Hands on experience in performance tuning and optimizing code running in Databricks environment
• Demonstrated analytical and problem-solving skills particularly those that apply to a big data environment • Data Engineering expertise in Big Data Management using Cloudera, Python, Hive, Scala, Data Warehouse, Data Lake, AWS, Azure

Technical / Professional Skills

Please provide at least 3 • Azure/AWS Databricks

• Python / Scala / Spark / PySpark
• HIVE / HBase / Impala / Parquet
• Sqoop, Kafka, Flume
• SQL and RDBMS
• Airflow
• Jenkins / Bamboo
• Github / Bitbucket
• Nexus