Sr Data Engineer

• The primary purpose of this role is to translate business requirements and functional specifications into logical program designs and to deliver modules, stable application systems, and Data Engineering solutions.
• This includes developing, configuring, or modifying complex integrated business and/or enterprise infrastructure or application solutions within various computing environments.
• This role facilitates the implementation and maintenance of complex business and enterprise Big Data Platform (Hadoop on-Prem and GCP Cloud) solutions to ensure successful deployment of released applications.

• In this role you will design, develop, and maintain scalable data pipelines and systems using Pyspark, SparkSQL on GCP platform. You will implement and optimize data engineering processes with a focus on performance, cost-efficiency, and reliability.
• You will be leading and executing data engineering projects from inception to completion, ensuring timely delivery and high quality while you continuously monitor, troubleshoot, and improve data pipelines and workflows to ensure optimal performance and cost-effectiveness.
• You will collaborate with cross-functional teams to understand data needs and deliver solutions that meet business requirements, mentor, and educate junior engineers on best practices, emerging technologies, and efficient data engineering techniques.

• 5+ years of Experience in data engineering.

• Bachelors degree in engineering, Computer Science, CIS, or related field (or equivalent work experience in a related field)

Hands on GCP experience preferably in the following services

• DataProc
• GCS (Google Cloud Storage)
• Big Query Tables
• Cloud Composer

• Pyspark (Spark with Python)
• Python scripting and programming
• Advanced SQL
• Airflow\any other orchestration tool
• CI\CD
• Git or any other VCS knowledge
• Data modelling\Data warehouse knowledge

• NoSQL and relational Databases
• ETL and ELT process
• Data mesh architecture
• Kafka or any other messaging queue experience
• Spark streaming (lambda and kappa architecture)
• Spark optimisation