Lead Data Engineer - SQL/Python (5-7 yrs)

Job Description :

Technical delivery lead on platform and project builds for building and maintaining optimised and highly available data pipelines that facilitate data management and data analytics solutions.

The role will drive the development of data processing frameworks and data architectural approaches that scale to handle the business's growing demands.

This hands-on role requires technical expertise to deliver data and analytical services, including data ingestion, transformation, storage, and reporting and strives to continuously develop new and improved data engineering capabilities.

Key Accountabilities :

- Design data solutions to meet business, technical and user requirements.

- This includes building modern data pipelines that meet functional/non-functional business requirements and provide end-to-end data solutions.

- Participate in proofs-of-concept on platform innovation, and effectively transition and scale those concepts into production at scale through, engineering, deployment and commercialisation.

- Contribute to ongoing management of the platform, including platform enhancements, capacity management, performance monitoring, troubleshooting and resolution of technical issues.

- Identify, design, and implement process improvements: automating manual processes, optimising data delivery, re-designing infrastructure for greater scalability.

- Technical ownership for data and data pipelines to ensure compliance with data standards, architectural standards, and achievement of business requirements.

- Build a strong understanding of platform architecture and the best way to leverage it to achieve effective outcomes for a given project.

- Support compliance with data, information and security management requirements.

Tech stack :

- A strong understanding and experience across modern data stacks airbyte/melanto/AWS DMS/AWS App flow, Apache Kafka/Aws Kinesis, Postgres / AWS redshift, DuckDB, DBT, Trino, / Athena, Clickhouse, cubeJs, metabase / looker / AWS Quicksight.

- AWS platform networking, IAM, security, EKS.

- Understand and read Elixir, Python, Typescript.

- Proficient in Spark, SQL, Python, EKS / Kubernetes / Terraform