Design and Implement Data pipelines: Develop, construct, test and maintain data architectures, including large-scale processing systems and data platforms, using GCP services like Dataproc, BigQuery, Composer.
Data Integration: Integrate data from various sources, ensuring data quality and consistency, and optimize data integration workflows to streamline data ingestion and processing
ETL Processes: Develop and manage frameworks for ETL processes to ingest, transform and store
Security and Compliance: Ensure data solutions comply with industry standards and best practices for data security
Monitor, troubleshoot, and optimize the performance of data processing systems
Implement data quality checks and validation as a reusable framework on multi-tenant platform
Stay current with new GCP services and features, and propose improvements to existing data solutions
Promote and implement best practices for data engineering
.
Your skills and experience
Must have good implementation experience on GCP Data Storage and Processing Services such as Big Query, Dataproc, Cloud Composer
IaC experience such as Terraform is a plus
Strong Experience with Apache spark and data processing workflows
Experience in orchestrating end-to-end data pipelines with tools like Cloud Composer
Experience in managing complex and reusable pipelines
Experience in building real-time and batch ingestion and pipelines on GCP
Experience in Multi-tenanted Data Platforms
Automate data processing tasks using scripting languages such as Python
Stay-up-to-date with latest GCP services and features and evaluate their potential use in Organization data infrastructure
How we ll support you
Training and development to help you excel in your career
Coaching and support from experts in your team
A culture of continuous learning to aid progression
A range of flexible benefits that you can tailor to suit your needs