Senior Cloud Data Engineer

Your key responsibilities

• Design and Implement Data pipelines: Develop, construct, test and maintain data architectures, including large-scale processing systems and data platforms, using GCP services like Dataproc, BigQuery, Composer.
• Data Integration: Integrate data from various sources, ensuring data quality and consistency, and optimize data integration workflows to streamline data ingestion and processing
• ETL Processes: Develop and manage frameworks for ETL processes to ingest, transform and store
• Security and Compliance: Ensure data solutions comply with industry standards and best practices for data security
• Monitor, troubleshoot, and optimize the performance of data processing systems
• Implement data quality checks and validation as a reusable framework on multi-tenant platform
• Stay current with new GCP services and features, and propose improvements to existing data solutions
• Promote and implement best practices for data engineering

.

Your skills and experience

• Must have good implementation experience on GCP Data Storage and Processing Services such as Big Query, Dataproc, Cloud Composer
• IaC experience such as Terraform is a plus
• Strong Experience with Apache spark and data processing workflows
• Experience in orchestrating end-to-end data pipelines with tools like Cloud Composer
• Experience in managing complex and reusable pipelines
• Experience in building real-time and batch ingestion and pipelines on GCP
• Experience in Multi-tenanted Data Platforms
• Automate data processing tasks using scripting languages such as Python
• Stay-up-to-date with latest GCP services and features and evaluate their potential use in Organization data infrastructure