Gcp Data Engineer

Working Location : Remote (PAN INDIA) preference should be close to Tech M offices in India

Working Hours : 2pm 10pm

Joining Time : 2-3 weeks

Detailed Job Description

The ideal candidate will have extensive experience with Google Cloud Platform (GCP) services, particularly Dataflow, BigQuery, Pub/Sub, and related services. Additionally, the candidate should be proficient in provisioning infrastructure on GCP using Infrastructure as Code (IaC) with Terraform. This role will be critical in building and managing data pipelines, ensuring data quality, and optimizing data storage and retrieval processes.

Responsibilities

• Design, develop, and maintain data pipelines using GCP services such as Dataflow, BigQuery, and Pub/Sub.
• Implement and manage data warehouse solutions using BigQuery, ensuring data is stored securely and efficiently.
• Use Pub/Sub for real-time data ingestion and streaming analytics.
• Provision and manage GCP infrastructure using Terraform, ensuring best practices in IaC are followed.
• Optimize data storage and retrieval processes to enhance performance and reduce costs.
• Monitor and troubleshoot data pipeline issues, ensuring high availability and reliability of data services.
• Ensure data quality and integrity through robust testing and validation processes.
• Stay updated with the latest GCP features and best practices, integrating them into existing workflows.
• Document data workflows, infrastructure setups, and processes for future reference and knowledge sharing.

Requirements

• Proven experience as a Data Engineer with a focus on GCP services.
• Strong proficiency in GCP services such as Dataflow, BigQuery, and Pub/Sub.
• Hands-on experience with Terraform for provisioning and managing GCP infrastructure.
• Proficiency in SQL and Python for data manipulation and analysis.
• Solid understanding of data warehousing concepts and ETL processes.
• Experience with real-time data processing and streaming analytics.
• Strong problem-solving skills and attention to detail.
• Excellent communication and collaboration skills.

Preferred Qualifications

• Google Cloud Professional Data Engineer certification.
• Experience with other GCP services like Cloud Storage, Cloud Functions, and Cloud Composer.
• Familiarity with other IaC tools such as Ansible or Cloud Deployment Manager.
• Knowledge of data security and governance practices.

Google Data plex(mandatory) for enterprise data fabric- meta data management and sync, data quality, data governance, data stewardship, defining roles for data users, consumers;

Big Query, Data Flow(Apache beam with Python), pub/sub, batch & streaming data pipeline, Composer/Airflow, Data forms (good to have), Terraform(good to have), Data Stream(good to have)