GCP Data Engineer

Role & responsibilities

• Build and maintain scalable, automated ETL/ELT pipelines using BigQuery, Cloud Dataflow, Dataproc, Pub/Sub, and other GCP services.
• Develop workflows for ingesting, processing, and transforming structured, semi-structured, and unstructured data from diverse sources.
• Implement complex data workflows using orchestration tools like Apache Airflow and Cloud Composer.
• Integrate external data systems with GCP services through APIs, Pub/Sub, and Cloud Functions.
• Design and optimize cloud-based data architectures with a focus on performance, scalability, and cost efficiency.
• Utilize GCP services such as Cloud Storage, Cloud SQL, and BigQuery for efficient data storage and retrieval.
• Perform data modeling and query optimization for BigQuery.
• Optimize data pipelines for real-time and batch processing to ensure low-latency and high-throughput operations.
• Implement GCP best practices for data security, encryption, and Identity and Access Management (IAM).
• Enforce data governance policies to maintain data quality, lineage, and compliance with regulatory standards.
• Use tools like GCP Operations Suite (Stackdriver) for monitoring and troubleshooting.
• Perform root cause analysis and optimize data pipelines for scalability and reliability.

Preferred candidate profile

• Proficient in BigQuery, Cloud Dataflow, Dataproc, Pub/Sub, Cloud Functions, Cloud SQL, and Cloud Storage.
• Expertise in Python and SQL for data processing, querying, and automation.
• Experience with Apache Beam for data processing.
• Strong experience in designing and implementing ETL/ELT workflows using GCP services and orchestration tools.
• Deep understanding of cloud-native architectures and optimization strategies within GCP.
• Knowledge of GCP IAM, VPC, and encryption best practices.
• Experience in implementing data governance frameworks.

Nice-To-Have

• Google Cloud Certified Professional Data Engineer or similar certifications.
• Experience with Apache Kafka, Apache Spark, Hadoop, or Kubernetes.
• Familiarity with Vertex AI or other GCP ML tools.
• Knowledge of data lineage, quality monitoring, and governance practices.