Cloud Data Engineer - Google Cloud Platform (5-8 yrs)

Job Title : Cloud Data Engineer

Location : Chennai

Experience : 5+ Years (Adjust as needed)

Position Type : Full-Time / Contract (Specify)

Budget : [Specify Budget]

Position Overview :

We are seeking a Cloud Data Engineer with extensive experience in Google Cloud Platform (GCP) and a strong background in data engineering, cloud infrastructure, and software development. The ideal candidate will play a critical role in designing, developing, and maintaining cloud-based data solutions, leveraging GCP services, automation tools, and data engineering best practices.

Key Responsibilities :

- Design and manage data pipelines in GCP using BigQuery, DataFlow, Dataproc, and DataFusion for efficient batch and real-time data processing.

- Develop and maintain infrastructure as code (IaC) using Terraform for provisioning and managing cloud resources.

- Implement event-driven data pipelines using Confluent Kafka and ensure reliable data transfer between systems.

- Develop and optimize API integrations for seamless data communication between different platforms.

- Write efficient, maintainable Python, PySpark, and shell scripts for data processing, automation, and orchestration tasks.

- Collaborate within an Agile development framework to deliver robust, scalable data solutions on time.

- Troubleshoot, monitor, and optimize data pipeline performance, ensuring data accuracy and system reliability.

- Ensure security and cost optimization in cloud infrastructure and data solutions.

Required Skills and Qualifications :

Technical Skills :

Google Cloud Platform (GCP) :

- BigQuery, DataFlow, Dataproc, DataPlex, DataFusion

- Cloud SQL, Cloud Storage, Terraform, Tekton

Programming & Scripting :

- Proficiency in Python, PySpark, and Shell Scripting

Data Engineering :

- Experience with data transfer utilities, Confluent Kafka, and API Development

Automation & Orchestration :

- Experience with tools like Airflow, Terraform, and Tekton

Version Control :

- Strong understanding of Git or other version control systems

Preferred Skills :

- Advanced knowledge in Python for automation and data transformation tasks.

- Familiarity with API development and consumption, particularly in cloud environments.

- Experience in optimizing cloud storage solutions for cost and performance efficiency.

Experience Required :

- 5+ years of experience in cloud-based data engineering and software development.

- 2+ years of hands-on experience with GCP services.

- 4+ years in Python and PySpark development.

- 2+ years in Confluent Kafka and API Development.

- 2+ years of experience in Agile environments.