Staff Software Engineer - Cloud Platform - 2

Responsibilities: 

• Design and implement cloud infrastructure using IaC Terraform etc.
• Automate provisioning and management of Dataproc clusters, Dataflow jobs, and other GCP resources
• Build and maintain CD pipelines for deploying data pipelines, streaming applications, and cloud infrastructure.
• Integrate tools like GitLab CI/CD, or Cloud Build for automated testing and deployment.
• Deploy and manage real-time and batch data pipelines using Dataflow, DataStream, and Apache Flink.
• Ensure seamless integration of data pipelines with other GCP services like Big Query, Cloud Storage, and Kafka or Pub/Sub.
• Implement monitoring and alerting solutions using Cloud Monitoring, Cloud Logging, and Prometheus.
• Monitor performance, reliability, and cost of Dataproc clusters, Dataflow jobs, and streaming applications.
• Optimize cloud infrastructure and data pipelines for performance, scalability, and cost-efficiency.
• Implement security best practices for GCP resources, including IAM policies, encryption, and network security.
• Ensure Observability is an integral part of the infrastructure platforms and provides adequate visibility about their health, utilization, and cost. 
• Collaborate extensively with cross functional teams to understand their requirements; educate them through documentation/trainings and improve the adoption of the platforms/tools.  

Education and Experience:

•  7+ years of overall experience in DevOps -cloud engineering, or data engineering.
•  3+ years of experience in DevOps, cloud engineering, or data engineering.
• Proficiency in Google Cloud Platform (GCP) services, including Dataflow, DataStream, Dataproc, Big Query, and Cloud Storage.
• Strong experience with Apache Spark and Apache Flink for distributed data processing.
• Knowledge of real-time data streaming technologies (e.g., Apache Kafka, Pub/Sub).
• Familiarity with data orchestration tools like Apache Airflow or Cloud Composer.
• Expertise in Infrastructure as Code (IaC) tools like Terraform or Cloud Deployment Manager.
• Experience with CI/CD tools like Jenkins, GitLab CI/CD, or Cloud Build.
• Knowledge of containerization and orchestration tools like Docker and Kubernetes.
• Strong scripting skills for automation (e.g., Bash, Python).
• Experience with monitoring tools like Cloud Monitoring, Prometheus, and Grafana.
• Familiarity with logging tools like Cloud Logging or ELK Stack.
• Strong problem-solving and analytical skills.
• Excellent communication and collaboration abilities.
• Ability to work in a fast-paced, agile environment.