Google Cloud Platform Data Engineer (5-8 yrs)

Job Summary :

We are seeking a highly motivated and experienced GCP Data Engineer to join our dynamic team. In this role, you will be responsible for designing, developing, and maintaining scalable and efficient data pipelines and analytics solutions on Google Cloud Platform (GCP). You will leverage your expertise in BigQuery, Dataflow, Cloud Composer, and DataProc to build robust and reliable data infrastructure that supports our business intelligence and analytical needs. The ideal candidate will be a strong problem-solver, possess excellent communication skills, and thrive in a fast-paced, collaborative environment.

Responsibilities :

- Design, develop, and deploy scalable and efficient data pipelines using Google Cloud Dataflow for both batch and stream processing.

- Orchestrate and automate data workflows using Cloud Composer (Apache Airflow) for scheduling and monitoring data pipelines.

- Develop and maintain data ingestion processes from various sources into BigQuery.

- Optimize data pipelines for performance, reliability, and cost-effectiveness.

- Design and implement efficient data models and schemas in BigQuery for analytical workloads.

- Write complex SQL queries for data analysis and transformation.

- Optimize BigQuery performance for large datasets and complex queries.

- Manage BigQuery resources and ensure data security and access control.

- Utilize Terraform to provision and manage GCP infrastructure as code.

- Monitor and troubleshoot GCP services, including Dataflow, BigQuery, Cloud Composer, and DataProc.

- Implement and maintain data security and governance policies on GCP.

- Work with DataProc for spark processing when needed.

- Develop and maintain Python scripts for data processing, automation, and API integration.

- Utilize Python libraries (e.g., Pandas, NumPy, Apache Beam SDK) for data manipulation and analysis.

- Write clean, efficient, and well-documented code.

- Collaborate with data scientists, analysts, and other engineers to understand data requirements and deliver effective solutions.

- Communicate technical concepts and solutions clearly and concisely to both technical and non-technical stakeholders.

- Participate in code reviews and contribute to team knowledge sharing.

- Work within an agile environment.

- Implement data quality checks and validation processes to ensure data accuracy and consistency.

- Adhere to data governance policies and best practices.

- Contribute to the development and maintenance of data documentation.

Key Requirements :

- Experience with Apache Spark and distributed data processing.

- Knowledge of data warehousing and data lake concepts.

- Experience with CI/CD pipelines for data engineering workflows.

- Familiarity with data visualization tools (e.g., Looker Studio).

- Experience with other cloud platforms such as AWS or Azure.

- Understanding of data security best practices.

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- 3+ years of professional experience in GCP data engineering or related fields.

- Proven experience in designing and implementing data pipelines and analytics solutions on GCP.

- Strong problem-solving and analytical skills.

- Excellent communication and collaboration skills.

- Ability to work independently and as part of a team.