Data Engineer - Google Cloud Platform (5-9 yrs)

Job Description :

For Data Engineering :

- Good Experience or strong understanding of GCP (Big query)

- Good Experience on Bigtable, Pub/Sub, Data proc, Data Flow

- Good Experience on SQL, Python/Airflow

Roles & Responsibilities :

- Design, build, and maintain robust data pipelines using Google Cloud Platform (GCP) tools, including Dataflow and Dataproc, to ensure seamless data ingestion and transformation.

Data Modeling and Architecture :

- Create and maintain efficient data models in BigQuery and Bigtable, optimizing for performance and scalability to support analytical workloads.

Real-time Data Processing :

- Implement real-time data processing solutions using Pub/Sub, ensuring timely availability of data for analytics and reporting.

SQL Development :

- Write and optimize complex SQL queries for data extraction, transformation, and loading (ETL) processes, ensuring data accuracy and integrity.

Python and Workflow Management :

- Develop and manage workflows using Python and Apache Airflow, orchestrating various data processes and ensuring smooth execution of data tasks.

Data Quality Assurance :

- Monitor and validate data pipelines to ensure high data quality and reliability, implementing data quality checks and automated testing frameworks.

Collaboration and Communication :

- Work closely with data analysts, data scientists, and other stakeholders to understand data requirements and provide necessary data support.

Performance Tuning :

- Analyze and optimize the performance of data pipelines and queries, recommending improvements as necessary to enhance efficiency.

Documentation and Best Practices :

- Document data processes, architecture, and best practices to ensure knowledge sharing and maintainability across the team.

Cloud Infrastructure Management :

- Manage and monitor GCP resources, ensuring cost-effective utilization and adherence to security best practices.

Continuous Learning and Improvement :

- Stay updated with the latest trends and technologies in data engineering and GCP, applying new knowledge to improve data solutions.

Troubleshooting and Support :

- Provide technical support and troubleshooting for data-related issues, identifying root causes and implementing solutions.