Data Engineer - Google Cloud Platform (4-6 yrs)

Company : Forefront(Payroll)

Client : Deloitte(C2H-Contract to Hire)

Job Title : GCP Data Engineer

Experience Level : 4-6 Years

Shift Timings : 2 PM to 11 PM

Location : Hyderabad, Bangalore, Chennai, Mumbai, Pune, Kolkata, Gurgaon

Mode of work : Hybrid (2-3 days in the office per week)

Position Overview :

We are looking for a skilled GCP Data Engineer with 4-6 years of experience to join our team. This role involves designing, developing, and optimizing data pipelines on the Google Cloud Platform (GCP) to support our data analytics and business intelligence needs. The ideal candidate will have experience in SQL, PySpark, Python, Hadoop, and GCP tools like BigQuery, DataProc, and Dataflow, with a focus on ensuring data quality, performance, and reliability.

Key Responsibilities :

1. Data Pipeline Development :

- Design, build, and manage scalable data pipelines using SQL, PySpark, and Python.

- Utilize industry best practices to handle data in a high-availability and distributed environment.

2. Data Migration :

- Leverage Hadoop to support the migration and management of large datasets.

- Facilitate seamless data transitions while ensuring minimal disruption to data flows and quality.

3. Data Warehousing and Analytics :

- Work with Google BigQuery for data warehousing, analytics, and reporting purposes.

- Build solutions that support efficient query processing and business intelligence.

4. GCP Data Tools :

- Utilize GCP's DataProc and Dataflow for data processing, transformation, and real-time data integration.

- Maintain the security and performance standards of all GCP components.

5. Data Integration and Quality Assurance :

- Integrate data from various internal and external sources, ensuring data quality, integrity, and consistency.

- Regularly monitor data pipelines to maintain high data quality and accuracy.

6. Performance Optimization :

- Tune and optimize data processing workflows to maximize efficiency in Spark and other frameworks.

- Continuously monitor and improve pipeline performance to meet business SLAs.

7. Data Modelling :

- Design and maintain data models that align with analytical and reporting requirements.

- Collaborate with data architects to ensure models are efficient and scalable.

8. Stakeholder Collaboration :

- Work closely with data scientists, analysts, and business teams to understand data needs and deliver solutions that meet business objectives.

9. Documentation and Knowledge Sharing :

- Create and maintain comprehensive documentation for all data processes, pipelines, and architecture.

- Share knowledge and best practices across the team to maintain high data engineering standards.

10. Troubleshooting and Support :

- Identify and resolve issues related to data pipeline failures, data quality, and processing performance.

- Act as a point of contact for data-related troubleshooting and issue resolution to ensure reliability.

Key Skills and Requirements :

- Technical Skills : SQL, PySpark, Python, Hadoop, Google BigQuery, GCP DataProc, and Dataflow.

- Experience Level : 4-6 years in data engineering with a focus on data pipeline development, optimization, and data integration.

- Analytical Skills : Ability to develop and optimize complex data workflows and understand large data sets.

- Problem-Solving : Strong troubleshooting skills with experience resolving data quality and reliability issues.

- Soft Skills : Strong collaboration, communication, and documentation skills.

- Availability : Hybrid work model, requiring 2-3 days in the office per week.