GCP Data Engineer

About the Role:

We are seeking a highly skilled GCP Data Engineer (Minimum 3yrs of relevant experience) to join our team and play a crucial role in managing our GCP Data Lake. This role requires a strong technical background, a passion for data engineering, and the ability to collaborate effectively with both internal teams and external partners.

Key Responsibilities:

• Data Ingestion:
• Design and implement robust data ingestion pipelines using Cloud Fusion to extract data from SAP systems.
• Develop and maintain Apache Airflow workflows to orchestrate data ingestion from various sources, including APIs and Cloud Storage buckets.
• Ensure data quality, consistency, and timely delivery through efficient data ingestion processes.
• Data Processing and Transformation:
• Utilize Big Query to process and transform raw data into a curated format.
• Develop and optimize data pipelines to handle large datasets efficiently.
• Implement data quality checks and validation procedures to maintain data integrity.
• Data Modeling:
• Design and implement effective data models, including dimensional and normalized models, to support business intelligence and analytics.
• Create data marts and data warehouses to facilitate data analysis and reporting.
• Data Pipeline Optimization:
• Monitor and optimize data pipelines to improve performance and reduce costs.
• Identify and resolve performance bottlenecks and errors.
• Implement best practices for data pipeline design and maintenance.
• Collaboration:
• Work closely with our cloud operations partner to ensure smooth and efficient data operations and adherence to SLAs.
• Collaborate with BI team to ensure report accuracy and reliability by solving issues in data issues.
• Collaborate on incident resolution and problem-solving activities.
• Coordinate with the partner to align on data engineering best practices and standards.
• Problem-Solving and Troubleshooting:
• Troubleshoot and resolve complex data pipeline issues.
• Analyze system logs and performance metrics to identify root causes of problems.
• Implement proactive monitoring and alerting to prevent issues.

Required Skills and Qualifications(Min 3 Years)

• Strong proficiency in SQL, Python, PySpark, and Big Query.
• Proven experience in designing and implementing data pipelines using Apache Airflow.
• In-depth knowledge of GCP data engineering technologies, including Cloud Storage,Data Fusion, and Composer.
• Experience with data modeling techniques and best practices.
• Strong problem-solving and analytical skills.
• Excellent communication and collaboration skills.
• GCP Data Engineer certification preferred.

Preferred Skills and Qualifications:

• Experience with machine learning and data science concepts.
• Knowledge of data security and privacy best practices.
• Experience with cloud cost optimization and management.