Lead - GCP & Data Engineer

8-9+ years of Total Experience
6+ years of direct hands-on experience with the GCP BigQuery, Apache Airflow/ GCP Cloud Composer
Extensive experience with PySpark: Strong expertise in developing and optimizing large-scale data processing solutions using Apache Spark with Python.
Big Data Ecosystem: Proficiency with Hadoop, HDFS, Hive, YARN, and other big data technologies.

Detailed JD:

9+ years of direct hands-on experience with the GCP BigQuery, Apache Airflow/ GCP Cloud Composer
Extensive experience with PySpark: Strong expertise in developing and optimizing large-scale data processing solutions using Apache Spark with Python.
Big Data Ecosystem: Proficiency with Hadoop, HDFS, Hive, YARN, and other big data technologies.
Experience with data architecture: Hands-on experience designing and managing scalable data architectures, including data lakes and data warehouses.
Cloud platforms: Deep knowledge of GoogleCloud Platform, particularly services like Apache Airflow,Cloud Composer, BigQuery
ETL pipeline development: Expertise in designing, building, and managing ETL processes with PySpark.
SQL expertise: Advanced knowledge of Spark SQL for data transformations and optimization of complex queries.
Experience in distributed systems: In-depth understanding of distributed computing principles, fault tolerance, and data partitioning.
DevOps practices: Experience with containerization (Docker), orchestration (Kubernetes), and managing Spark workloads in containerized environments.
Performance tuning: Proven ability to optimize Spark jobs, including memory management, partitioning strategies, and cluster resource management.
Version control and CI/CD: Proficiency with Git, Jenkins, CI/CD tools to automate data pipeline deployment and management.
Team leadership: Experience leading teams, mentoring junior engineers, and fostering a collaborative environment.

Skills:

• Analytical Thinking
• Google Cloud Platform
• Python
• Data Warehousing
• Hadoop Hive