Kloud9 - Technical Architect - Data Engineering (11-20 yrs)

We are seeking a highly skilled Data Engineering Technical Architect with deep expertise in Google Cloud Platform (GCP), Apache Spark, and SAP S/4HANA (Data ingestion) to architect, design and implement scalable, high-performance data lake solutions.

The ideal candidate will have extensive experience in building data ingestion pipelines, managing big data processing using Apache Spark, and integrating SAP S/4HANA with GCP cloud infrastructure.

Key Requirements :

- Over 10 years of professional experience in data engineering, specializing in implementing large-scale enterprise Data Engineering projects with the latest technologies.

- Over 5 years of hands-on experience in GCP technologies and over 3 years of lead experience.

- Design and implement end-to-end data architectures leveraging GCP services (e.g., Big Query, Cloud Storage, Dataflow, Pub/Sub, Cloud Composer) for large-scale data ingestion and processing.

- Build and optimize large-scale data pipelines using Apache Spark on GCP (via Dataproc or other Spark services). Ensure high performance and scalability in Spark-based data processing workloads.

- Lead the integration of SAP S/4HANA data with GCP for real-time and batch data processing.

- Manage data extraction, transformation, and loading (ETL) processes from SAP S/4HANA into cloud storage and data lakes.

- Develop and manage scalable data ingestion pipelines for structured and unstructured data using tools like Cloud Dataflow, Cloud Pub/Sub, and Apache Spark.

- Provide architectural guidance for designing secure, scalable, and efficient data solutions on the Google Cloud Platform, integrating with on-premise/cloud systems like SAP S/4HANA.

- Implement both real-time streaming and batch processing pipelines using Apache Spark, Dataflow, and other GCP services to meet business requirements.

- Implement data governance, access controls, and security best practices to ensure the integrity, confidentiality, and compliance of data across systems.

- Collaborate with business stakeholders, data scientists, and engineering teams to define data requirements, ensuring the architecture aligns with business goals.

- Optimize Apache Spark jobs for performance, scalability, and cost-efficiency, ensuring that the architecture can handle growing data volumes.

- Provide technical leadership to the data engineering team, mentoring junior engineers in data architecture, Apache Spark development, and GCP best practices.