Kafka Administrator - Google Cloud Platform

Responsibilities :

- Lead the migration of Kafka workloads from AWS to GCP utilising Kubernetes and GCP-native services.

- Architect, implement, and manage a scalable Kafka infrastructure on GKE.

- Develop and maintain comprehensive performance metrics dashboards for the Kafka ecosystem.

- Collaborate with cross-functional teams to optimise Kafka API integration.

- Conduct in-depth assessments of existing AWS Kafka deployments for migration planning.

- Develop and execute a robust migration strategy encompassing data migration and consumer repointing.

- Optimise Kafka performance and scalability for high-throughput data processing.

- Implement robust monitoring and alerting for system health and performance.

- Guarantee data integrity and security throughout the migration process.

- Collaborate with engineering and data teams for seamless integration and optimal performance.

- Design and implement a multi-tenant GKE environment to accommodate diverse workloads.

- Deploy and configure Kafka Strimzi on the underlying infrastructure.

- Identify and implement efficient data ingestion pipelines (e.g., Kafka Connect, Dataflow).

- Conduct thorough fault simulation testing to mitigate potential risks.

- Stay updated on Kafka developments and GCP services to recommend and implement improvements.

Requirements :

- Bachelor's degree in Computer Science, Engineering, or related field (Master's degree preferred).

- Proven hands on experience as a Kafka Lead or similar role, with a minimum of 5 years of experience.

- Deep knowledge of Kafka internals and ecosystem, including Kafka Connect, Kafka Streams, and KSQL.

- In-depth knowledge of Apache Kafka architecture, internals, and ecosystem components.

- Proficiency in scripting and automation for Kafka management and migration.

- Hands-on experience with Kafka administration, including cluster setup, configuration, and tuning.

- Proficiency in Kafka APIs, including Producer, Consumer, Streams, and Connect.

- Strong programming skills in Java, Scala, or Python.

- Experience with Kafka monitoring and management tools such as Confluent Control Center, Kafka Manager, or similar.

- Solid understanding of distributed systems, data pipelines, and stream processing.

- Experience leading migration projects to Google Cloud Platform (GCP), including migrating Kafka workloads.

- Familiarity with GCP services such as Google Kubernetes Engine (GKE), Google Cloud Storage, Google Cloud Pub/Sub, and BigQuery.

- Excellent communication and collaboration skills.

- Ability to work independently and manage multiple tasks in a fast-paced environment.

- Knowledge of containerization technologies such as Docker and Kubernetes.

- Familiarity with related technologies such as Apache ZooKeeper, Apache Flink, or Apache Spark.

- Working knowledge of hands-on experience with Strimzi Kafka

Nice to Have :

- Kafka certification (e.g., Confluent Certified Developer, Confluent Certified Operator).

- Google Cloud certification (e.g., Google Cloud Certified - Professional Cloud Architect, Google Cloud Certified - Professional Data Engineer).

- Additional experience with other cloud platforms such as AWS or Azure.