Kafka Administrator

We are seeking a skilled Kafka Engineer to join our team and contribute to the design, development,administration, and optimization of our Kafka-based Google Cloud Platform (GCP) data pipelines.

The ideal candidate will have a strong understanding of Kafka architecture, internals, and best practices. You will be responsible for developing and implementing Kafka-based solutions, as well as administering and maintaining the Kafka cluster to support our data processing and messaging needs.

Responsibilities:

• Design, develop, and implement Kafka-based data pipelines and applications. Write efficient and scalable Kafka producers and consumers.
• Manage and maintain Kafka clusters, including installation, configuration, and monitoring.
• Perform cluster upgrades, troubleshooting, and performance tuning.
• Hands-on collaboration with data engineers and data scientists to understand data
• requirements and build effective Kafka solutions. Ensure data quality and integrity in Kafka
• pipelines.
• Optimise Kafka performance and throughput. Identify and implement performance
• improvements.
• Troubleshoot and resolve Kafka-related issues, including performance bottlenecks, data loss,
• and consumer failures.
• Implement monitoring and alerting mechanisms to ensure Kafka cluster health and
• performance.
• Implement security best practices for Kafka, including authentication, authorisation, and
• encryption.

Requirements:

• Bachelors degree in computer science, Engineering, or related field (Masters degree

preferred).

• Proven hands-on experience as a Kafka Engineer or similar role, with a minimum of 2+ years of

experience.

• Deep knowledge of Kafka internals and ecosystem, including Kafka Connect, Kafka Streams, and KSQL

• In-depth knowledge of cloud platforms GCP Must and AWS, Azure optional.

• Experience with containerization technologies (Docker, Kubernetes).

• Familiarity with data processing frameworks (Spark, Flink).

• Strong understanding of Kafka architecture, internals, and APIs.

• Proficiency in Java, Scala, or Python programming languages.

• Experience with Kafka producers, consumers, and connectors.

• Knowledge of data streaming and messaging concepts.

• Experience with data ingestion and transformation pipelines.

• Ability to write clean, efficient, and maintainable code.

• Strong problem-solving and troubleshooting skills.

• Excellent communication and collaboration skills.

• Experience with Kafka administration, including cluster setup, configuration, and monitor

Nice to Have:

• Kafka certification (e.g., Confluent Certified Developer, Confluent Certified Operator).

• Google Cloud certification (e.g. Google Cloud Certified - Professional Data Engineer).

• Additional experience with other cloud platforms such as AWS or Azure.