Subject Matter Expert - Confluent Kafka (10-15 yrs)

Responsibilities :

- Design and architect robust and scalable Kafka-based solutions for real-time data streaming, event processing, and analytics.

- Provide expertise in Kafka's core concepts such as topics, partitions, producers, consumers, consumer groups, and replication.

- Ensure high availability, fault tolerance, and performance of Kafka clusters.

- Lead the integration of Kafka with other platforms, services, and databases, ensuring seamless data flow across systems.

- Collaborate with software engineers to design and implement Kafka producers, consumers, and stream processing pipelines.

- Analyze and optimize Kafka cluster performance, including memory, disk, network utilization, and throughput.

- Troubleshoot and resolve issues related to Kafka performance, reliability, and scalability.

- Act as the go-to expert for troubleshooting Kafka-related issues in production and non-production environments.

- Provide guidance on monitoring Kafka clusters, implementing alerts, and using tools such as Kafka

Manager, Confluent Control Center, and others.

- Establish and enforce best practices for deploying, managing, and scaling Kafka clusters.

- Create and maintain documentation for Kafka deployment, monitoring, configuration, and

troubleshooting procedures.

- Mentor team members on Kafka-related topics, ensuring that the team stays up-to-date with best practices and new Kafka features.

- Conduct knowledge-sharing sessions and workshops to promote Kafka adoption within the organization.

- Implement security measures for Kafka clusters, including authentication, authorization, encryption, and access control.

- Ensure compliance with data privacy regulations and organizational security policies related to data streaming.