Kafka Architect

Key Responsibilities

• Architectural Design: Develop and maintain Kafka-based streaming architecture to support real-time data pipelines and event-driven applications.
• Platform Oversight: Manage the Kafka ecosystem, including Kafka brokers, Zookeeper, Kafka Connect, Kafka Streams, and Schema Registry.
• Data Flow Management: Design and implement data flows, ensuring scalability, fault tolerance, and high availability.
• Integration: Collaborate with data engineering, DevOps, and software development teams to integrate Kafka into broader data architectures.
• Optimization: Monitor and optimize Kafka performance, resource utilization, and throughput for a wide variety of use cases.
• Security: Implement best practices for Kafka security, including encryption, authentication, and authorization.
• Documentation: Prepare detailed documentation for Kafka processes, architecture, and guidelines for usage.
• Innovation: Research and implement emerging technologies to enhance the Kafka ecosystem.
• Troubleshooting: Diagnose and resolve production issues related to Kafka, including root cause analysis and remediation.

Required Skills

• Proven experience as a Kafka Architect or similar role, designing and implementing large-scale Kafka solutions.
• In-depth knowledge of Kafka components, including Kafka Streams, Kafka Connect, and Schema Registry.
• Strong programming skills in Java, Scala, or Python.
• Experience with distributed systems, event-driven architecture, and stream processing.
• Familiarity with related technologies such as Spark, Flink, Hadoop, and Elasticsearch.
• Knowledge of DevOps practices, including CI/CD pipelines, monitoring tools, and containerization (Docker, Kubernetes).
• Proficiency with cloud platforms (GCP) and their Kafka-managed services (MSK, Confluent Cloud).
• Strong problem-solving and analytical skills.

Education

• Bachelors or Masters degree in Computer Science, Information Technology, or related field.