Kafka Engineer - Configuration Management (6-8 yrs)

Job Description :

Role : Kafka Engineer

- 6+ years of experience.

- Based in Chennai, Tamil Nadu.

- Immediately available or maximum 30 days notice period.

- Atos India, preferably Chennai (for all 4 resources).

- Expectation is for the resource to visit SCB Chennai office and work with squad leads.

Joining Date : Preferably from 1st Dec 2024.

Skill Set & Experience Mandatory :

Kafka Expertise :

- In-depth knowledge of Kafka architecture, concepts, and components.

- Proficiency in Kafka topic creation, partitioning, replication, and configuration management.

Deployment Experience :

- Hands-on experience deploying Kafka clusters in various environments.

- Familiarity with cluster sizing, configuration, and security considerations.

Good-to-Have :

- Knowledge on observability stack including Prometheus, ELK, Traces, Logs, Metrices, Alerts.

Kafka Monitoring and Management :

- Familiarity with Kafka monitoring tools (e., Confluent Control Center, Prometheus) to track cluster health, performance metrics, and alerts.

Scripting and Automation :

- Ability to write scripts (e. Bash, Python) for automating routine tasks such as deployment, configuration management, and monitoring.

- Experience in ELK Stack configuration and development, comprising of Elastic Search, Logstash & Kibana.

Qualifications :

- Bachelor's degree in Computer Science, Information Technology, or related field (Master's degree preferred).

- Min 6+ years of professional experience as a Kafka Engineer or similar role.

- Strong understanding of distributed systems and event-driven architectures.

- Excellent problem-solving skills and the ability to troubleshoot complex issues.

- Strong communication and interpersonal skills to collaborate effectively with cross-functional teams.

- Kafka certification(s) would be favorable.

Core responsibilities :

- 5 years or more experience with solid hands-on technologies (Apache Kafka, Zookeeper, Connect, Ansible, Docker, Podman, Kubernetes, Java).

- Hands-on experience with installing/managing/supporting 3 or more Kafka cluster enterprise wide.

- Hands-on experience with writing Kafka connect, adapters for integration with Elastic Search, Grafana tech stack.

- Experience with AWS Cloud, Terraform automation and Overall Cloud Native architecture knowledge would be favorable.

Kafka Topic Management :

- Design, create, and manage Kafka topics based on business requirements.

- Ensure proper partitioning, replication, and configuration settings for optimal performance and data distribution.

Kafka Deployment :

- Deploy Kafka clusters, brokers, and related components in various environments (development, testing, production).

- Manage cluster configurations, scaling, and security settings.

Kafka Infrastructure Maintenance :

- Perform routine maintenance tasks such as Kafka version upgrades, automated patch management, and security enhancements.

Performance Optimization :

- Knowledge on tuning Kafka for performance.

- Collaborate with development teams to tune Kafka configurations and parameters for optimal throughput, latency, and resource utilization.

- Identify and address bottlenecks in the Kafka ecosystem.

Capacity Planning :

- Analyze current and projected data volumes to plan for Kafka cluster capacity.

- Scale up or out as needed to accommodate growing data demands.

Integration and Middleware :

- Integrate Kafka with other systems, databases, and middleware components to facilitate seamless data exchange and event-driven architectures.

Cross-Functional Collaboration :

- Collaborate with cross-functional teams including developers, data engineers, DevOps, and QA to understand requirements and provide Kafka expertise.

Other responsibilities :

- Mentor other team members around the development of technical skills and practices.

- Participates in backlog discussions, amigos meetings, and estimation sessions.

- Produce clear and comprehensive written documentation for automation solutions, including user guides and technical specifications.

- Peer reviews team members scripts / code for quality check (QA), before sharing with customer.

- Any other assignments as instructed by superior as and when required.

- Making good and timely decisions that keep the organization moving forward.

- Developing people to meet both their career goals and the organization's goals.

- Painting a compelling picture of the vision and strategy that motivates others to action.

- Providing direction, delegating, and removing obstacles to get work done.

- Taking on new opportunities and tough challenges with a sense of urgency, energy and enthusiasm.

- Building partnerships and working collaboratively with others to meet shared objectives.

- Stepping up to address difficult issues and saying what needs to be said.

- Building strong customer relationships and delivering customer-centric solutions.