Criticalriver Technologies - Kafka Lead - Google Cloud Platform (10-12 yrs)

About the Job :

We are seeking a highly skilled and motivated Kafka Engineer to join our team. The ideal candidate will have expertise in building and maintaining distributed streaming platforms, specifically leveraging Apache Kafka. As a Kafka Engineer, you will be responsible for the design, implementation, and maintenance of highly scalable and reliable Kafka systems, ensuring that data streams are processed efficiently in a variety of real-time applications.

Roles & Responsibilities :

- Design, implement, and maintain Kafka-based streaming architectures for real-time data processing.

- Develop and optimize Kafka producers, consumers, and streams to handle large-scale data volumes.

- Build highly available, fault-tolerant, and scalable Kafka systems to ensure smooth data flow.

- Design Kafka topics, partitions, and retention policies to ensure efficient and scalable data storage.

- Integrate Kafka with ETL pipelines, data lakes, databases, and third-party services.

- Collaborate with data engineers and application teams to integrate Kafka with other data services, ensuring smooth data ingestion and processing.

- Implement Kafka Connectors for various external systems like HDFS, Elasticsearch, MySQL, MongoDB, etc.

- Ensure the implementation of security best practices for Kafka environments, including authentication, authorization, and encryption.

- Work with Kerberos, SSL, SASL, and ACLs for securing Kafka clusters.

- Manage and ensure data governance, ensuring compliance with regulatory standards, including data retention policies and auditing.

- Collaborate with application developers, data engineers, and DevOps teams to ensure that Kafka solutions meet system requirements.

- Provide technical support for Kafka-related issues and troubleshoot issues in production environments.

- Mentor junior engineers and assist with best practices for Kafka usage and development.

Technical Skills & Qualifications :

Core Skills :

- In-depth experience in setting up, managing, and optimizing Apache Kafka clusters for high throughput, fault tolerance, and scalability.

- Strong knowledge of Kafka Streams, Kafka Connect, and Kafka Consumer APIs.

- Expertise in building real-time data streaming pipelines and systems using Kafka.

- Proficient in Java, Scala, or Python for building Kafka producers and consumers.

- Experience with other messaging systems like RabbitMQ, ActiveMQ, or Kinesis is a plus.

Additional Skills :

- Strong understanding of distributed systems, data consistency, partitioning, and replication.

- Knowledge of integrating Kafka with data processing and storage systems (e.g., HDFS, S3, Relational Databases, NoSQL databases).

- Proficiency in monitoring Kafka with tools such as Prometheus, Grafana, Zookeeper, and Confluent Control Center.

- Experience with Kafka on cloud platforms such as AWS MSK, Azure Event Hubs, or Google Cloud Pub/Sub.

- Experience in securing Kafka clusters using SSL/TLS encryption, Kerberos, SASL, and ACLs

Why Join Us ?

At CriticalRiver, you'll have the opportunity to lead and shape a key practice within our organization, work with high-caliber clients, and make a meaningful impact on their digital transformation journey.

We are committed to fostering a culture of innovation, growth, and collaboration where your ideas and leadership will thrive.

CriticalRiver Inc.is an employee-owned technology consulting and IT services firm that empowers its workforce to have a direct stake in the company's growth and success.

This ownership model fosters a collaborative and committed culture where employees are motivated to deliver high-quality solutions and services for clients.

By giving its team members a shared interest in the company's achievements, CriticalRiver enhances innovation, accountability, and a strong sense of community across all levels of the organization.