Kafka Engineer

• Help facilitate the implementation of Confluent Kafka streaming and enhance the middleware administration
• Responsible for setting up Kafka brokers, Kafka Mirror Makers, and Kafka Zookeeper on hosts in collaboration with the Infrastructure team
• Design, build and maintains Kafka topics
• Contribute to the tuning and architecture with a strong understanding of related Kafka Connect and Linux fundamentals
• Carefully observe Kafka health metrics and alerts, taking action in a timely manner
• Implement a real-time and batch data input pipeline employing best practices in data modeling and ETL/ELT operations
• Participate in technological decisions and work with smart colleagues
• Review code, implementations, and provide useful input to assist others in developing better solutions
• Develop documentation on design, architecture, and solutions
• Provide assistance and coaching to peers and more junior engineers
• Build good working relationships at all levels of the organization and across functional teams
• Assume accountability for the project's timetables and deliverables
• Create dataflows and pipelines ranging from simple to complicated
• Support the investigation and resolution of production difficulties
• Work to keep the system and data security at a high level, ensuring that the application's confidentiality, integrity, and availability are not jeopardized
• Express stakeholder's needs into familiar language that can be adopted for use with Behavior Driven Development (BDD) or Test-Driven Development (TDD)
• Build solutions that are stable, scalable, and easy to use while fitting into the broader data architecture
• Assists in the formation of Communities of Practice
• Utilize industry-standard approaches to continuously improve the performance of source code
• Consistently enhance the performance of source code using industry-standard methodologies
• Steer the technology direction and options by proferring suggestions based on experience and research
• Encourages the creation of group norms and procedures

Job Requirements:

• Bachelor s/Master s degree in Engineering, Computer Science (or equivalent experience)
• 7+ years of direct expertise with data pipelines and application integrations
• Experience in the design, development of Clusters, and Producers/Consumers
• Proficiency in enabling Cloud/hybrid Cloud using Confluent Kafka Data streaming through Kafka, SQS/SNS queuing, etc
• Strong container expertise, especially Docker
• Prolific skills working with technologies such as Ansible, Puppet, Terraform, OpenShift, Kubernetes, AWS, AWS Lambda, and Event Streaming
• Working experience in a public cloud environment as well as on-premise infrastructure
• DataDog, Splunk, KSQL, Spark, and PySpark experience is a plus
• Excellent knowledge of distributed architectures, including Microservices, SOA, RESTful APIs, and data integration architectures
• Familiarity with any of the following message/file formats: Parquet, Avro, ORC
• Excellent understanding of AWS Cloud Data Lake technologies, including Kinesis/Kafka, S3, Glue, and Athena
• It's advantageous to know RabbitMQ and Tibco Messaging technologies
• Previous expertise in designing and implementing data models for applications, operations, or analytics
• Track record of working with information repositories, data modelling, and business analytics tools is a strong suit
• Be familiar with databases, data lakes, and schemas with advanced expertise and experience in online transactional (OLTP) and analytical processing (OLAP)
• Experience in Streaming Service, EMS, MQ, Java, XSD, File Adapter, and ESB-based application design and development experience
• Capable of working in a fast-paced team to keep the data and reporting pipeline running smoothly