Senior Data Engineer - Spark/Hadoop (6-8 yrs)

Job Title : Senior Data Engineer

Location : Trivandrum or Chennai (Hybrid working arrangement)

Work Schedule : Normal working hours

Experience : 7+ years

Job Description :

We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. The ideal candidate will have in-depth experience designing, developing, and maintaining large-scale data streams and systems, particularly using Apache Kafka. This is a fantastic opportunity to work on cutting-edge technology in a fast-paced environment with an organization that values innovation and collaboration.

Key Responsibilities :

- Design, develop, and maintain large-scale data streams : Work with Apache Kafka to build and maintain high-performance, reliable, and scalable data systems.

- Data processing with Apache Flink or Apache Spark : Implement real-time and batch data processing workflows to handle complex, large-scale data operations.

- Orchestrate workflows : Utilize tools such as Temporal IO or other workflow orchestration frameworks to manage and automate processes.

- API Development : Build and maintain RESTful APIs to enable smooth data flow and interaction between various components.

- Troubleshooting & Optimization : Identify bottlenecks and optimize the performance of data pipelines to ensure high efficiency and reliability.

- NoSQL Databases : Utilize and manage NoSQL databases like MongoDB and Cassandra for optimal data storage solutions.

- Cloud & Containerization : Implement cloud-based solutions and containerized environments (e.g., AWS, Docker) for scalable, flexible, and cost-effective infrastructure.

Key Requirements :

- 7+ years of experience in data engineering with a focus on large-scale data processing and streaming systems.

- Expertise in Apache Kafka for data stream management and processing.

- Experience in real-time and batch data processing using technologies like Apache Flink or Apache Spark.

- Strong proficiency in Java and Spring Boot for building backend applications.

- Proven experience in building RESTful APIs for data interaction and integration.

- Deep understanding of NoSQL databases such as MongoDB, Cassandra, and their use in

scalable data architecture.

- Familiarity with cloud platforms (preferably AWS) and containerization tools like Docker for creating flexible and scalable solutions.

- Familiarity with Temporal IO or other workflow orchestration frameworks.

Desirable Skills :

- Excellent problem-solving and troubleshooting skills.

- Ability to work independently as well as part of a team in a hybrid working environment.

- Strong communication skills and the ability to collaborate with cross-functional teams to ensure seamless integration of data systems.