Senior Data Engineer - Apache Spark

Urgent Requirement : Senior Data Engineer

Location : Trivandrum or Chennai (Hybrid working arrangement)

Work Schedule : Normal working hours

Experience : 7+ years

Key Requirements :

- Proven experience in real-time and batch data processing with Apache Flink or Apache Spark

- Expertise in workflow orchestration using Temporal IO or similar frameworks

- Proficiency in Java and Spring Boot

- Strong experience building RESTful APIs

- In-depth knowledge of NoSQL databases such as MongoDB or Cassandra

- Familiarity with cloud platforms (AWS) and containerization technologies (Docker)

Responsibilities :

- Design, develop, and maintain high-performance, scalable data streams using Apache Kafka, ensuring efficient and fault-tolerant data transmission across distributed systems

- Implement and manage real-time and batch data processing workflows with Apache Flink or Apache

Spark, optimizing for performance and resource utilization

- Troubleshoot, debug, and optimize existing data pipelines to improve data throughput, reduce latency, and enhance overall pipeline reliability

- Build and maintain robust and efficient data pipelines that integrate seamlessly with various data sources, including NoSQL databases like MongoDB or Cassandra

- Work closely with cross-functional teams to gather and understand requirements, translating them into scalable and efficient data solutions

- Create and manage RESTful APIs for seamless data communication between systems, ensuring security, performance, and reliability

- Implement and maintain best practices for data security, governance, and compliance within data systems

- Collaborate with cloud engineers to ensure proper deployment of data solutions in cloud environments (AWS), optimizing for scalability, cost-efficiency, and high availability

- Design and develop containerized solutions using Docker to ensure portability and seamless deployment of data applications across environments

- Continuously evaluate emerging data technologies, frameworks, and tools, and propose improvements to existing data architecture and workflows

Why Join Us ?

- Work on innovative, cutting-edge data projects

- Grow your career with learning and mentorship opportunities

- Collaborate in a dynamic, team-oriented environment

- Make an impact with data-driven solutions

- Enjoy a flexible hybrid working arrangement

- Work with the latest tech stack: Apache Kafka, Flink, Spark, AWS, Docker

- Competitive salary and benefits