Big Data Engineer (R&D)

Role: Sr Big Data Engineer

Exp: 4 - 9 Years

Location: Kochi/Trivandrum

Job Description:

• Design, develop, and maintain scalable and efficient big data processing pipelines distributed computing systems
• Collaborate with cross-functional teams to understand data requirements and design appropriate data solutions.
• Implement data ingestion, processing, and transformation processes to support various analytical and machine learning use cases.
• Optimize and tune data pipelines for performance, scalability, and reliability.
• Monitor and troubleshoot pipeline performance issues, identifying and resolving bottlenecks.
• Ensure data quality and integrity throughout the pipeline, implementing data validation and error handling mechanisms.
• Stay updated on emerging technologies and best practices in big data processing and analytics, incorporating them into our data engineering practices.
• Document design decisions, technical specifications, and data workflows.
  

Required Qualifications:

• Bachelor's degree in computer science, Engineering, or related field. Master's degree preferred.
• Proven experience (3+ years) as a Data Engineer or similar role, with a focus on big data processing.
• Strong proficiency in programming languages such as Python, Java, & Scala.
• Experience with big data processing frameworks and technologies such as Apache Hadoop, Spark, Kafka, Flink, etc.
• Hands-on experience with distributed computing, parallel processing, and cloud-based data platforms (e.g., AWS, Azure, Google Cloud).
• Proficiency in SQL and database technologies (e.g., SQL Server, PostgreSQL, MySQL).
• Experience with data warehousing, ETL (Extract, Transform, Load) processes, and data modeling.
• Familiarity with containerization technologies (e.g., Docker, Kubernetes) and DevOps practices.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills, with the ability to work

Tools and Technologies:

• IDEs: IntelliJ IDEA, Eclipse
• Build Tools: Maven, Gradle
• Testing Frameworks: JUnit, Mockito, TestNG
• Containerization: Docker, Kubernetes
• API Documentation: Swagger, OpenAPI
• Monitoring and Logging: Prometheus, Grafana, ELK Stack
• Database: MySQL, PostgreSQL, MongoDB, Redis
• ORM Frameworks: Hibernate, Spring Data
• Message Brokers: Kafka