Senior / Lead Big Data Engineer

Job Title: Senior/ Lead Big Data Engineer

Location: Chennai / Pune

Experience: 7+ Years

Job Description:

We are seeking an experienced Senior/Lead Big Data Engineer to join our dynamic team. The ideal candidate will have a strong background in Big Data technologies, distributed computing, and data processing frameworks, with expertise in Hadoop, Spark, Scala, and Cloudera. Proficiency in either Java or Python is essential.

Key Responsibilities:

• Design, develop, and implement scalable big data solutions using Hadoop, Spark, and related technologies.
• Build and optimize data pipelines and workflows to support business intelligence and analytics.
• Develop high-performance distributed applications using Scala, Java, or Python.
• Work with large-scale structured and unstructured datasets to perform data ingestion, transformation, and storage.
• Implement best practices for data governance, security, and performance tuning.
• Collaborate with cross-functional teams including data scientists, analysts, and DevOps engineers.
• Optimize and troubleshoot Hadoop ecosystem components for maximum efficiency.
• Deploy and manage Cloudera-based big data platforms.
• Automate data processing and ETL tasks to enhance operational efficiency.

Required Skills & Qualifications:

• 8+ years of experience in Big Data Engineering or a related field.
• Strong expertise in Hadoop ecosystem (HDFS, YARN, Hive, HBase, Oozie, Sqoop, etc.).
• Hands-on experience with Apache Spark for large-scale data processing.
• Proficiency in Scala, Java, or Python.
• Experience working with Cloudera or other enterprise big data platforms.
• Solid understanding of distributed computing principles and data architecture.
• Experience with data modeling, ETL pipelines, and real-time data processing.
• Familiarity with cloud services (AWS, Azure, or GCP) is a plus.
• Strong problem-solving and analytical skills.
• Ability to work independently and within a team in a fast-paced environment.

Preferred Qualifications:

• Cloudera/Hortonworks certification.
• Experience with streaming technologies like Kafka, Flink, or Storm.
• Knowledge of containerization (Docker, Kubernetes) and CI/CD pipelines.