Big Data Engineer - Hadoop (8-13 yrs)

Job Description :

We are seeking a highly skilled Big Data Engineer with expertise in distributed systems, the Hadoop ecosystem, and advanced programming techniques. The ideal candidate will possess strong technical knowledge, thought leadership, and a passion for creating scalable, high-performance systems.

Key Responsibilities :

- Design, develop, and optimize distributed systems and Big Data solutions.

- Implement and maintain batch and streaming pipelines using Scala and Spark.

- Leverage experience in the Hadoop ecosystem, including technologies like Hive, Oozie, and Kafka.

- Build and maintain CI/CD pipelines to ensure efficient code deployment and integration.

- Apply design patterns, optimization techniques, and locking principles to enhance system performance and reliability.

- Scale systems and optimize performance through effective caching mechanisms.

- Collaborate with cross-functional teams to influence and drive innovative solutions.

- Provide technical leadership, mentorship, and presentations to guide team members and stakeholders.

Preferred Qualifications :

- Scala Python, spark(preferably scala should be an experienced spark exposure- Should have ML exposure also)

- Good understanding of Spark Architecture including Spark Core, Spark SQL, data Frames, Spark Streaming, Driver Node, Worker Node, stages, Executors and Tasks.

- Should have experience with Hadoop ecosystem-HDFS, hive (must), Oozie, sqoop etc.

- Should have prior experience working with big data ingestion from Various sources handling TB' of data.

- Excellent Knowledge of SQL and coding skills.

- Experience with scripting languages such as Shell.

- Effective working in a team as well as working independently.

Experience of working in Agile delivery.