Big Data/Java Developer - Spark/Hadoop (3-8 yrs)

Location- Mumbai, Bangalore, Gurgaon, Indore

Experience 3-8 yrs

Job Description

1. Proficient in Core JAVA (Multithreading and Collections framework)

2. Proficient in cleaning, transforming, and analyzing vast amounts of raw data from various systems using Spark to provide ready-to-use data to our feature developers and business analysts.

3. Proficient in designing efficient data processing pipelines by writing suitable Spark batch and streaming Jobs involving various transformations and aggregations.

4. Proficient in Spark data structures and APIs based on RDDs, Dataframe and Datasets.

5. Proficient in Spark query tuning and performance optimization

6. Proficient in usage of Hadoop Distributed File System (HDFS)

7. Proficient in data formats such as JSON, ORC, Parquet, Avro, etc.

8. Desirable: Understanding of distributed systems (e.g. CAP theorem, partitioning, replication, consistency, and consensus)

9. Desirable : Working with unstructured database (Mongo DB, HBase, Cassandra)

10. Desirable : Understanding and experience of working on SQL databases, such as Oracle, MySQL or Postgres

11. Desirable : Experience with Zookeeper, Hive, Sqoop and other utilities of Hadoop Ecosystem.

12. Must have strong troubleshooting skill in debugging the failure of Spark Jobs on production cluster.