Big Data Developer - Hadoop/Spark (6-7 yrs)

Job Description :

We are looking for an experienced Big Data Developer with strong expertise in Hadoop, Spark, Scala, and data pipeline development. The ideal candidate will have a solid background in big data technologies, scalable data processing, and API development, combined with hands-on skills in distributed computing.

This role involves designing and implementing efficient data pipelines, developing APIs, and maintaining large-scale data infrastructure to support data-driven decision-making.

Experience : 6+ years

Education : Bachelor's or Master's degree in Computer Science, Engineering, Mathematics, or a related field.

Roles and Responsibilities :

1. Data Pipeline Development :

- Design and develop robust, scalable data pipelines using Scala and Spark for efficient data processing.

- Collaborate with data analysts and scientists to understand and translate data requirements into technical solutions.

2. Spark and API Development :

- Implement and optimize Spark applications for data transformation and manipulation.

- Develop RESTful APIs to expose processed data to various applications and services.

3. Big Data Infrastructure Management :

- Optimize and maintain the big data infrastructure, including Hadoop, Spark, and other related components.

- Troubleshoot and debug issues related to big data systems, ensuring smooth data processing workflows.

4. Continuous Learning and Innovation : Stay updated with advancements in big data technologies and adopt best practices in big data development.

Skills and Qualifications Required :

1. Big Data Technologies (6+ years) :

- Extensive experience with Big Data frameworks, particularly Hadoop and Spark.

- Strong understanding of the Hadoop ecosystem, including HDFS, YARN, and MapReduce.

- Familiarity with big data workflow schedulers like Oozie and Airflow.

2. Scala and Spark Expertise (4+ years each) :

- Proficiency in Scala programming for big data applications.

- In-depth knowledge of Spark APIs and experience developing efficient Spark applications.

3. Data Pipeline & API Development :

- Proven experience designing and building data pipelines using Scala.

- Skilled in developing RESTful APIs to expose data to external applications.

Additional Skills :

- Experience with Apache Kafka for real-time data streaming (2+ years).

- Strong SQL proficiency for data querying and manipulation (4+ years).

- Knowledge of Unix/Linux operating systems and shell scripting (3-5 years).

- Familiarity with Citi VDI (Virtual Development Infrastructure) is a plus.

Hands-On Development Experience :

- Experience with PySpark, ScalaSpark, and distributed computing.

- 4-6 years of experience in developing and implementing big data applications.

- Knowledge of micro-services architecture and cloud is a plus.

- Java and Scala knowledge is an added advantage.

Preferred Qualifications :

- Experience with Python for big data processing (4-6 years).

- Familiarity with the Hadoop platform, including Hive, HDFS, and Spark