Hadoop Developer - Big Data Infrastructure (3-5 yrs)

Job Summary :

We are seeking a skilled and experienced Hadoop Developer to join our growing data engineering team.

This role will be crucial in developing and maintaining our big data infrastructure using the Hadoop ecosystem.

The ideal candidate will have a strong understanding of Hadoop architecture, hands-on experience with core Hadoop components, and proficiency in related technologies.

Responsibilities :

- Hadoop Development : Design, develop, and implement scalable and efficient data processing solutions using Hadoop and related technologies.

- HDFS Management : Manage and maintain the Hadoop Distributed File System (HDFS), ensuring data availability, integrity, and performance.

- MapReduce/Spark Development : Develop MapReduce or Spark jobs for large-scale data processing and analysis. Optimize jobs for performance and efficiency.

- Data Ingestion : Ingest data from various sources (e.g, relational databases, NoSQL databases, streaming data) into Hadoop using tools like Sqoop, Flume, or Kafka.

- Data Transformation : Transform and process data within Hadoop using tools like Hive, Pig, or Spark.

- Cluster Management : Contribute to the management and monitoring of the Hadoop cluster, including resource allocation, performance tuning, and troubleshooting.

- Performance Optimization : Identify and address performance bottlenecks in Hadoop jobs and the cluster.

- Code Quality : Write clean, efficient, and well-documented code. Adhere to coding best practices.

- Testing : Perform thorough testing of Hadoop jobs and data pipelines.

- Collaboration : Work closely with data scientists, analysts, and other engineers to understand data requirements and deliver effective solutions.

- Staying Current : Stay up-to-date with the latest Hadoop ecosystem technologies and trends.

Technical Skills :

- Hadoop Core Components : HDFS, YARN, MapReduce (Essential)

- Data Processing Frameworks : Spark (Essential), Hive (Essential), Pig (Preferred)

- Data Ingestion Tools : Sqoop, Flume, Kafka (Preferred)

- Programming Languages : Java (Essential for MapReduce), Scala or Python (Essential for Spark), SQL (Essential)

- Operating Systems : Linux (Essential)

- Version Control : Git (Essential)

- Build Tools : Maven, Gradle (Preferred)

- Workflow Orchestration : Oozie, Airflow (Preferred)

- Cloud Platforms (Preferred) : AWS (EMR), Azure (HDInsight), GCP (Dataproc)

- NoSQL Databases (Preferred) : HBase, Cassandra, MongoDB

- Data Warehousing (Preferred) : Experience with data warehousing concepts and tools.

Qualifications :

- Bachelor's degree in Computer Science, Engineering, or a related field.

- Proven experience as a Hadoop Developer.

- Strong understanding of Hadoop architecture and ecosystem.

- Proficiency in Java, Scala or Python, and SQL.

- Experience with MapReduce or Spark development.

- Experience with data ingestion and transformation tools.

- Excellent problem-solving and communication skills