Big Data Engineer - Hadoop/Spark (3-5 yrs)

Job Description :

Responsibilities :

- Collaborate with data analysts and stakeholders to gather requirements, understand data needs, and translate them into technical solutions.

- Design and develop data ingestion frameworks using Spark, Hive, Sqoop, and other relevant technologies.

- Perform data transformations, cleaning, and manipulation to ensure data quality and consistency.

- Utilize SQL for data extraction, transformation, and analysis.

- Manage and optimize data pipelines for performance, scalability, and efficiency.

- Develop and maintain data structures and metadata for efficient data storage and retrieval.

- Implement workload management strategies to optimize resource utilization within the Big Data ecosystem (Hadoop ecosystem).

- Monitor and troubleshoot data pipelines to identify and resolve issues.

- Stay up-to-date with the latest advancements in Big Data technologies and best practices.

Qualifications :

- Proven experience 3+ years as a Big Data Engineer or a related field.

- Hands-on experience with Big Data technologies, including Hadoop, Spark, Hive, Sqoop, and related tools.

- Strong understanding of data structures, metadata, and data quality principles.

- Proficiency in SQL for data manipulation and querying.

- Experience with data ingestion and ETL (Extract, Transform, Load) processes.

- Familiarity with Big Data workload management tools and frameworks.

- Excellent problem-solving and analytical skills.

- Effective communication and collaboration skills to work independently and as part of a team