Big Data Engineer - ETL/Hadoop (6-8 yrs)

Responsibilities :

- Design, develop, and implement scalable data pipelines using Python and big data tools within the Hadoop ecosystem (HDFS, Hive, Sqoop, NiFi, Airflow etc.

- Build and maintain robust Data Marts, including Cubes, Dimensions, Facts, and ensure efficient data warehousing (DWH) practices.

- Collaborate with data analysts and business stakeholders to understand data requirements and translate them into technical specifications.

- Write and execute complex SQL queries to extract, transform, and load (ETL) big data for analysis.

- Develop and maintain data pipelines using scripting languages (Shell scripting preferred).

- Manage and optimize data quality throughout the data lifecycle.

- Monitor and troubleshoot data pipelines to ensure smooth operation and identify potential issues.

- Stay up-to-date with the latest big data technologies and trends, continuously seeking opportunities to improve our data architecture.

- Document data pipelines and processes for clear communication and maintainability.

Qualifications :

- Minimum of 6-8 years of experience as a Big Data Engineer or a related role.

- In-depth knowledge of Hadoop ecosystems, including HDFS, Hive, Sqoop, NiFi, Airflow, and Python scripting.

- Solid experience in building and maintaining Data Marts, Cubes, Dimensions, and Facts.

- Proven experience working in a Data Warehouse (DWH) environment.

- Proficiency in SQL for data manipulation and querying.

- Experience with Shell scripting for automating data processing tasks (a plus).

- Familiarity with big data technologies like PySpark and HBase (a plus).

- Strong analytical and problem-solving skills.

- Excellent communication, collaboration, and interpersonal skills.

- A passion for big data and a commitment to continuous learning