Big Data Engineer - Python/Hadoop (5-8 yrs)

We are hiring for Big Data Engineer with strong exp in Python programing along with Big Data Engineer

Company Name : Hum Technologies

Client : One of the Fortune 500 Companies

Location : Remote/Hybrid

Skills : Python, Cloudera Hadoop, Spark, HDFS, Hive, Impala, SQL, Airflow, PySpark, Shell scripting, Cloudera manager, Yarn, Parquet

Responsibilities :

- Devise technical solution using Cloudera Hadoop and Spark ecosystem technologies

- Data pipeline design, implementation and monitoring

- Data ingestions from text and json files using pyspark programs

- Data ingestions from relational databases using sqoop commands

- Data manipulation stored in Hive tables and HDFS file system

- Data transformations using pyspark programs and impala queries

- Monitoring the data pipelines through Airflow

- Wrapping python, pyspark , impala, spark-submit commands into shell scripts

- Help to ensure work carried out is in line with quality standards

Essential Skills :

- Strong knowledge of Hadoop/Big Data ecosystem technologies

- Good working knowledge of Apache Spark with Python

- Experience in working with Cloudera clusters

- Must have hands on experience in HDFS commands and Hive table creation

- Must have hands on experience in data ingestions to Hadoop datalake using sqoop and pyspark

- Must have hands on experience in analyzing data sets using Hive/Impala SQL though Hue interface

- Must have hands on experience in writing transformations using PySpark and SQL

- Experience in monitoring data pipelines through airflow

- Experience in debugging and troubleshooting Spark Jobs, sql queries and shell scripts

- Experience in Azure Databricks and ADF is a plus