Appglide - Data Engineer - Python (3-5 yrs)

Job Description :

- Responsible for the documentation, design, development, and architecture of Hadoop applications.

- Must have at least 3 years of hands-on working knowledge on Big Data technologies such as Impala, Hive, Hadoop, Spark, Spark streaming, Kafka etc.

- Excellent programming skills in python.

- Experience with stream-processing systems : Storm, Spark Streaming, etc.

- Experience with relational SQL and NoSQL databases, including Vertica.

- Create and maintain scalable big data applications using Python, Spark, Hive, and Impala.

- Develop and optimize data processing pipelines to handle large datasets.

- Implement data ingestion, transformation, and loading processes.

- Work with data scientists and analysts to meet data requirements.

- Ensure data quality, integrity, and security

- Monitor and troubleshoot performance issues to improve efficiency.

- Participate in code reviews, testing, and documentation.

- Stay updated with industry trends and advancements in big data technologies.

Requirements :

- Bachelors or Masters degree in Computer Science, IT, or related field.

- Proficiency in Python.

- Strong experience with Apache Spark.

- Hands-on experience with Hive and Impala.

- Familiarity with Hadoop, HDFS, Kafka, and other big data tools.

- Knowledge of data modeling, ETL processes, and data warehousing concepts.

Soft Skills :

- Excellent problem-solving, communication, and teamwork skills.