Data Engineer - Kafka

We are looking for a talented Data Engineer with hands-on experience in various big data technologies and automation tools.

The ideal candidate will have a strong background in Spark, Hive, and PySpark, along with proficiency in Python, Linux, and Bash scripting.

Experience with R for automation tasks and managing scheduled jobs using Crontab is also required.

Responsibilities :

• - Develop, implement, and optimize data pipelines using Spark, Hive, and PySpark.
• - Write efficient and reusable Python code for data processing and analysis.
• - Manage and maintain Linux-based data environments.
• - Create and maintain Bash scripts for automation and data processing tasks.
• - Utilize R for specific automation tasks related to data processing.
• - Schedule and manage jobs using Crontab to ensure timely data processing and reporting.
• - Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
• - Monitor and troubleshoot data workflows to ensure reliability and performance.
• - Document processes and maintain comprehensive technical documentation.

Required Skills and Experience:

• - Medium to advanced level knowledge and hands-on experience with Spark, Hive, and PySpark.
• - Strong proficiency in Python for data engineering tasks.
• - Solid experience with Linux operating systems and command-line tools.
• - Expertise in Bash scripting for automation.
• - Familiarity with R for specific automation tasks.
• - Experience with Crontab for job scheduling.
• - Strong problem-solving skills and attention to detail.
• - Excellent communication skills and ability to work in a team-oriented environment.

5. adb/adf/pyspark with Kafka or any streaming exp.

Scala, Kafka, Hadoop, Gcp, Spark