Required Python Data Engineer with Data Quality Experience

Job Description: Python Data Engineer with DQ Experience

Key Responsibilities:

• Develop, maintain, and monitor batch data pipelines/jobs using tools such as PySpark.
• Ensure consistency, accuracy, and minimal lag time in data flows developed in Python, Spark, Hive & SQL.
• Investigate and provide transparency on data quality issues with a forensic mindset.
• Work with real-time streaming applications running on Kafka, ensuring their reliability and performance.
• Execute and manage data quality (DQ) jobs via APIs.
• Track and report on data quality operations metrics, providing actionable insights for improvement.

Qualifications:

• Strong Python programming skills with experience in data engineering tasks.
• Expertise in SQL development for querying and transforming data.
• Hands-on experience with PySpark and other big data tools.
• Understanding of Kafka and real-time streaming applications.
• Familiarity with data quality tools and processes, with the ability to execute DQ jobs programmatically.
• Strong organizational skills with the ability to track, analyze, and report on DQ metrics.

Preferred Skills:

• Experience with Hive and Presto.
• Previous experience in building or monitoring data flows in distributed systems.
• Analytical mindset with strong problem-solving skills.
• Ability to work independently and collaboratively in a fast-paced environment.