QA Automation Python- Contractor

We are seeking a skilled Python Developer to join our quality engineering team. The successful candidate will focus on automating data ingestion, processing, and validation workflows using PySpark, ensuring robust, scalable, and efficient data pipelines. You will work closely with data engineers, analysts, and stakeholders to deliver impactful data solutions in a fast-paced environment.

Key Responsibilities

• Collaborate with data engineers and stakeholders to define requirements and deliver automated solutions.
• Design, develop, and maintain scalable and efficient PySpark-based automation for data ingestion, processing and calculation.
• Automate the reading and integration of data from multiple sources, such as databases, APIs, flat files, and cloud storage.
• Implement and optimize ETL (Extract, Transform, Load) workflows for high-performance data pipelines.
• Ensure data quality by incorporating validation checks and handling exceptions in data processing pipelines.
• Troubleshoot and resolve issues in data pipelines to maintain operational efficiency.

Required Skills and Qualifications

• 3+ years strong proficiency in Python for data handling and validation.
• Strong experience in Python libraries like, pandas, NumPy, duckdb etc.
• Familiarity with cloud platforms such as AWS, Azure, or GCP (e.g., S3, Databricks, or BigQuery).
• Experience with data pipeline orchestration tools such as Apache Airflow, or similar.
• Proficiency in SQL for querying and manipulating data.
• Experience in handling structured, semi-structured, and unstructured data.
• Familiarity with CI/CD processes and version control tools such as Git.
• Knowledge of performance tuning and optimization techniques in PySpark.
• Strong analytical and problem-solving skills.

NICE TO HAVE

• knowledge of Spark architecture, including RDDs, DataFrames, and Spark SQL
• Knowledge of Keyword driven automation framework.
• Quality Engineering background with prior experience of building automation solutions for data heavy applications.
• Familiarity with REST APIs and data integration techniques.
• Understanding of data governance, compliance, and security principles.