Data Quality Assurance Engineer (2-6 yrs)

Job Description : QA Data Engineer

Experience : 2 to 6 Years

Location : Hinjewadi Phase 1, Pune

Requirement :

Data Quality Assurance :

- Ensure that the data being ingested, processed, and delivered is accurate, complete, and meets the business requirements.

- Implement data quality rules and monitor for anomalies, missing data, or inconsistent data patterns.

- Validate data transformations to ensure accuracy and compliance with the data models and specifications.

Test Data Pipelines :

- Design and execute test cases to validate the performance and functionality of data pipelines.

- Ensure that data flows correctly from source to destination, including data extraction, transformation, and loading (ETL).

- Validate data consistency and integrity across various stages of the pipeline (e.g., staging, transformations, and target systems).

Data Validation and Reconciliation :

- Cross-check data between source and target systems to confirm that data is transformed and loaded correctly.

- Perform data reconciliation to ensure that no data is lost or altered incorrectly during the ETL process.

- Validate key data metrics and aggregations to ensure they match expected outcomes.

Automation of Data Testing :

- Develop automated test scripts for data validation, data integrity, and regression testing of data pipelines.

- Implement automation tools such as Apache Airflow, DBT, or other testing frameworks for continuous data testing.

Performance Testing :

- Test the performance of data pipelines to ensure that they can handle large data volumes and meet processing time requirements.

- Identify bottlenecks and recommend optimizations to improve data processing speed, efficiency, and scalability.

Monitoring and Reporting :

- Continuously monitor data pipelines to identify failures or deviations from expected results.

- Generate reports on data pipeline performance, data quality issues, and testing outcomes.

- Proactively report any data inconsistencies, failures, or inefficiencies to the development team.