Senior Data Tester - Azure ADF and Databricks

About the role

We are looking for an experienced Senior Data Tester to join our data engineering team responsible for designing and implementing complex data solutions using Azure Data Factory (ADF), Azure Databricks, and Cosmos DB. The primary focus of this role is to ensure the quality, accuracy, and reliability of data pipelines, data transformations, and data storage solutions. The Senior Data Tester will collaborate closely with data engineers, architects, and business stakeholders to develop and execute comprehensive testing strategies, including functional, integration, performance, and security testing.

The ideal candidate should possess deep expertise in data validation, automation testing, and performance testing within Azure cloud environments. This role also involves strategic quality improvements and mentoring junior testers in best practices and advanced testing methodologies.

Key Responsibilities

• Test Planning and Strategy:
  • Develop detailed test plans, test cases, and test scripts based on data engineering requirements and architecture designs.
  • Design comprehensive testing strategies for data pipelines, ETL processes, and data transformations.
  • Collaborate with architects and data engineers to understand complex data flows and ensure robust test coverage.
• Test Execution and Automation:
  • Execute functional, integration, regression, and performance tests for data pipelines built using ADF, Azure Databricks, and Cosmos DB.
  • Develop and maintain automated data validation scripts using tools like PyTest, Data Validator, or custom frameworks.
  • Conduct end-to-end data validation ensuring data consistency, accuracy, and integrity.
• Quality Assurance and Governance:
  • Establish data quality metrics and governance practices to maintain high data integrity standards.
  • Ensure compliance with security standards and data privacy regulations.
  • Collaborate with DevOps teams to integrate testing automation into CI/CD pipelines.
• Collaboration and Mentoring:
  • Work closely with data engineers, analysts, and architects to ensure comprehensive test coverage and quality.
  • Mentor and guide junior testers, promoting best practices in test automation and data validation.

Primary Skills

• Data Testing: End-to-end data validation, ETL testing, and data integrity testing.
• Test Automation: PyTest, Selenium, or similar testing frameworks.
• Azure Data Engineering: Azure Data Factory (ADF), Databricks, Cosmos DB.
• Performance Testing: Load testing and performance optimization strategies.
• CI/CD and DevOps: Experience with CI/CD pipeline integration in Azure DevOps.

Secondary Skills

• Data Governance: Data quality metrics and governance practices.
• Security and Compliance Testing: Data security, privacy regulations, and compliance testing.
• Agile Testing Methodologies: Experience in Agile/Scrum environments.
• Communication and Collaboration: Strong communication skills for cross-functional collaboration.

Soft Skills

• Strong problem-solving abilities and attention to detail.
• Excellent communication skills, both verbal and written.
• Effective time management and organizational capabilities.
• Ability to work independently and within a collaborative team environment.
• Strong interpersonal skills to engage with cross-functional teams.

Educational Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.
• Highly desirable relevant certifications in Azure and Data Testing, such as:

• Microsoft Certified: Azure Data Engineer Associate
• Certified Data Quality Engineer (CQE)
• ISTQB Advanced Test Analyst or Test Automation Enginee

About the Team
As a Senior Data Tester, you will be working with a dynamic, cross-functional team that includes developers, product managers, and other quality engineers. You will be a key player in the quality assurance process, helping shape testing strategies and ensuring the delivery of high-quality web applications.