Senior Test Engineer

Job Overview:

We are looking for a Software Development Engineer in Test (SDET) to join our Data Engineering team. SDET has a dual role: They are both Data Engineers and quality assurance engineers. They write code to test software, focusing on creating automated tests, frameworks, and tools that help ensure the quality of data pipelines and data itself. The SDET works closely with Scrum teams and stakeholders to ensure the quality and reliability of complex data pipelines, ETL processes, and large-scale data systems. The role requires automating testing at various stages of the development lifecycle, from unit testing frameworks to system, integration, UAT and monitoring production data.

Key Responsibilities:

1. Collaborating with Scrum Teams:
   • Actively participate in daily Scrum meetings, planning and refinement sessions to develop overall test strategy for Epic/Project
   • Work closely with Product Owners, Business Analysts and Data Engineers to ensure that quality is baked into the design and development of data pipelines by identifying functional and non-functional test cases
   • Engage with cross-functional teams like release/deployment, Infrastructure to ensure testing is integrated into the entire data engineering delivery process.
2. Test Strategy and Framework Development:
   • Design, develop, and implement unit testing frameworks for data pipelines, ETL processes, and distributed data systems.
   • Develop automated testing frameworks for system, integration, and UAT.
   • Define test strategies and best practices for ensuring accuracy, reliability, scalability, and performance of the data engineering solutions.
3. Automation of Testing Processes:
   • Automate tests for data validation, schema verification, data transformation, and data quality checks across different stages of the pipeline.
   • Write scripts and programs in languages such as Python, Java, or Scala to automate test execution.
   • Use advanced features in DBT to write generic and custom test cases and auto-execution.
   • Ensure testing is integral part of continuous integration (CI) and continuous delivery (CD) practices to enable automation during the delivery lifecycle.
4. Integration and System Testing:
   • Ensure the quality of data ingestion, transformation, and storage systems by automating integration tests across multiple data sources and sinks.
   • Perform system testing to validate the end-to-end data flow from data ingestion to final reporting and analytics systems.
   • Collaborate with DevOps teams to ensure automated testing is integrated into the deployment pipelines.
5. Performance and Load Testing:
   • Design and implement performance tests to evaluate data pipeline throughput, latency, and scalability under various data volumes and conditions.
   • Identify and resolve bottlenecks in data processing pipelines, working closely with Data Engineers to optimize performance.
6. Test Data Management:
   • Create and manage test data sets for use in automation scripts, ensuring data variety and consistency for different testing scenarios.
   • Develop strategies for handling large-scale test data, ensuring its accuracy and integrity during testing.
7. Defect Identification and Tracking:
   • Identify, document, and track defects found during testing.
   • Work closely with development teams to ensure defects are resolved in a timely manner and that testing is adapted accordingly.
8. Continuous Improvement and Innovation:
   • Continuously evolve testing strategies and frameworks to keep up with new technologies, tools, and best practices in data engineering and quality assurance.
   • Promote a culture of quality, innovation, and automation within the team.

   • Required Skills & Qualifications:

   • Education: Bachelor s degree in computer science, Data Engineering, or a related field. A Master s degree is a plus.
   • Experience:
     • 3-5 years of experience in software testing, quality assurance, or data engineering roles, with a focus on automation.
     • Experience in automating unit tests, system tests, and integration tests in data engineering environments.
     • Hands-on experience with data pipelines, ETL frameworks, and data processing systems (e.g., Hadoop, Spark, Kafka, Airflow, etc.).
   • Technical Skills:
     • Proficiency in programming languages: Python (particularly PyTest Or Nose2 Or Behave), Java(junit), Scala, or similar for developing automation scripts and frameworks.
     • Proficiency with automation testing frameworks: Cucumber, Gherkin, TestNG, or similar for data testing
     • Strong knowledge of SQL and experience in testing databases and data warehouses with DBT (e.g., Snowflake, Redshift, BigQuery).
     • Strong Knowledge of workload automation platforms like Apache Airflow and DBT ( Data Build Tool)
     • Familiarity with CI/CD tools (e.g., Jenkins, Azure CICD ) and experience integrating automated tests into pipelines.
     • Experience with cloud platforms (AWS, GCP, Azure) for testing and deploying data solutions.
     • Knowledge of performance testing and load testing tools such as Apache JMeter or Gatling.
   • Data Engineering Concepts:
     • Deep understanding of data warehousing, data modelling, ETL/ELT processes, and data quality assurance.
     • Knowledge of big data technologies like Apache Spark, Hadoop, Kafka, and distributed data systems.
   • Understanding of TDD and BDD for Data Engineering
   • Agile Methodology:
     • Experience working within Scrum teams and contributing to sprint planning, backlog refinement, and sprint reviews.
     • Strong understanding of the Agile testing lifecycle and how to incorporate automated testing into the Agile development process.

   • Preferred Qualifications:

   • Experience with test data management tools and strategies.
   • Familiarity with data validation and data anomaly detection frameworks like Great Expectations or Deequ.
   • Experience working in a DevOps environment with an emphasis on automating deployments and testing.
   • Knowledge of security testing related to data privacy and compliance, particularly in sensitive data environments.

   • Soft Skills:

   • Strong mindset of software testing and data quality. Focus on not just detecting defects but also on prevention
   • trong analytical and problem-solving skills, with an emphasis on attention to detail.
   • Excellent communication and collaboration skills, especially within cross-functional teams.
   • Ability to handle multiple tasks and projects in a fast-paced environment.