Cloudaeon - Senior Data Engineer - Automation Testing (6-8 yrs)

About Cloudaeon :

Cloudaeon is a global technology consulting and services company.

We support companies in managing cloud infrastructure and solutions with the help of big data, DevOps and analytics.

We offer first-class solutions and services that use big data and always exceed customer expectations.

Our deep vertical knowledge, combined with expertise in several enterprise-class big data platforms, helps develop targeted solutions to meet our customers' business needs.

Our global team consists of experienced professionals with experience in various tech stacks.

Every member of our team is very active and committed to helping our customers achieve their goals.

Senior Data Engineer with Test Automation Expertise :

We are seeking a highly skilled Senior Data Engineer with expertise in automation testing and data engineering to support our test team in validating data pipelines and integration processes while also assisting the build team as required.

This role is critical in ensuring data quality, performance, and reliability in a large-scale enterprise environment, integrating data from SAP S/4HANA, BW/4HANA, and Kafka/DataSphere into an Azure Medallion Architecture and analytical platforms like Power BI.

Key Responsibilities :

Automation Testing & Data Validation :

- Develop, automate, and maintain data validation workflows for seamless integration into Azure's Bronze, Silver, and Gold layers.

- Validate data pipelines for accuracy, completeness, and consistency across integration layers.

- Test incoming data from SAP S/4HANA, BW/4HANA, and Kafka/DataSphere for quality and adherence to standards.

- Conduct performance and scalability testing for large-scale data pipelines.

- Ensure data governance, compliance, and security standards are met throughout the testing lifecycle.

Automation Framework Development :

- Design and implement reusable automation frameworks for testing data pipelines and migrations in Azure.

- Develop automation scripts using SQL, Python, and tools like Databricks and Azure Data Services.

- Integrate automated testing processes into CI/CD pipelines to enable continuous deployment.

Collaboration and Support :

- Collaborate with the test manager and with the wider team to support test planning, defect tracking, and reporting.

- Assist the build team in troubleshooting and enhancing data pipelines or frameworks as needed.

- Provide technical guidance and mentorship to team members on data testing and pipeline automation.

Required Skills & Experience :

- 6+ years of experience in data engineering, with a focus on testing and automation for data migration and integration projects.

- Strong knowledge of Azure Medallion Architecture (Bronze, Silver, Gold layers).

- Proven experience validating data pipelines and optimizing their performance.

Must-Have : Databricks, SQL, Python, Azure Data Services (Data Factory, Synapse, Databricks).

Nice-to-Have : Kafka, TOSCA, Postman, SOAP UI.

- Experience with Power BI integration and reporting.

- Familiarity with SAP S/4HANA and BW/4HANA data structures.

- Strong problem-solving and analytical capabilities with effective communication skills.