Migration Engineer - Workflow Orchestration System (8-11 yrs)

About the Role

We are looking for a Software Engineer to play a key role in migrating our workflow orchestration system from Azkaban to Apache Airflow This is a high-impact opportunity to contribute to modernizing our data processing infrastructure. The ideal candidate should have strong skills in Python and Kubernetes stack and experience in workflow orchestration systems (preferably Airflow).

Key Responsibilities

- Design, develop and execute the migration strategy between a legacy system to a new system.

- Develop Python scripts and automation tools to support the migration process.

- Implement best practices for testing and data validation to ensure the accuracy and reliability of flows.

- Work with dev team on automating several manual processes related to release, upgrades, deployments etc

- Collaborate with platform owners and data teams to ensure workflows continue to operate with the same performance post-migration.

- Optimize Airflow DAGs for efficiency, modularity, and maintainability.

- Have previous DevOps experience focusing on continuous deployment and monitoring.

Qualifications and Skills :

- Bachelor's degree in Computer Science, Software Engineering, or a related field.

- Programming languages

- Proficiency in python.

- Should be able to write python scripts to automate tasks such as calling APIs, perform database interaction etc

- Kubernetes ecosystem

- Experience creating YAMLs, helm charts to create kubernetes resources

- Have DevOps experience

Orchestration System :

- Proficiency with Airflow or one orchestration system.

- Create DAGs to run custom workflows is a plus

- Familiarity with jupyter notebooks is a plus