Apache Airflow Administrator (7-10 yrs)

Job Description :

We are seeking an experienced Apache Airflow Administrator to join our team. The ideal candidate will be responsible for the efficient administration and management of the Apache Airflow platform. This role involves configuring, monitoring, and maintaining Airflow workflows to ensure their smooth running and optimal performance.

Key Responsibilities :

- Installation and Configuration : Install and configure all components of Apache Airflow, tailoring the setup to the specific environment.

- Workflow Management : Author and manage workflows as Directed Acyclic Graphs (DAGs) of tasks in Python, ensuring they are scheduled and managed effectively.

- Monitoring and Troubleshooting : Monitor Airflow workflows, identify and resolve issues, and ensure optimal performance.

- Scalability : Scale the Airflow platform to handle large workloads with multiple executors and workers. 7 to 10 years remote

- Integration : Integrate Airflow with various systems such as Apache Kafka for event-based workflows.

- User Interface Management : Utilize the Airflow UI to inspect and manage workflows, view logs, and troubleshoot issues.

- Best Practices : Implement best practices for version control, resource allocation, and workflow validation.

- EKS Performance Optimization : Optimize the performance of Apache Airflow running on Amazon EKS, ensuring efficient resource utilization and scalability.

Qualifications :

- Experience : Proven experience in administering Apache Airflow or similar workflow management systems.

- Technical Skills : Proficiency in Python, SQL, and experience with cloud platforms such as AWS, Google Cloud, or Azure.

- EKS Skills : Experience with running and optimizing Apache Airflow on Amazon EKS.

- Problem-Solving : Strong analytical and problem-solving skills to troubleshoot and resolve workflow issues.

- Communication : Excellent communication and collaboration skills to work with cross-functional teams.

- Certifications : Relevant certifications in cloud platforms or data engineering are a plus.

Preferred Qualifications :

- Experience with data pipeline orchestration and ETL processes.

- Familiarity with machine learning concepts and data processing pipelines.

- Knowledge of best practices for workflow management and automation.