Airflow Data Engineer

Key Responsibilities:

• Design and Development: Lead the design, development, and optimization of complex, scalable data pipelines and workflows using Apache Airflow. Create reusable components and efficient DAGs (Directed Acyclic Graphs).
  
• Workflow Automation: Build and automate end-to-end data pipelines for batch and real-time processing using Airflow, integrating data sources and systems with appropriate transformations.
  
• Cloud Platform Integration: Leverage cloud environments (AWS) to develop, deploy, and manage scalable Airflow DAGs, ensuring fault tolerance and high availability of data workflows.
  
• Performance Optimization: Monitor, analyze, and optimize pipeline performance and execution times to meet SLAs, resolving issues proactively.
  
• Collaboration & Mentorship: Work closely with data engineers, data scientists, and stakeholders to understand requirements and ensure alignment. Mentor junior developers and provide guidance on Airflow best practices.
  
• Code Quality & Best Practices: Follow and enforce best practices in terms of coding standards, version control, and documentation. Maintain high-quality, readable, and maintainable code.
  
• Troubleshooting & Issue Resolution: Investigate and resolve Airflow-related issues, such as task failures, data inconsistencies, and workflow bottlenecks, ensuring high system reliability.
  
• CI/CD and Automation: Implement and manage Continuous Integration and Continuous Deployment (CI/CD) pipelines for Airflow DAGs, ensuring streamlined deployment cycles.
  
• Data Quality & Security: Ensure data integrity, quality, and security by enforcing data governance and privacy policies within the Airflow workflows.
  
• Documentation: Create comprehensive documentation for developed workflows, architecture, and deployment processes.
  
  
  

  Required Skills and Qualifications:
  

  • Experience: 7-8 years of professional experience in data engineering or software development, with at least 3-4 years focused on Apache Airflow.
    
  • Airflow Expertise: Deep understanding of Airflow architecture, DAG creation, and task management. Experience with Airflow 2.x and its features such as dynamic pipelines, Airflow REST API, and Scheduler.
    
  • Python Proficiency: Strong expertise in Python, particularly in data processing and integration. Experience with libraries like Pandas, NumPy, etc.
    
  • Cloud Platforms: Experience with cloud platforms such as AWS (S3, EC2, Lambda, etc.).
    
  • Database & SQL: Strong skills in SQL and experience working with relational databases (PostgreSQL, MySQL, etc.).
    
  • Version Control: Proficiency in Git or other version control systems for managing source code.
    
  • Workflow Orchestration: Knowledge of additional workflow orchestration tools such as Kubernetes, Celery.
    
  • Problem-Solving: Strong troubleshooting skills, with the ability to identify and resolve issues in data workflows quickly.
    
  • Collaboration: Excellent communication skills and experience working in cross-functional teams. Ability to effectively mentor and collaborate with junior developers.
    
  • Agile Development: Familiarity with Agile methodologies and tools such as Jira, Confluence, or similar platforms.