CGI - Software Engineer - Python/Airflow (3-6 yrs)

Key Responsibilities :

1. ETL Development :

- Design, develop, and optimize scalable ETL workflows and pipelines using Python and Apache Airflow.

- Handle data extraction, transformation, and loading from various sources into GCP services like Cloud Storage and BigQuery.

2. Airflow Implementation & Management :

- Set up, configure, and maintain Apache Airflow in production preferably on GCP (Composer or on VM).

- Develop dynamic and reusable DAGs and workflows for complex ETL processes.

- Monitor and debug Airflow workflows to ensure reliability and performance.

3. GCP Integration :

- Utilize GCP services for ETL processes, including BigQuery, Cloud Storage, Dataflow, and Pub/Sub.

- Implement and manage secure connections to GCP services using IAM roles and service accounts.

4. Optimization & Scaling :

- Handle large-scale data processing efficiently, optimizing for performance and cost.

- Automate data quality checks, error handling, and retry mechanisms in workflows.

5. Collaboration & Documentation :

- Collaborate with Architects, analysts, and other stakeholders to understand ETL requirements.

- Document workflows, configurations, and best practices for Airflow pipelines.