AWS, Python, Apache airflow, SQL, snowflake, AWS S3, AWS Glue, AWS AMR,. Design, develop, and maintain complex data pipelines using Python for efficient data processing and orchestration.
Collaborate with cross-functional teams to understand data requirements and architect robust solutions within the AWS environment.
Implement data integration and transformation processes to ensure optimal performance and reliability of data pipelines.
Optimize and fine-tune existing data pipelines / Airflow to improve efficiency, scalability, and maintainability.
Troubleshoot and resolve issues related to data pipelines, ensuring smooth operation and minimal downtime.
Work closely with AWS services like S3, Glue, EMR, Redshift, and other related technologies to design and optimize data infrastructure.
Develop and maintain documentation for data pipelines, processes, and system architecture.
Stay updated with the latest industry trends and best practices related to data engineering and AWS services