AWS Data Engineering

Job Overview:

We are seeking a Lead Data Engineer (AWS) with 5-6 years of experience in AWS Cloud platform, ETL development, and data processing tools. This role involves monitoring and support application workflows, ensuring system connectivity and performance. The ideal candidate should have hands-on experience with in depth knowledge on AWS services, AWS Glue, Redshift, Airflow, and other data processing tools such as Spark, with the ability to guide and mentor a team in a collaborative environment. This role requires leading a team to uphold operational standards hence the candidate should be having an experience handling a team.

Primary Responsibilities:

Application Monitoring & Workflow Management:

• Oversee job monitoring, application flow, and system connectivity to ensure smooth operations.
• Utilizing AWS Glue and Redshift for data processing
• Candidate should possess strong knowledge in AWS Glue, Redshift, and Airflow processes and should be resolving bottlenecks or issues.
• Retrigger failed jobs and manage job orchestration through Airflow for timely task completion.

Cloud Monitoring & Notifications:

• Utilize AWS CloudWatch to monitor system health, performance, and job status.
• Setup AWS Push Notifications to alert the team to system issues, such as database downtime or performance degradation.

Troubleshooting & Data Handling:

• Apply SQL and Spark processing skills to diagnose and resolve issues in data pipelines.
• Work with source systems to address database downtime and connection issues, minimizing workflow disruption.
• Use AWS Glue and Redshift for efficient data processing, transformation, and storage.

ETL Development & Data Warehousing:

• Implementing and maintaining robust ETL processes using AWS Glue, Data Pipeline, or custom scripts.
• Ensure efficient extraction, transformation, and loading of data into the data warehouse.
• Design and maintain scalable, reliable data warehouse solutions on AWS, optimizing data models for AWS Redshift.

SQL Expertise & Performance Optimization:

• Write complex SQL queries for data analysis, reporting, and business intelligence needs.
• Optimize SQL queries for performance, and troubleshoot issues related to data retrieval.

Team Leadership & Management:

• Lead and mentor a team of engineers to ensure effective collaboration, operational efficiency, and timely project delivery.
• Manage rotational shifts for the team to avoid dependency on individual resources and handle escalations as needed.
• Create and deliver reports on team performance, project status, ticket resolution, hours spent, and operational efficiency.

Reporting & Analytics:

• Maintain detailed analytics for project tracking, team performance, and system health.
• Regularly present findings to stakeholders, identifying challenges and proposing improvements.

Documentation:

• Document ETL processes, data models, and system configurations for knowledge transfer.
• Ensure comprehensive documentation of solutions for future reference.

Qualifications And Skills:

Experience:

• 5 to 6years of experience with AWS cloud management, AWS Glue, Redshift, Airflow, and related tools for data processing such as Spark and good knowledge in Airflow orchestration.
• Proven experience in ETL development, data modelling, and designing efficient data warehouses on AWS.
• Strong experience in SQL, Spark processing, and troubleshooting complex workflows.
• Proven leadership experience, managing teams and handling rotational shifts effectively.
• Expertise in writing and optimizing complex SQL queries.
• Ability to troubleshoot and resolve technical challenges with data workflows.
• Good skills with Analytics and Reporting. Candidate should be proficient in creating creating and analysing reports for team performance, project status, and system health and share it to stakeholders and to the management.

Soft Skills:

• Excellent communication and collaboration skills, emphasizing clear and transparent reporting.
• Effective communicator and listener, capable of working in a fast-paced environment.
• Strong ability to manage multiple priorities and ensure team alignment with project goals.

Education & Certification:

• Bachelor's degree in Computer Science, IT, or a related field.
• AWS Cloud technology certifications is a plus.