AWS Data Engineer - Redshift/PySpark (4-7 yrs)

Summary :

Hiring for a technology company headquartered in the US, with centers in Chennai and Bengaluru, focusing on Data Strategy, Data Engineering, and Artificial Intelligence

Looking for an AWS Data Engineer with experience and skills listed below :

Exp. range : 4-7 yrs (min. 4 yrs. of exp., with 3 to 3.5 yrs of relevant experience)

Education : B.E, MCA, or any computers background

Working model : Work from office

Skills :

- Advanced experience in Redshift database - query optimization for better performance, manage large tables partitions

- Must have experience in PySpark

- Must have experience working with AWS tools and services (Glue, S3, Lambda)

- Data Pipelining skills - Data blending using python and sql

- Need to be advanced in both python and sql

- Applied knowledge of visualization tools such as Power BI and Tableau

- Experience in Documentation - create technical specifications and able to translate existing technical designs into data transformation

- Development/Delivery methodologies - Agile Scrum DevOps

Roles and Responsibilities :

Database Management and Optimization :

- Work extensively with Amazon Redshift to manage large tables, optimize queries, and enhance database performance.

- Implement and maintain table partitioning strategies to handle large-scale datasets effectively.

Data Engineering with PySpark :

- Develop and manage ETL pipelines using PySpark for efficient data transformation and processing.

- Optimize PySpark jobs for performance and scalability.

AWS Tools and Services Expertise :

- Leverage AWS services such as Glue, S3, and Lambda to build, deploy, and manage data pipelines.

- Utilize AWS Glue for cataloging data and building serverless ETL processes.

Data Pipelining and Integration :

- Design and implement data blending and integration workflows using Python and SQL.

- Handle complex data transformations to support analytics and reporting needs.

Programming and Query Writing :

- Write advanced Python scripts and SQL queries to perform data manipulation and transformation.

- Ensure the efficiency, reliability, and scalability of code and queries.

Visualization and Reporting :

- Collaborate with teams to design and implement visualizations using tools such as Power BI and Tableau.

- Provide actionable insights through interactive dashboards and reports.

Technical Documentation :

- Create comprehensive technical specifications and documentation for data workflows.

- Translate existing technical designs into clear and actionable data transformation processes.

Agile and DevOps Practices :

- Work within Agile Scrum frameworks, participating in sprint planning, stand-ups, and retrospectives.

- Incorporate DevOps practices to automate and streamline the data engineering lifecycle.

Collaboration and Stakeholder Management :

- Collaborate with cross-functional teams, including analysts, data scientists, and business stakeholders, to understand and address data needs.

- Communicate technical concepts effectively to both technical and non-technical stakeholders.

Performance Monitoring and Troubleshooting :

- Monitor data pipelines for performance and reliability, addressing bottlenecks and failures promptly.

- Troubleshoot and debug issues in data workflows and infrastructure.

Continuous Improvement :

- Stay updated on the latest AWS services and data engineering best practices.

- Innovate and improve existing workflows to increase efficiency and quality.