Algoscale - Data Engineer - AWS Glue (4-6 yrs)

Role Overview :

We are seeking a highly skilled Data Engineer to join our team.

The ideal candidate will have a strong background in building and optimizing scalable data pipelines, working with AWS services, and managing data integration processes.

This role requires a blend of technical expertise, client management skills, and the ability to work collaboratively with global stakeholders in a dynamic, fast-paced environment.

Key Responsibilities :

Data Pipeline Development :

- Design, develop, and maintain robust and efficient data pipelines for ingestion, processing, and integration.

- Leverage AWS Glue and PySpark to build, manage, and optimize ETL processes.

- Create and maintain the necessary infrastructure for ETL jobs using S3, Snowflake, and other AWS services.

Data Storage & Management :

- Implement and optimize data storage solutions using Amazon S3, SQL, and NoSQL technologies.

- Ensure efficient and secure data storage, retrieval, and management practices.

Snowflake Analytics Solutions :

- Assist in the design and implementation of Snowflake-based analytics solutions (data lakes and warehouses) on AWS.

- Define requirements, analyze source data, and design logical and physical data models.

- Develop Snowflake deployment and usage best practices.

Optimization & Cost Management :

- Optimize AWS services (e.g , Glue, Lambda, EMR Clusters) for cost-effectiveness and performance.

- Monitor and troubleshoot pipeline and infrastructure issues promptly.

Collaboration & Stakeholder Management :

- Work closely with data architects, data scientists, and analysts to address technical challenges.

- Engage with global stakeholders across time zones to define requirements and deliver solutions.

Technical Leadership & Agile Delivery :

- Translate complex functional requirements into technical solutions in collaboration with product managers and business analysts.

- Deliver high-quality solutions in an Agile/Lean development environment.

Required Skills & Qualifications :

Experience :

- 8-12 years of hands-on experience in data engineering and pipeline development.

- Proven experience in creating and optimizing data pipeline architectures.

Technical Proficiency :

- Expertise in AWS data services, including Glue, Lambda, EMR, and DMS.

- Strong proficiency in PySpark for developing data pipelines.

- Hands-on experience with data storage technologies such as Amazon S3, SQL, and NoSQL databases.

- Advanced SQL skills, with the ability to write and optimize complex queries.

- Data modeling expertise and familiarity with enterprise architecture principles.

AWS Ecosystem Knowledge :

- Strong understanding of AWS infrastructure and services.

- Experience in Snowflake implementation, including designing data lakes and warehouses.

- Optimization of AWS services for performance and cost-efficiency.

Tools & Technologies :

- Proficiency in ETL frameworks and AWS ecosystem tools (e.g , Glue, Athena, Redshift).

- Familiarity with monitoring tools and practices for AWS pipelines.

- Git or other version control systems.

Soft Skills :

- Excellent problem-solving and troubleshooting skills.

- Strong communication skills to collaborate effectively with technical and non-technical stakeholders.

- Ability to work in a globally distributed team and manage stakeholders in different time zones.

- Self-motivated and capable of managing multiple priorities in a fast-paced environment.

Preferred Skills :

- Hands-on experience with Data Migration Services (DMS) for large-scale migrations.

- Experience in designing and implementing solutions using Agile/Lean methodologies.

- Familiarity with performance tuning for large-scale distributed systems.

- Experience with data governance, data quality, and security best practices