AWS Glue Developer - ETL (3-5 yrs)

Responsibilities :

- Design and develop ETL jobs using AWS Glue, including crawlers, ETL scripts (PySpark/Scala), and workflows.

- Implement data transformation logic using PySpark or Scala within AWS Glue jobs.

- Optimize Glue jobs for performance and scalability, considering factors like data partitioning and job execution.

- Create and manage Glue crawlers to discover and catalog data schemas from various sources.

- Develop and maintain Glue workflows to orchestrate ETL processes and dependencies.

- Integrate AWS Glue with other AWS services, such as S3, Redshift, Athena, and Lake Formation.

- Monitor and troubleshoot Glue jobs, ensuring data quality and pipeline reliability.

- Implement data quality checks and validation rules within ETL processes.

- Collaborate with data engineers and data scientists to understand data requirements and build appropriate pipelines.

- Stay up-to-date with the latest AWS Glue features and best practices.

Qualifications :

- Bachelor's degree in Computer Science or a related field.

- 3-5 years of experience in data engineering and ETL development.

- Strong understanding of AWS Glue and its components (crawlers, jobs, workflows, data catalogs).

- Proficiency in PySpark or Scala for data transformation within Glue jobs.

- Experience with data warehousing concepts and dimensional modeling.

- Knowledge of AWS services related to data storage and processing (S3, Redshift, Athena, Lake Formation).

- Ability to write SQL queries for data extraction and manipulation.

- Excellent problem-solving and analytical skills.

- Strong communication and collaboration skills.

Additional Skills :

- Experience with other ETL tools and technologies.

- Familiarity with data lake architectures.

- Knowledge of data governance and security best practices.

- Experience with CI/CD pipelines for ETL deployments.

- AWS Certifications (e.g, AWS Certified Data Analytics - Specialty) are a plus