Dataeconomy.io - PySpark Developer - AWS Glue (5-10 yrs)

Key Responsibilities :

- Model Development : Lead the development of advanced models using AWS services such as EMR, Glue, and Glue Notebooks.

- Cloud Infrastructure : Design, build, and optimize scalable cloud infrastructure solutions with a minimum of 5 years of experience.

- ETL Pipeline Development : Create, manage, and optimize ETL pipelines using PySpark for large-scale data processing.

- CI/CD Implementation : Build and maintain CI/CD pipelines for deploying and maintaining cloud-based applications.

- Data Analysis : Perform detailed data analysis and deliver actionable insights to stakeholders.

- Collaboration : Work closely with cross-functional teams to understand requirements, present solutions, and ensure alignment with business goals.

- Agile Methodology : Operate effectively in agile or hybrid agile environments, delivering high-quality results within tight deadlines.

- Framework Development : Enhance and expand existing frameworks and capabilities to support evolving business needs.

- Documentation and Communication : Create clear documentation and present technical solutions to both technical and non-technical audiences.

Required Qualifications :

Experience :

- 10+ years of experience with Python programming.

- 5+ years of experience in cloud infrastructure, particularly AWS.

- 3+ years of experience with PySpark, including usage with EMR or Glue Notebooks.

- 3+ years of experience with Apache Airflow for workflow orchestration.

- Solid experience with data analysis in fast-paced environments.

Domain Expertise :

- Strong understanding of capital markets, financial systems, or prior experience in the financial domain is a must.

Technical Skills :

- Proficiency with cloud-native technologies and frameworks.

- Familiarity with CI/CD practices and tools like Jenkins, GitLab CI/CD, or AWS CodePipeline.

- Experience with notebooks (e.g., Jupyter, Glue Notebooks) for interactive development.

Soft Skills :

- Excellent problem-solving skills and ability to handle complex technical challenges.

- Strong communication and interpersonal skills for collaboration across teams and presenting solutions to diverse audiences.

- Ability to thrive in a fast-paced, dynamic environment.

Preferred Qualifications :

- Experience with data governance and compliance frameworks.

- Familiarity with machine learning frameworks and techniques for financial modeling.

- Certifications in AWS or other cloud technologies.