Data Engineer - Python/AWS (7-12 yrs)

Data Engineer

Overview /Objective :

Roles and Responsibilities :

- Coordinate with data scientists/analysts, product managers and business leaders to understand data needs and deliver on those needs and setting expectations with Product Managers/Analysts.

- Build the infrastructure for optimal extraction, transformation and loading data from a wide variety of data sources using big data technologies and integration across different systems and platforms

- Develop, build, optimize and implement pipeline, data ingestion, storage, wrangling, cataloguing, quality, security features from various data sources on cloud

- Automate jobs (ingestion & pipelines), notifications and reports

- Write quality code and participate in code reviews

- Experienced in implementing standardized pipelines with automated testing, Airflow scheduling, Azure DevOps for CI/CD

- Continuously improve systems through performance enhancements and cost reductions in compute and storage

- identify opportunities for improvement in engineering practices and software delivery

- Implement end to end Machine Learning workflows using MLFlow, including model training tracking and deployment ensuring scalability and performance optimization

- Data Processing and API Integration : Utilize Spark Structured Streaming for real-time data processing and integrate data outputs with REST APIs

- Prioritize to manage ad-hoc requests in parallel with ongoing sprints

- Participate with the team to execute sound solutions and approaches to meet business expectations in an efficient manner; deliver and execute the technical and product roadmap for data engineering across a variety of technologies.

- Experienced with Scrum and Agile Methodologies to coordinate global delivery teams, run scrum ceremonies, manage backlog items, and handle escalations

Required Skill Sets :

- Experience working in AWS, Redshift, Glue

- Prior exposure to Data Ingestion and Curation work (such as working with Data Lakehouse)

- Knowledge in SQL for purpose of data analysis/investigation

- Work with the Data Product Owners and Data Analysts to help manage and deliver on the product technical roadmap for the data platform

Preferred Qualifications :

- Degree in computer science, statistics, or related discipline

- 7+ years as a data engineer

- Comfortable making decisions and leading

- Familiar with version control and relational databases