Senior Data Engineer (4-8 yrs)

Job Description :

Responsibilities :

- Ability to design and architect complex data lakes on the cloud.

- Build scalable data pipeline and orchestration leveraging state-of-the-art big data technologies.

- Work with data to solve business problems, building and maintaining the infrastructure to answer questions and improve processes.

- Work closely with the data science and decision science teams to develop data stores, orchestration workflows, and pipelines for research, reporting, and machine learning.

- Be an advocate for best practices and continued learning.

- Model front-end and back-end data sources to help draw a more comprehensive picture of user flows throughout our system and enable powerful data analysis.

- Build data pipelines that clean, transform, and aggregate data from disparate sources.

- Build and maintain the MLOps platform to deploy models that can be used to make predictions and answer questions for the overall business.

Requirements :

- Strong Big data engineering (Batch and Streaming) experience in the cloud leveraging techs like EMR, Spark, and Spark streaming.

- 5 or more years of experience with Big data and data lake technology stacks like EMR, Spark, and SQL.

- Familiarity with the AWS ecosystem specifically EMR, Athena, messaging brokers, and RDS.

- Experience building scalable data pipelines or ETL processes.

- Experience building RESTful APIs, architecting complex systems with batch and real-time workloads, and monitoring and instrumenting production systems.