Senior Data Engineer - Hive/Spark (3-5 yrs)

Job Description :

- Ensure right stakeholders gets right information at right time

- Understand analytical requirement and design data pipelines around it

- Develop, test and maintain optimal and scalable end-to-end data pipelines for Batch as well as Real Time data processing.

- Leverage open source / AWS/Databricks infrastructure/services for creation and automation of data pipelines

- Work with stakeholders including the Product, Data and Development teams to assist with data-related technical issues and support their data infrastructure needs.

- Participate actively in data-marts design discussions and execution

- Write code (queries/scripts) in Spark / Hive / Athena / Python etc that is both functional and elegant, following appropriate design patterns

- Build integrations for data ingestion across various types of data stores

- Identify data quality issues and write data cleanup jobs

- Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metric

- Create and maintain documentation of entire data landscape

Job Specifications :

Mandatory Requirements :

- BE/B.Tech in Computer Science/IT or relevant field from Top Colleges

- Good knowledge in Big Data warehousing, Data modeling, Python, SQL, APIs

- Good knowledge of cloud platforms and their managed services like AWS, GCP

- Good understanding of different types of File Formats.

- Knowledge of SQL query writing (any SQL- like language)

- Good understanding of NoSQL and SQL databases

- Good understanding of Big Data, RDBMS and Cloud Computing concepts

- Knowledge of fundamentals of Data Engineering, structured, semi-structured and unstructured data.

- Strong analytical and communication skills.

- Strong problem Solving Skills

- Ability to find the root cause of issues and provide solutions.

- Smart, motivated, quick learner and team oriented.

- Basic knowledge of ML concepts, MLOps Pipeline and GenAI fundamental.

- Knowledge of Linux and shell scripting