Senior Engineer - Big Data EcoSystem (1-2 yrs)

Responsibilities :

- Works closely with the business owner / Product owner / Cross functional Team and gathering technical requirements.

- Experience in building and maintaining reliable and scalable ETL pipeline on big data & / or Cloud platform through the collection, storage, processing, and transformation of large datasets.

- Collaborate with other teams to design, develop and deploy data tools that support both operations and product use cases.

- Work with the team in solving problems in big data technologies and prototype solutions to improve our data processing architecture.

Must Have :

- Ensuring proper execution of tasks and alignment with business vision and objectives.

- Oversee activities of the junior data engineering teams.

- Works closely with the business owner / Product owner / Cross functional Team and gathering technical requirements.

- Experience in building and maintaining reliable and scalable ETL pipeline on big data/Cloud platform through the collection, storage, processing, and transformation of large data-sets.

- Experience working with varied forms of data infrastructure inclusive of relational databases such as SQL, Hadoop, Spark.

- Proficiency in scripting languages in PySpark, python.

- Experience in AWS cloud & DevOps.

- Experience in Databricks.

- Experience in Database design/data modelling.

- Must have strong experience in data warehouse concepts, understanding Data Lake, Lake House.

- Concepts, any new Big Data eco system.

- Experience in testing and validation in order to support the accuracy of data transformations and data verification.

- Should be able to independently drive the requirements, have solutioning mindset facing business stakeholders.

Soft Skills :

- Excellent Verbal and Communication skills needed.

- Should be a excellent team player.

- Good knowledge on Agile principles and experience working in scrum teams using Jira.

- Should be comfortable to mentor junior engineers in the team (For Senior Data Engineers).

- Should be able to operate in an ambiguous environment with minimal guidance.