Senior Associate Data Engineering

Overview:-

NPCI is looking for Data Engineers, who are well versed with the Big data technologies. The candidate will be working on-premises handling the massive data, that will be served to the data science and data modelling team.

He / She is responsible for data availability in the existing data pipelines also should be able to establish and make the data available in the new pipelines including specified criteria using latest tools and technologies in Data Engineering.

Should be able to understand, analyze the requirement and well versed in providing the data using file or dashboards.

Key Responsibilities:-

• Understanding the requirement and design the data pipelines to make the data available in the data lake for the further processing and usage for the downstream systems
• Take responsibility for 24/7 data availability, by implementing reliable, fault tolerant system and maintainability and availability of the data.
• Good knowledge on the data base including the tables, schema, view and executing the SQL complex queries based on the business requirements and providing the required output for the relevant stakeholders.
• Write Python, Scala scripts for different types of data filtering, Data conversion, Data cleaning.
• Should have good knowledge on the Bigdata concepts and the Hadoop eco systems.
• Should be a quick learner and capable of easily adopting to the new technologies.
• Good knowledge on Data Warehouse, Data Lake and Data mart.

Candidate Profile: -

• Experience in building the data pipeline and data flow.
• Experience in handing the data Wrangling, data mining, data processing etc
• Experience in NOSQL data base.
• Working experience or knowledge on DBT, Dagster, Trino SQL / Hive SQL, MINio, S3 , Superset will be an advantage.
• Experience in Stream processing like Kafka , Spark and Flink.
• Experience in different data engineering tools and their configurations.
• Should be able to write code in different languages that includes Python, Scala , Java and basic working knowledge in Linux etc.
• Should take complete ownership on the tasks implementing the above tasks.
• Should be good in BI, visualizing technics, displaying the data insights appropriately to the end user and have good experience in the dashboards like Tableau, Superset etc.