Sr Data Engineer

• Be the owner for data engineering solutions, ensure the continuity of data processes and the associated batch jobs.
• Identify, design and implement/coordinate implementation of scalable processes and infrastructure to have good governance of automated data processes.
• Manage the end-to-end data solutions of our customers: from raw data analysis to data flow and predictive framework configurations.
• Should be able to work on a problem independently and prepare client ready deliverable with minimal supervision. Elicit, analyze, and validate customer data from ingestion to production
• Monitor all data update processes and outputs to ensure predictive quality

• Communicate with customers to discuss any issues with received data and help them identify and fix data issues
• Solve day-to-day Data problems and customer challenges
• Own the automation, deployment and operation of data pipelines on MS Azure
• Build tools and mechanism to monitor and optimize different parts of the systems
• Build custom integrations between cloud-based systems using APIs

• Expertise in ETL tools like Informatica/DataStage/ADF
• Familiarity with Linux/Unix scripting, advanced SQL proficiency and using various languages (Python, JavaScript, R, SnowSQL) to create sprocs, functions, DDL, and DML
• Exposure to data modeling, creating conceptual and logical model, create DFD, ERD to understand relationship of data and possess good Database concepts.
• Exposure to cloud onboard from legacy data sets.
• Working experience on Azure Data platform, Apache Spark distributed processing system and Cloud computing
• Demonstrates a knowledge of Azure cloud and networking architecture including various storage types: file storage, containers (example)ADLSG2, Azure SQL databases , Azure Data Factory, Azure Databricks and other cloud integration capabilities like Logic Apps, Function Apps, Serverless technology, VNet etc
• Experience in creating data orchestration using ADF, optimizing them through regular monitoring
• Validates logic and data with practices that include data profiling and automated validation tests

Good to have:

• Experience in batch scheduling & rationalization through Control M
• Exposure to reporting tools like PowerBI/ Tableau etc