Data Engineer

Responsibilities:

• Develop a comprehensive technical plan for the migration, including data ingestion,

transformation, storage, and access control in Azure's Data Factory and data lake.

• Design and implement scalable and efficient data pipelines to ensure smooth data movement

from multiple sources using Azure Databricks

• Developing scalable and re-usable frameworks for ingesting of data sets
• Ensure data quality and integrity throughout the entire data pipeline, implementing robust

data validation and cleansing mechanisms.

• Working with event based / streaming technologies to ingest and process data.
• Provide technical guidance and support to the team, resolving any technical challenges or

issues that may arise during the migration and post-migration phases.

• Stay up to date with the latest advancements in cloud computing, data engineering, and

analytics technologies, and recommend best practices and industry standards for

implementing the data lake solution.

Requirements:

• 3 to 7 years of IT experience
• Min. 3 years of Experience working with Azure Databricks
• Experience in Data Modelling & Source System Analysis
• Familiarity with PySpark
• Mastery of SQL
• Knowledge of components: Azure Data Factory, Azure Data Lake, Azure SQL DW, Azure SQL, ETL Development, Data warehouse
• Experience with Python programming language used for data Engineering purpose.