Azure Data Engineer

Skills and Experience

Key Developer and Architect Knowledge:

• Azure Data Lake (Azure Data Lake Storage, Azure Key Vault, Azure Logic App, Azure Functions, Azure Automation, Azure Data Factory, etc)
• Databricks (Unity Catalog, Lakehouse, Workflows, Clusters/SQL Warehouse, Delta Sharing, Volumes, External Locations, Storage Credentials, etc)
• Machine Learning (Supervised, Unsupervised learning models, MLOps, etc)
• Power BI (DAX, Power Query, Power Pivot, Filters, Power BI Gateway, Data Access Mode like Direct Query/Import/Live Connection, Data Refresh Method, Power BI Dataset, PBI Workspace, Method to publish app and report, Various visuals chart types, etc.)
• Data Modelling Framework (Entity Relationship, Data Vault Schema, Star Schema, Snowflake Schema, Data Mart, Materialized View, Data Harmonization, etc.)
• Design and implement data pipelines using Azure Data Factory, Azure SQL Database, Azure Databricks, and Azure Synapse Analytics.
• Programming and Scripting Languages: Strong experience with SQL for querying and data manipulation, and proficiency in Python or Scala for data processing tasks.
• Data Warehousing Concepts: In-depth understanding of data warehousing architecture, design, and optimization techniques. Knowledge of dimensional and relational data modeling concepts, with the ability to create efficient and scalable data models.

Key Management Knowledge
Agile Development Framework (backlog, sprint, scrum, user story & story point, epic, etc.)
Stakeholder Reporting (Agile Development Sprint Report, Waterfall Project Report)
ITIL Process (especially Change & Release Management)
Design Documentation Management
Code Management (CI/CD Pipeline)
Waterfall Project Management
Meta Data Management

Data Source System & Domain Knowledge
Enterprise Systems such as Success Factors, HRMS, ERP, MES
Commercial IT Systems such as SFA, DMS, TPM

Data Engineer with 5-8 years experiences in data platform management, operations, development.

Architecture Design

• Provide the end-to-end solution design when receiving the requests from global/regional data product managers, the design may cover infrastructure, network, data extraction approach, ETL flows, and cube.
• Design the suitable data models (bronze, silver, gold and data mart) to address the business requirements
• Review the local solution designs and provide go/no-go decision.
• Explore the modern tech in the market and keep evolving the current architecture, you may engage with Microsoft or Databricks solution architects to evaluate new releases, features, or products

Technical Development

• Hands-on development for the ETL Pipelines using Azure Stack, including but not limited to ADLS, Azure Data Factory, Logic Apps, Azure Databricks, Azure Automation, Azure Functions, Azure Key Vault, etc.
• Hands-on development for the centralized OLAP Cube (semantic model) using Visual Studio, Bsim Normalizer, Tabular Editor, Dax Studio.
• Design & manage the CI/CD pipelines to make sure smooth deployment from Dev, to QA, and Production environments for Databricks and Azure Data Factory.

Advanced Analytics

• Design the end-to-end advanced analytics solutions, platforms and operation framework.
• Provide consultancies or explanations to the senior leaders to help them understand the ML/Advanced Analytics logics.
• Provide support to build interactive chatbots or GenAI apps to enable insights & better data visibilities.

Operations

• You will have the direct reports of the operation managers.
• Guide operation managers to run the ITIL process, including CAB, Change Request, Incident, Service Request, Problem Management, etc.
• Develop long-term data operation framework and processes across data engineering, analytics enablement, and machine learning.

Governance

• Design the end-to-end governance framework & platform (Unity Catalog & Purview)
• The governance scope will include:
• Global & Local Dataset, this is to define the clear boundary between global & local datasets so that they can be stored in correct environments
• Upper Stream details, this is to prevent ETL pipelines & logics get affected by any changes from upper stream systems
• Data Source management, this is to understand what types of data, which approach the team is using to extract from various data sources
• Lineage Management, this is to have a full view of the data life cycle, the lineage will cover data source - Bronze - Silver - Gold - DMT - PowerBI data set
• Data Quality Management, this is to make sure the data can be ingested into the lakehouse correctly & timely.
• Access Management, this is to make sure the access will not be over-granted to internal/external developers, data scientists, BA/PMS.