PwC Azure Data Engineer Interview Questions, Process, and Tips

Partition key is a field used to distribute data across multiple partitions in a database for scalability and performance.

• Partition key determines the partition in which a row will be stored in a database.

• It helps in distributing data evenly across multiple partitions to improve query performance.

• Choosing the right partition key is crucial for efficient data storage and retrieval.

• For example, in Azure Cosmos DB, partit...read more

Data bricks is a unified analytics platform for big data and machine learning, while ADF (Azure Data Factory) is a cloud-based data integration service.

• Data bricks is a unified analytics platform that provides a collaborative environment for big data and machine learning projects.

• ADF is a cloud-based data integration service that allows you to create, schedule, and manage data pipelines.

• Data bricks supports multiple pr...read more

Delta load in ADF is achieved by comparing source and target data and only loading the changed data.

• Use a Lookup activity to retrieve the latest watermark or timestamp from the target table

• Use a Source activity to extract data from the source system based on the watermark or timestamp

• Use a Join activity to compare the source and target data and identify the changed records

• Use a Sink activity to load only the changed re

Blob is a storage service for unstructured data, while ADLS is optimized for big data analytics workloads.

• Blob is a general-purpose object storage service for unstructured data, while ADLS is optimized for big data analytics workloads.

• ADLS offers features like file system semantics, file-level security, and scalability for big data analytics, while Blob storage is simpler and more cost-effective for general storage nee...read more

There are three types of triggers available in Azure Data Factory: Schedule, Tumbling Window, and Event.

• Schedule trigger: Runs pipelines on a specified schedule.

• Tumbling Window trigger: Runs pipelines at specified time intervals.

• Event trigger: Runs pipelines in response to events like a file being added to a storage account.