Tech Mahindra Azure Data Engineer Interview Questions, Process, and Tips

The output after inner join of table 1 and table 2 will be 2,3,5.

• Inner join only includes rows that have matching values in both tables.

• Values 2, 3, and 5 are present in both tables, so they will be included in the output.

• Null values are not considered as matching values in inner join.

The project involves building a data pipeline to ingest, process, and analyze large volumes of data from various sources in Azure.

• Utilizing Azure Data Factory for data ingestion and orchestration

• Implementing Azure Databricks for data processing and transformation

• Storing processed data in Azure Data Lake Storage

• Using Azure Synapse Analytics for data warehousing and analytics

• Leveraging Azure DevOps for CI/CD pipeline aut

Designing an effective ADF pipeline involves considering various metrics and factors.

• Understand the data sources and destinations

• Identify the dependencies between activities

• Optimize data movement and processing for performance

• Monitor and track pipeline execution for troubleshooting

• Consider security and compliance requirements

• Use parameterization and dynamic content for flexibility

• Implement error handling and retries fo

The project architecture includes Spark transformations for processing large volumes of data.

• Spark transformations are used to manipulate data in distributed computing environments.

• Examples of Spark transformations include map, filter, reduceByKey, join, etc.

Use window functions like ROW_NUMBER() to find highest sales from each city in SQL.

• Use PARTITION BY clause in ROW_NUMBER() to partition data by city

• Order the data by sales in descending order

• Filter the results to only include rows with row number 1

Databricks can be mounted using the Databricks CLI or the Databricks REST API.

• Use the Databricks CLI command 'databricks fs mount' to mount a storage account to a Databricks workspace.

• Alternatively, you can use the Databricks REST API to programmatically mount storage.