Tech Mahindra Azure Data Engineer Interview Questions, Process, and Tips

The output after inner join of table 1 and table 2 will be 2,3,5.

• Inner join only includes rows that have matching values in both tables.

• Values 2, 3, and 5 are present in both tables, so they will be included in the output.

• Null values are not considered as matching values in inner join.

The project involves building a data pipeline to ingest, process, and analyze large volumes of data from various sources in Azure.

• Utilizing Azure Data Factory for data ingestion and orchestration

• Implementing Azure Databricks for data processing and transformation

• Storing processed data in Azure Data Lake Storage

• Using Azure Synapse Analytics for data warehousing and analytics

• Leveraging Azure DevOps for CI/CD pipeline aut

Designing an effective ADF pipeline involves considering various metrics and factors.

• Understand the data sources and destinations

• Identify the dependencies between activities

• Optimize data movement and processing for performance

• Monitor and track pipeline execution for troubleshooting

• Consider security and compliance requirements

• Use parameterization and dynamic content for flexibility

• Implement error handling and retries fo

Optimization techniques in Spark improve performance and efficiency of data processing.

• Partitioning data to distribute workload evenly

• Caching frequently accessed data in memory

• Using broadcast variables for small lookup tables

• Avoiding shuffling operations whenever possible

• Tuning configuration settings like memory allocation and parallelism

Methods to transfer data from on-premises storage to Azure Data Lake Storage Gen2

• Use Azure Data Factory to create pipelines for data transfer

• Utilize Azure Data Box for offline data transfer

• Leverage Azure Storage Explorer for manual data transfer

• Implement Azure Data Migration Service for large-scale data migration

Types of joins include inner, outer, left, right, and full joins in Spark queries.

• Inner join: Returns rows that have matching values in both tables

• Outer join: Returns all rows when there is a match in one of the tables

• Left join: Returns all rows from the left table and the matched rows from the right table

• Right join: Returns all rows from the right table and the matched rows from the left table

• Full join: Returns rows w

I am willing to work in a firm office environment.

• I am comfortable working in a structured office setting

• I value collaboration and communication with colleagues

• I am adaptable to different office environments and cultures

The project architecture includes Spark transformations for processing large volumes of data.

• Spark transformations are used to manipulate data in distributed computing environments.

• Examples of Spark transformations include map, filter, reduceByKey, join, etc.

Use window functions like ROW_NUMBER() to find highest sales from each city in SQL.

• Use PARTITION BY clause in ROW_NUMBER() to partition data by city

• Order the data by sales in descending order

• Filter the results to only include rows with row number 1

Databricks can be mounted using the Databricks CLI or the Databricks REST API.

• Use the Databricks CLI command 'databricks fs mount' to mount a storage account to a Databricks workspace.

• Alternatively, you can use the Databricks REST API to programmatically mount storage.