DELTA Data Engineer Interview Questions and Answers

Spark architecture is based on a master-slave architecture with a cluster manager to coordinate tasks.

• Spark has a master node that manages the cluster and worker nodes that execute tasks.

• It uses a cluster manager like YARN or Mesos to allocate resources and schedule tasks.

• Spark applications are run as independent sets of processes on a cluster, coordinated by the SparkContext object.

• It has a resilient distributed datas...read more

Python program to group anagrams in an array of strings

• Create a dictionary to store sorted strings as keys and corresponding anagrams as values

• Iterate through the array of strings, sort each string and add it to the dictionary

• Return the values of the dictionary as grouped anagrams

AWS cross account sharing can be achieved by using IAM roles and policies.

• Create an IAM role in the account that will share resources

• Define a trust policy in the role to allow the other account to assume the role

• Attach a policy to the role granting the necessary permissions

• In the receiving account, create an IAM role with a trust policy allowing the sharing account to assume the role

• Use the AWS CLI or console to assume...read more

Logical plan represents the high-level abstract representation of the computation to be performed, while physical plan represents the actual execution plan with specific optimizations and details.

• Logical plan is a high-level abstract representation of the computation to be performed.

• Physical plan is the actual execution plan with specific optimizations and details.

• Logical plan is created first and then optimized to gen...read more

SHIR stands for Self-hosted Integration Runtime, enabling data integration in Azure Data Factory.

• SHIR allows data movement between on-premises and cloud data sources.

• It supports various data integration scenarios, such as ETL processes.

• Example: Using SHIR to connect an on-premises SQL Server to Azure Data Factory.

• SHIR can be installed on any machine that meets the system requirements.

• It provides secure data transfer by...read more

An anti join in PySpark returns rows from one DataFrame that do not have matching rows in another DataFrame.

• An anti join is useful for filtering out records that exist in another DataFrame.

• In PySpark, you can perform an anti join using the 'join' method with 'how' set to 'left_anti'.

• Example: df1.join(df2, on='id', how='left_anti') returns rows from df1 where 'id' is not in df2.

• Anti joins can help in data cleaning by id...read more