DELTA Data Engineer Interview Questions, Process, and Tips

Spark architecture is based on a master-slave architecture with a cluster manager to coordinate tasks.

• Spark has a master node that manages the cluster and worker nodes that execute tasks.

• It uses a cluster manager like YARN or Mesos to allocate resources and schedule tasks.

• Spark applications are run as independent sets of processes on a cluster, coordinated by the SparkContext object.

• It has a resilient distributed datas...read more

Python program to group anagrams in an array of strings

• Create a dictionary to store sorted strings as keys and corresponding anagrams as values

• Iterate through the array of strings, sort each string and add it to the dictionary

• Return the values of the dictionary as grouped anagrams

AWS cross account sharing can be achieved by using IAM roles and policies.

• Create an IAM role in the account that will share resources

• Define a trust policy in the role to allow the other account to assume the role

• Attach a policy to the role granting the necessary permissions

• In the receiving account, create an IAM role with a trust policy allowing the sharing account to assume the role

• Use the AWS CLI or console to assume

Logical plan represents the high-level abstract representation of the computation to be performed, while physical plan represents the actual execution plan with specific optimizations and details.

• Logical plan is a high-level abstract representation of the computation to be performed.

• Physical plan is the actual execution plan with specific optimizations and details.

• Logical plan is created first and then optimized to gen...read more