Sigmoid Data Science Associate Interview Questions and Answers

RDD stands for Resilient Distributed Dataset and is the fundamental data structure of Apache Spark.

• RDD is a distributed collection of objects that can be operated on in parallel.

• DataFrames and Datasets are higher-level abstractions built on top of RDDs.

• RDDs are more low-level and offer more control over data processing compared to DataFrames and Datasets.

Partitioning is the process of dividing data into smaller chunks for better organization and processing in distributed systems.

• Partitioning helps in distributing data across multiple nodes for parallel processing.

• Coalesce is used to reduce the number of partitions without shuffling data, while repartition is used to increase the number of partitions by shuffling data.

• Example: coalesce(5) will merge partitions into 5 pa...read more

Spark is a distributed computing framework that provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.

• Spark has a master-slave architecture with a driver program that communicates with a cluster manager to distribute work across worker nodes.

• It uses Resilient Distributed Datasets (RDDs) for fault-tolerant distributed data processing.

• Spark supports various programming l...read more

DAG stands for Directed Acyclic Graph. It is a finite directed graph with no cycles.

• DAG is a collection of nodes connected by edges where each edge goes from one node to another, but no cycles are allowed.

• In the context of Spark, a DAG represents the sequence of transformations that need to be applied to the input data to get the final output.

• When a Spark job is submitted, Spark creates a DAG of the transformations spe...read more