Explain Spark RDD,DF

Question

Accepted Answer

Spark RDD and DF are two data structures in Apache Spark for processing and analyzing data.

RDD (Resilient Distributed Dataset) is a distributed collection of elements that can be operated on in parallel. It is immutable and fault-tolerant.
DF (DataFrame) is a distributed collection of data organized into named columns. It provides a more structured and efficient way to work with data compared to RDDs.
RDD is low-level and more suitable for unstructured data processing, while DF is high-level and more suitable for structured data processing.
DFs are built on top of RDDs and provide optimizations like query optimization and code generation.
Example: Creating an RDD from a list - val rdd = sc.parallelize(List(1, 2, 3, 4, 5))
Example: Creating a DF from a JSON file - val df = spark.read.json("path/to/file.json")

Top HCLTech Data Engineer interview questions & answers