What is coalesce and reparation in Apache spark

Question

Accepted Answer

Coalesce is used to reduce the number of partitions in a DataFrame or RDD, while repartition is used to increase the number of partitions.

Coalesce is a narrow transformation that can only decrease the number of partitions.
Repartition is a wide transformation that can increase or decrease the number of partitions.
Coalesce is preferred over repartition when reducing the number of partitions.
Repartition shuffles the data across the cluster, which can be an expensive operation.
Example: df.coalesce(2) reduces the number of partitions in DataFrame df to 2.
Example: df.repartition(4) increases or decreases the number of partitions in DataFrame df to 4.

Popular interview questions of Big Data Engineer