Spark memory optimisation techniques

• Use broadcast variables to reduce memory usage

• Use persist() or cache() to store RDDs in memory

• Use partitioning to reduce shuffling and memory usage

• Use off-heap memo...read more