Data skewness in Spark can be handled by partitioning, bucketing, or using salting techniques.

• Partitioning the data based on a key column can distribute the data evenly across the nodes.

• Bucketing can ...read more

if one executer got the lot of load in work node after the data shuffling we call it as a data skewness.

1. Repartition by Column(s)

The first solution is to logically re-partition your data based on the transformations in your script. In short, if you’re grouping or joining, partitioning by the groupBy/j...read more

We handle skewness via

1) log transform

2) square root transform

We can drop the tables including back-up tables associated with that db to reduce skewness