Coforge Senior Data Engineer Interview Questions and Answers for Experienced

Data skewness handling in Spark involves redistributing data to balance workload and optimize performance.

• Use repartition() or coalesce() to redistribute data evenly across partitions

• Consider using broadcast joins for small tables to avoid data shuffling

• Implement custom partitioning strategies for specific use cases

• Monitor job performance and adjust partitioning as needed

Optimisation techniques in Spark improve performance by efficiently utilizing resources.

• Use partitioning to distribute data evenly across nodes

• Cache intermediate results to avoid recomputation

• Use broadcast variables for small lookup tables

• Optimize shuffle operations to reduce data movement