What are the optimization techniques applied in pyspark code?

Question

Accepted Answer

Optimization techniques in PySpark code include partitioning, caching, and using broadcast variables.

Partitioning data based on key columns to optimize join operations
Caching frequently accessed data in memory to avoid recomputation
Using broadcast variables to efficiently share small data across nodes
Using appropriate data types and avoiding unnecessary type conversions
Avoiding shuffling of data by using appropriate transformations and actions
Using appropriate data structures like DataFrames or Datasets for efficient processing

Accepted Answer

Broadcast join: is recommended If you have one dataset which is smaller than other dataset during join.

To use the Broadcast join:

(df1. join(broadcast(df2))

When you have a small dataset which needs be used multiple times in your program, we cache that dataset.

Cache() – Always in Memory

Persist() – Memory and disks

Serialisation: Spark can use serializer called ‘Kryo’ serializer for better performance.
Use reduceByKey instead of groupByKey. to avoid any shuffle operation
Parquet File Format selection for storage, which has snappy compression techniques
Maintain level of Parallelism by

Accepted Answer

Cache & Persist

File format selection

Repartition & colease

Serialization

Top KPMG India Data Engineer interview questions & answers