Write a PySpark function to flatten a DataFrame with nested structures (arrays and structs). The function should take a DataFrame as input and return a DataFrame where all nested structures are flattened into individual columns. Include handling for null values in arrays. Provide an example usage with sample data.

Question

Accepted Answer

The function flattens a DataFrame by expanding complex fields like arrays and structs.

Identifies complex fields (arrays and structs) in the DataFrame schema.
Uses explode_outer for ArrayType columns to create new rows for each element.
Expands StructType columns by creating new columns for each field in the struct.
Updates the list of complex fields after processing each column until none remain.
Example: A struct {'city': 'New York', 'state': 'NY'} becomes two columns: 'location_city' and 'location_state'.

TCS Associate interview questions & answers

Popular interview questions of Associate