Lumiq Data Engineer Interview Questions and Answers

Q1. How to optimize spark jobs ?
Ans. 
Optimizing Spark jobs involves tuning various parameters such as memory allocation, partitioning, and caching.
• Increase memory allocation for executors and driver

• Partition data appropriately to avoid skewness

• Cache frequently accessed data to avoid recomputation

• Use broadcast variables for small data sets

• Avoid shuffling data unnecessarily

• Use appropriate serialization format

• Use appropriate hardware for cluster

• Monitor and o
Answered by AI
Add your answer
Q2. What is coalesce and repartition ?
Ans. 
Coalesce is used to select the first non-null value from a set of columns. Repartition is used to shuffle data across nodes.
• Coalesce is a function that returns the first non-null value from a set of columns.

• Repartition is used to shuffle data across nodes to increase parallelism.

• Coalesce reduces the number of columns in a DataFrame.

• Repartition increases or decreases the number of partitions in a DataFrame.

• Coalesce is u...read more
Answered by AI
Add your answer

Basic coding questions

Q1. Based on resume
Add your answer