Mphasis Senior Data Engineer Interview Questions and Answers

reduceByKey is more efficient than groupByKey for aggregating data in Spark due to reduced shuffling.

• reduceByKey combines values for each key in each partition before shuffling data

• groupByKey shuffles all data to a single partition before combining values for each key

• reduceByKey is preferred for large datasets to minimize data movement and improve performance

Scala provides a simple way to count words in a string using built-in functions.

• Use the split function to split the string into an array of words

• Use the length function to get the count of words in the array

Use SQL query with ORDER BY and LIMIT to find the second highest salary.

• Use ORDER BY clause to sort salaries in descending order

• Use LIMIT 1,1 to skip the first highest salary and get the second highest salary