MathCo Data Engineer Interview Questions and Answers

Slowly changing data handling in Spark involves updating data over time.

• Slowly changing dimensions (SCD) are used to track changes in data over time.

• SCD Type 1 updates the data in place, overwriting the old values.

• SCD Type 2 creates a new record for each change, with a start and end date.

• SCD Type 3 adds a new column to the existing record to track changes.

• Spark provides functions like `from_unixtime` and `unix_timestam

Explanation of cumulative sum and rank functions in Spark

• Cumulative sum function calculates the running total of a column

• Rank function assigns a rank to each row based on the order of values in a column

• Both functions can be used with window functions in Spark

• Example: df.withColumn('cumulative_sum', F.sum('column').over(Window.orderBy('order_column').rowsBetween(Window.unboundedPreceding, Window.currentRow)))

• Example: df...read more

Answering about joins in SQL and modeling/visualization in PowerBI

• Joins in SQL are used to combine data from two or more tables based on a related column

• There are different types of joins such as inner join, left join, right join, and full outer join

• PowerBI is a data visualization tool that allows users to create interactive reports and dashboards

• Data modeling in PowerBI involves creating relationships between tables a...read more