Genpact Senior Data Engineer Interview Questions, Process, and Tips

Working with nested JSON using PySpark involves using the StructType and StructField classes to define the schema and then using the select function to access nested fields.

• Define the schema using StructType and StructField classes

• Use the select function to access nested fields

• Use dot notation to access nested fields, for example df.select('nested_field.sub_field')

Implementing SCD2 involves tracking historical changes in data over time.

• Identify the business key that uniquely identifies each record

• Add effective start and end dates to track when the record was valid

• Insert new records with updated data and end date of '9999-12-31'

• Update end date of previous record when a change occurs

Use a SQL query to select data from table 2 where data exists in table 1

• Use a JOIN statement to link the two tables based on a common column

• Specify the columns you want to select from table 2

• Use a WHERE clause to check for existence of data in table 1

The number of records retrieved after performing joins depends on the type of join - inner, left, right, or outer.

• Inner join retrieves only the matching records from both tables

• Left join retrieves all records from the left table and matching records from the right table

• Right join retrieves all records from the right table and matching records from the left table

• Outer join retrieves all records from both tables, filling