PwC Data Engineer Interview Questions, Process, and Tips

Spark is a fast and general-purpose cluster computing system for big data processing.

• Spark provides APIs in Java, Scala, Python, and R for distributed data processing.

• It includes components like Spark SQL for SQL and structured data processing, Spark Streaming for real-time data processing, MLlib for machine learning, and GraphX for graph processing.

• Spark can run on top of Hadoop, Mesos, Kubernetes, or in standalone mo...read more

Transformations are operations performed on data to convert it from one form to another. There are mainly two types of transformations: narrow and wide.

• Transformations are operations performed on data to convert it from one form to another.

• Narrow transformations are those where each input partition will contribute to only one output partition, e.g., map, filter.

• Wide transformations are those where each input partition ...read more

Spark job process involves job submission, DAG creation, task scheduling, and task execution.

• Spark job is submitted to the SparkContext by the user.

• Spark creates a Directed Acyclic Graph (DAG) of the job's stages and tasks.

• Tasks are scheduled by the Spark scheduler based on data locality and resource availability.

• Tasks are executed on worker nodes in the cluster.

• Output is collected and returned to the user.

Coalesce and repartition are concepts used in data processing to control the number of partitions in a dataset.

• Coalesce is used to reduce the number of partitions in a dataset without shuffling the data, which can improve performance.

• Repartition is used to increase or decrease the number of partitions in a dataset by shuffling the data across the cluster.

• Coalesce is preferred over repartition when reducing partitions t...read more

OOM stands for Out Of Memory and driverhead memory refers to the memory allocated to the driver in a Spark application.

• OOM occurs when a system runs out of memory to allocate for processes, leading to crashes or performance issues.

• Driverhead memory in Spark is the memory allocated to the driver program, which coordinates tasks and manages the overall execution of the application.

• Adjusting memory settings like executor ...read more

Data skewness is a measure of asymmetry in the distribution of data values.

• Data skewness indicates the lack of symmetry in the data distribution.

• Positive skewness means the tail on the right side of the distribution is longer or fatter.

• Negative skewness means the tail on the left side of the distribution is longer or fatter.

• Skewness value of 0 indicates a perfectly symmetrical distribution.

Coalesce is used to return the first non-null value among its arguments, while reparation is not a standard function in SQL.

• Coalesce is a standard SQL function, while reparation is not.

• Coalesce returns the first non-null value among its arguments.

• Reparation is not a standard SQL function and may refer to a custom function or process specific to a certain system or application.

Use SQL query with ORDER BY and LIMIT to find the third highest salary from a table.

• Use ORDER BY clause to sort salaries in descending order

• Use LIMIT 1 OFFSET 2 to skip the first two highest salaries

• Example: SELECT salary FROM employees ORDER BY salary DESC LIMIT 1 OFFSET 2

repartition vs coalesce, persist vs cache

• repartition is used to increase or decrease the number of partitions in a DataFrame, while coalesce is used to decrease the number of partitions without shuffling

• persist is used to persist the DataFrame in memory or disk for faster access, while cache is a shorthand for persisting the DataFrame in memory only

• repartition example: df.repartition(10)

• coalesce example: df.coalesce(5)

• ...read more