Persistent Systems Data Engineer Interview Questions and Answers

Optimizing SQL queries involves using indexes, avoiding unnecessary joins, and optimizing the query structure.

• Use indexes on columns frequently used in WHERE clauses

• Avoid using SELECT * and only retrieve necessary columns

• Optimize joins by using INNER JOIN instead of OUTER JOIN when possible

• Use EXPLAIN to analyze query performance and make necessary adjustments

Performance optimization in Spark involves tuning configurations, optimizing code, and utilizing caching.

• Tune Spark configurations such as executor memory, number of executors, and shuffle partitions.

• Optimize code by reducing unnecessary shuffles, using efficient transformations, and avoiding unnecessary data movements.

• Utilize caching to store intermediate results in memory and avoid recomputation.

• Example: In my projec...read more

SparkContext is the main entry point for Spark functionality, while SparkSession is the entry point for Spark SQL.

• SparkContext is the entry point for low-level API functionality in Spark.

• SparkSession is the entry point for Spark SQL functionality.

• SparkContext is used to create RDDs (Resilient Distributed Datasets) in Spark.

• SparkSession provides a unified entry point for reading data from various sources and performing

When a spark job is submitted, various steps are executed at the backend to process the job.

• The job is submitted to the Spark driver program.

• The driver program communicates with the cluster manager to request resources.

• The cluster manager allocates resources (CPU, memory) to the job.

• The driver program creates DAG (Directed Acyclic Graph) of the job stages and tasks.

• Tasks are then scheduled and executed on worker nodes ...read more

Calculate second highest salary using SQL and pyspark

• Use SQL query with ORDER BY and LIMIT to get the second highest salary

• In pyspark, use orderBy() and take() functions to achieve the same result

The two types of modes for Spark architecture are standalone mode and cluster mode.

• Standalone mode: Spark runs on a single machine with a single JVM and is suitable for development and testing.

• Cluster mode: Spark runs on a cluster of machines managed by a cluster manager like YARN or Mesos for production workloads.

Client mode is better for very less latency due to direct communication with the cluster.

• Client mode allows direct communication with the cluster, reducing latency.

• Standalone mode requires an additional layer of communication, increasing latency.

• Client mode is preferred for real-time applications where low latency is crucial.