Deloitte Interview Questions and Answers

reduceBy is used to aggregate data based on key, while groupBy is used to group data based on key.

• reduceBy is a transformation that combines the values of each key using an associative function and a neutral 'zero value'.

• groupBy is a transformation that groups the data based on a key and returns a grouped data set.

• reduceBy is more efficient for aggregating data as it reduces the data before shuffling, while groupBy shuffles all the data before grouping.

• reduceBy is typically u...read more

RDD is a low-level abstraction representing a distributed collection of objects, while DataFrame is a higher-level abstraction representing a distributed collection of data organized into named columns.

• RDD is more suitable for unstructured data and low-level transformations, while DataFrame is more suitable for structured data and high-level abstractions.

• DataFrames provide optimizations like query optimization and code generation, while RDDs do not.

• DataFrames support SQL quer...read more

PySpark is a Python API for Apache Spark, used for big data processing and analytics.

• PySpark is a Python API for Apache Spark, a fast and general-purpose cluster computing system.

• It allows for easy integration with Python libraries and provides high-level APIs in Python.

• PySpark can be used for processing large datasets, machine learning, real-time data streaming, and more.

• It supports various data sources such as HDFS, Apache Hive, JSON, Parquet, and more.

• PySpark is widely use...read more

The different modes of execution in Apache Spark include local mode, standalone mode, YARN mode, and Mesos mode.

• Local mode: Spark runs on a single machine with one executor.

• Standalone mode: Spark runs on a cluster managed by a standalone cluster manager.

• YARN mode: Spark runs on a Hadoop cluster using YARN as the resource manager.

• Mesos mode: Spark runs on a Mesos cluster with Mesos as the resource manager.

Apache Spark architecture includes a cluster manager, worker nodes, and driver program.

• Apache Spark architecture consists of a cluster manager, which allocates resources and schedules tasks.

• Worker nodes execute tasks and store data in memory or disk.

• Driver program coordinates tasks and communicates with the cluster manager.

• Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object.

• Data is processed in parallel across the worke...read more

PySpark is a Python API for Apache Spark, while Python is a general-purpose programming language.

• PySpark is specifically designed for big data processing using Spark, while Python is a versatile programming language used for various applications.

• PySpark allows for distributed computing and parallel processing, while Python is primarily used for sequential programming.

• PySpark provides libraries and tools for working with large datasets efficiently, while Python may require add...read more