Impetus Technologies Big Data Engineer Interview Questions, Process, and Tips

Spark can handle upserts using merge() function

• Use merge() function to handle upserts in Spark

• Specify the primary key column(s) to identify matching rows

• Specify the update column(s) to update existing rows

• Specify the insert column(s) to insert new rows

• Example: df1.merge(df2, on='id', whenMatched='update', whenNotMatched='insert')

Java collection is a single interface while collections is a utility class.

• Java collection is an interface that provides a unified architecture for manipulating and storing groups of objects.

• Collections is a utility class that provides static methods for working with collections.

• Java collection is a part of the Java Collections Framework while collections is not.

• Examples of Java collections include List, Set, and Map w...read more

Spark memory optimisation techniques

• Use broadcast variables to reduce memory usage

• Use persist() or cache() to store RDDs in memory

• Use partitioning to reduce shuffling and memory usage

• Use off-heap memory to avoid garbage collection overhead

• Tune memory settings such as spark.driver.memory and spark.executor.memory

Hadoop serialisation techniques are used to convert data into a format that can be stored and processed in Hadoop.

• Hadoop uses Writable interface for serialisation and deserialisation of data

• Avro, Thrift, and Protocol Buffers are popular serialisation frameworks used in Hadoop

• Serialisation can be customised using custom Writable classes or external libraries

• Serialisation plays a crucial role in Hadoop performance and ef