AstaGuru Auction House Interview Questions and Answers

Explaining partitioning, bucketing, joins, optimization, broadcast variables, ORC vs Parquet, RDD vs Dataframe, project architecture and responsibilities for Big Data Engineer role.

• Partitioning is dividing data into smaller chunks for parallel processing, while bucketing is organizing data into buckets based on a hash function.

• Types of joins in Spark include inner join, outer join, left join, right join, and full outer join.

• Optimization techniques in Spark include caching, re...read more

Spark can handle upserts using merge() function

• Use merge() function to handle upserts in Spark

• Specify the primary key column(s) to identify matching rows

• Specify the update column(s) to update existing rows

• Specify the insert column(s) to insert new rows

• Example: df1.merge(df2, on='id', whenMatched='update', whenNotMatched='insert')

Remove duplicate records and find 5th highest salary department wise using SQL.

• Use DISTINCT keyword to remove duplicate records.

• Use GROUP BY clause to group the records by department.

• Use ORDER BY clause to sort the salaries in descending order.

• Use LIMIT clause to get the 5th highest salary.

• Combine all the above clauses to get the desired result.

Spark memory optimisation techniques

• Use broadcast variables to reduce memory usage

• Use persist() or cache() to store RDDs in memory

• Use partitioning to reduce shuffling and memory usage

• Use off-heap memory to avoid garbage collection overhead

• Tune memory settings such as spark.driver.memory and spark.executor.memory

Hadoop serialisation techniques are used to convert data into a format that can be stored and processed in Hadoop.

• Hadoop uses Writable interface for serialisation and deserialisation of data

• Avro, Thrift, and Protocol Buffers are popular serialisation frameworks used in Hadoop

• Serialisation can be customised using custom Writable classes or external libraries

• Serialisation plays a crucial role in Hadoop performance and efficiency