Interview Questions and Answers

We use Hadoop Distributed File System (HDFS) for our project.

• HDFS is a distributed file system designed to run on commodity hardware.

• It provides high-throughput access to application data and is fault-tolerant.

• HDFS is used by many big data processing frameworks like Hadoop, Spark, etc.

• It stores data in a distributed manner across multiple nodes in a cluster.

• HDFS is optimized for large files and sequential reads and writes.

Use 'hdfs diskbalancer' command to check disk utilisation and health in Hadoop

• Run 'hdfs diskbalancer -report' to get a report on disk utilisation

• Use 'hdfs diskbalancer -plan ' to generate a plan for balancing disk usage

• Check the Hadoop logs for any disk health issues

To create a pivot table in SQL from a non-pivot table, you can use the CASE statement with aggregate functions.

• Use the CASE statement to categorize data into columns

• Apply aggregate functions like SUM, COUNT, AVG, etc. to calculate values for each category

• Group the data by the columns you want to pivot on

Creating triggers in a database involves defining the trigger, specifying the event that will activate it, and writing the code to be executed.

• Define the trigger using the CREATE TRIGGER statement

• Specify the event that will activate the trigger (e.g. INSERT, UPDATE, DELETE)

• Write the code or actions to be executed when the trigger is activated

• Test the trigger to ensure it functions as intended

Optimization techniques in Spark improve performance and efficiency of data processing.

• Partitioning data to distribute workload evenly

• Caching frequently accessed data in memory

• Using broadcast variables for small lookup tables

• Avoiding shuffling operations whenever possible

• Tuning memory settings and garbage collection parameters