Virtusa Consulting Services Big Data Engineer Interview Questions, Process, and Tips

Use 'hdfs diskbalancer' command to check disk utilisation and health in Hadoop

• Run 'hdfs diskbalancer -report' to get a report on disk utilisation

• Use 'hdfs diskbalancer -plan <path>' to generate a plan for balancing disk usage

• Check the Hadoop logs for any disk health issues

Spark Architecture consists of Driver, Cluster Manager, and Executors. Driver manages the execution of Spark jobs.

• Driver: Manages the execution of Spark jobs, converts user code into tasks, and coordinates with Cluster Manager.

• Cluster Manager: Manages resources across the cluster and allocates resources to Spark applications.

• Executors: Execute tasks assigned by the Driver and store data in memory or disk for further pr...read more

Optimization techniques in Spark improve performance and efficiency of data processing.

• Partitioning data to distribute workload evenly

• Caching frequently accessed data in memory

• Using broadcast variables for small lookup tables

• Avoiding shuffling operations whenever possible

• Tuning memory settings and garbage collection parameters

I am unable to provide this information as it is confidential.

• Confidential information about salaries in previous organizations should not be disclosed.

• It is important to respect the privacy and confidentiality of past employers.

• Discussing specific salary details may not be appropriate in a professional setting.

To create a pivot table in SQL from a non-pivot table, you can use the CASE statement with aggregate functions.

• Use the CASE statement to categorize data into columns

• Apply aggregate functions like SUM, COUNT, AVG, etc. to calculate values for each category

• Group the data by the columns you want to pivot on

Creating triggers in a database involves defining the trigger, specifying the event that will activate it, and writing the code to be executed.

• Define the trigger using the CREATE TRIGGER statement

• Specify the event that will activate the trigger (e.g. INSERT, UPDATE, DELETE)

• Write the code or actions to be executed when the trigger is activated

• Test the trigger to ensure it functions as intended

Spark optimization techniques improve performance and efficiency of Spark applications.

• Partitioning data to reduce shuffling

• Caching frequently used data

• Using broadcast variables for small data

• Using efficient data formats like Parquet

• Tuning memory and CPU usage

• Using appropriate cluster size

• Avoiding unnecessary data shuffling

• Using appropriate serialization formats

• Using appropriate join strategies

We use Hadoop Distributed File System (HDFS) for our project.

• HDFS is a distributed file system designed to run on commodity hardware.

• It provides high-throughput access to application data and is fault-tolerant.

• HDFS is used by many big data processing frameworks like Hadoop, Spark, etc.

• It stores data in a distributed manner across multiple nodes in a cluster.

• HDFS is optimized for large files and sequential reads and wri