10+ MCI-IIT-JEE Interview Questions and Answers

UDF stands for User-Defined Function in Spark. It allows users to define their own functions to process data.

• UDFs can be written in different programming languages like Python, Scala, and Java.

• UDFs can be used to perform complex operations on data that are not available in built-in functions.

• PySpark code to check the validity of mobile_number column can be written using regular expressions and the `regexp_extract` function.

• Example: `df.select('mobile_number', regexp_extract('...read more

Merge two unsorted lists into a sorted list using inbuilt sorting functions.

• Use inbuilt sorting functions to sort the input lists

• Merge the sorted lists using a merge algorithm

• Return the merged and sorted list

SQL query to retrieve the second highest salary from each department

• Use the RANK() function to assign a rank to each salary within each department

• Filter the results to only include rows with a rank of 2

• Group the results by department to get the second highest salary for each department

To delete duplicate rows from a table, use the DISTINCT keyword or GROUP BY clause.

• Use the DISTINCT keyword to select unique rows from the table.

• Use the GROUP BY clause to group the rows by a specific column and select the unique rows.

• Use the DELETE statement with a subquery to delete the duplicate rows.

• Create a new table with the unique rows and drop the old table.

Window functions are used to perform calculations across a set of rows that are related to the current row.

• Commonly used window functions include ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD, FIRST_VALUE, LAST_VALUE, and NTILE.

• Window functions are used in conjunction with the OVER clause to define the window or set of rows to perform the calculation on.

• Window functions can be used to calculate running totals, moving averages, and other aggregate calculations.

• Window functions are s...read more

Null values in PySpark are handled using functions such as dropna(), fillna(), and replace().

• dropna() function is used to drop rows or columns with null values

• fillna() function is used to fill null values with a specified value or method

• replace() function is used to replace null values with a specified value

• coalesce() function is used to replace null values with the first non-null value in a list of columns

Imputer function in PySpark is used to replace missing values in a DataFrame.

• Imputer is a transformer in PySpark ML library.

• It replaces missing values in a DataFrame with either mean, median, or mode of the column.

• It can be used with both numerical and categorical columns.

• Example: imputer = Imputer(inputCols=['col1', 'col2'], outputCols=['col1_imputed', 'col2_imputed'], strategy='mean')

• Example: imputed_df = imputer.fit(df).transform(df)

Lazy evaluation in Spark delays the execution of transformations until an action is called.

• Lazy evaluation allows Spark to optimize the execution plan by combining multiple transformations into a single stage.

• Transformations are not executed immediately, but are stored as a directed acyclic graph (DAG) of operations.

• Actions trigger the execution of the DAG and produce results.

• Example: map() and filter() are transformations that are lazily evaluated until an action like collec...read more

Skewness is a measure of asymmetry in a distribution. Skewed tables are tables with imbalanced data distribution.

• Skewness is a statistical measure that describes the asymmetry of the data distribution around the mean.

• Positive skewness indicates a longer tail on the right side of the distribution, while negative skewness indicates a longer tail on the left side.

• Skewed tables in data engineering refer to tables with imbalanced data distribution, which can impact query performan...read more

Spark is a distributed computing framework designed for big data processing.

• Spark is built around the concept of Resilient Distributed Datasets (RDDs) which allow for fault-tolerant parallel processing of data.

• It provides high-level APIs in Java, Scala, Python, and R for ease of use.

• Spark can run on top of Hadoop, Mesos, Kubernetes, or in standalone mode.

• It includes modules for SQL, streaming, machine learning, and graph processing.

• Spark uses in-memory processing to speed up ...read more