Oswaal Books and Learning Private Limited Interview Questions and Answers

IR stands for Integration Runtime. Dataset is a representation of data, while linked service is a connection to the data source.

• IR is a compute infrastructure used to provide data integration capabilities

• Dataset is a structured representation of data used in data engineering tasks

• Linked service is a connection to a data source, providing access to the data

• IR enables data movement and transformation between different data sources

• Dataset defines the schema and structure of the ...read more

Use regular expression to remove special characters from a string

• Use the regex pattern [^a-zA-Z0-9\s] to match any character that is not a letter, digit, or whitespace

• Use the replace() function in your programming language to replace the matched special characters with an empty string

• Example: input string 'Hello! How are you?' will become 'Hello How are you' after removing special characters

Use Google Cloud Storage to load CSV data into BigQuery

• Upload the CSV file to Google Cloud Storage

• Create a BigQuery table with the appropriate schema

• Use the 'bq load' command to load the data from the CSV file into the BigQuery table

Rank assigns unique rank to each row, while dense rank assigns consecutive ranks without gaps.

• Rank leaves gaps in rank sequence if there are ties, while dense rank does not

• Rank function is used to assign a unique rank to each row based on a specified column

• Dense rank function is used to assign consecutive ranks to rows without any gaps

• Example: If there are two rows with rank 1 in a dataset, rank function will assign 1 and 2, while dense rank will assign 1 and 1

Optimization techniques in Spark

• Partitioning data to optimize data locality

• Caching frequently accessed data

• Using broadcast variables for small data sets

• Using appropriate data structures and algorithms

• Avoiding unnecessary shuffling of data