10+ Xora Software Systems Interview Questions and Answers

Only one job will run in parallel in Spark with four cores and four worker nodes.

• In Spark, each core can only run one task at a time, so with four cores, only four tasks can run concurrently.

• Since there are four worker nodes, each with four cores, a total of four tasks can run in parallel.

• Therefore, only one job will run in parallel in this scenario.

I have used techniques like indexing, query optimization, and parallel processing in my projects.

• Indexing: Used to improve the speed of data retrieval by creating indexes on columns frequently used in queries.

• Query optimization: Rewriting queries to improve efficiency and reduce execution time.

• Parallel processing: Distributing tasks across multiple processors to speed up data processing.

• Caching: Storing frequently accessed data in memory to reduce the need for repeated retrie...read more

Use SQL to calculate the difference in marks for each student ID across different years.

• Use a self join on the table to compare marks for the same student ID across different years.

• Calculate the difference in marks by subtracting the marks from different years.

• Group the results by student ID to get the difference in marks for each student.

The answer to the question is that in which state which gender makes the most purchases.

• Aggregate the data by state and gender to calculate the total purchases made by each gender in each state.

• Identify the gender with the highest total purchases in each state.

• Present the results in a table or chart for easy visualization.

Spark handles fault tolerance through resilient distributed datasets (RDDs) and lineage tracking.

• Spark achieves fault tolerance through RDDs, which are immutable distributed collections of objects that can be rebuilt if a partition is lost.

• RDDs track the lineage of transformations applied to the data, allowing lost partitions to be recomputed based on the original data and transformations.

• Spark also replicates data partitions across multiple nodes to ensure availability in ca...read more

ADF stands for Azure Data Factory, a cloud-based data integration service that allows you to create, schedule, and manage data pipelines.

• ADF is used for building, scheduling, and monitoring data pipelines to move and transform data from various sources to destinations.

• It supports data integration between various data stores such as Azure SQL Database, Azure Blob Storage, and on-premises data sources.

• ADF provides a visual interface for designing and monitoring data pipelines, ...read more

DAG stands for Directed Acyclic Graph, a data structure used to represent dependencies between tasks in a workflow.

• DAG is a collection of nodes connected by edges, where each edge has a direction and there are no cycles.

• It is commonly used in data engineering for representing data pipelines and workflows.

• DAGs help in visualizing and optimizing the order of tasks to be executed in a workflow.

• Popular tools like Apache Airflow use DAGs to define and schedule data pipelines.

Lineage refers to the history and origin of data, including its source, transformations, and dependencies.

• Lineage helps in understanding how data is generated, processed, and transformed throughout its lifecycle.

• It tracks the flow of data from its source to its destination, including any intermediate steps or transformations.

• Lineage is important for data governance, data quality, and troubleshooting data issues.

• Examples of lineage tools include Apache Atlas, Informatica Metad...read more

Identify duplicate rows in a table

• Use SQL query with GROUP BY and HAVING clause to identify duplicate rows based on specific columns

• Example: SELECT column1, column2, COUNT(*) FROM table_name GROUP BY column1, column2 HAVING COUNT(*) > 1

Joins are used to combine data from two or more tables based on a related column between them.

• Inner Join: returns only the matching rows from both tables

• Left Join: returns all rows from the left table and matching rows from the right table

• Right Join: returns all rows from the right table and matching rows from the left table

• Full Outer Join: returns all rows from both tables

• Cross Join: returns the Cartesian product of both tables

Joins combine rows from two or more tables based on a related column, while unions combine rows from two or more tables with the same structure.

• Joins are used to combine data from different tables based on a related column

• Unions are used to combine data from tables with the same structure

• Joins can be inner, left, right, or full, while unions are always a combination of all rows

• Joins can have multiple conditions, while unions require the same number of columns and data types

Use PySpark code to connect data to ADLS with partitioning

• Use SparkSession to create a Spark application

• Set the configuration for ADLS storage account and container

• Read data from ADLS using Spark DataFrame API

• Partition the data based on a specific column while writing back to ADLS