TCS Interview Questions and Answers

Only one job will run in parallel in Spark with four cores and four worker nodes.

• In Spark, each core can only run one task at a time, so with four cores, only four tasks can run concurrently.

• Since there are four worker nodes, each with four cores, a total of four tasks can run in parallel.

• Therefore, only one job will run in parallel in this scenario.

I have used techniques like indexing, query optimization, and parallel processing in my projects.

• Indexing: Used to improve the speed of data retrieval by creating indexes on columns frequently used in queries.

• Query optimization: Rewriting queries to improve efficiency and reduce execution time.

• Parallel processing: Distributing tasks across multiple processors to speed up data processing.

• Caching: Storing frequently accessed data in memory to reduce the need for repeated retrie...read more

Use SQL to calculate the difference in marks for each student ID across different years.

• Use a self join on the table to compare marks for the same student ID across different years.

• Calculate the difference in marks by subtracting the marks from different years.

• Group the results by student ID to get the difference in marks for each student.

The answer to the question is that in which state which gender makes the most purchases.

• Aggregate the data by state and gender to calculate the total purchases made by each gender in each state.

• Identify the gender with the highest total purchases in each state.

• Present the results in a table or chart for easy visualization.

Spark handles fault tolerance through resilient distributed datasets (RDDs) and lineage tracking.

• Spark achieves fault tolerance through RDDs, which are immutable distributed collections of objects that can be rebuilt if a partition is lost.

• RDDs track the lineage of transformations applied to the data, allowing lost partitions to be recomputed based on the original data and transformations.

• Spark also replicates data partitions across multiple nodes to ensure availability in ca...read more

ADF stands for Azure Data Factory, a cloud-based data integration service that allows you to create, schedule, and manage data pipelines.

• ADF is used for building, scheduling, and monitoring data pipelines to move and transform data from various sources to destinations.

• It supports data integration between various data stores such as Azure SQL Database, Azure Blob Storage, and on-premises data sources.

• ADF provides a visual interface for designing and monitoring data pipelines, ...read more

DAG stands for Directed Acyclic Graph, a data structure used to represent dependencies between tasks in a workflow.

• DAG is a collection of nodes connected by edges, where each edge has a direction and there are no cycles.

• It is commonly used in data engineering for representing data pipelines and workflows.

• DAGs help in visualizing and optimizing the order of tasks to be executed in a workflow.

• Popular tools like Apache Airflow use DAGs to define and schedule data pipelines.

Lineage refers to the history and origin of data, including its source, transformations, and dependencies.

• Lineage helps in understanding how data is generated, processed, and transformed throughout its lifecycle.

• It tracks the flow of data from its source to its destination, including any intermediate steps or transformations.

• Lineage is important for data governance, data quality, and troubleshooting data issues.

• Examples of lineage tools include Apache Atlas, Informatica Metad...read more