TCS Pyspark Developer Interview Questions and Answers

Hadoop + Spark MCQ online test

Q1. Spark Architecture
Add your answer
Q2. Transformations vs Actions
Ans. 
Transformations are lazy operations that create new RDDs, while Actions are operations that trigger computation and return results.
• Transformations are operations like map, filter, and reduceByKey that create a new RDD from an existing one.

• Actions are operations like count, collect, and saveAsTextFile that trigger computation on an RDD and return results.

• Transformations are lazy and are only executed when an action is c...read more
Answered by AI
Add your answer

Q1. Why Spark is used?
Ans. 
Spark is used for big data processing due to its speed, scalability, and ease of use.
• Spark is used for processing large volumes of data quickly and efficiently.

• It offers in-memory processing which makes it faster than traditional MapReduce.

• Spark provides a wide range of libraries for diverse tasks like SQL, streaming, machine learning, and graph processing.

• It can run on various platforms like Hadoop, Kubernetes, and st...read more
Answered by AI
Add your answer
Q2. What are RDDs and DataFrames
Ans. 
RDDs and DataFrames are data structures in Apache Spark for processing and analyzing large datasets.
• RDDs (Resilient Distributed Datasets) are the fundamental data structure of Spark, representing a collection of elements that can be operated on in parallel.

• DataFrames are distributed collections of data organized into named columns, similar to a table in a relational database.

• DataFrames are built on top of RDDs, providi...read more
Answered by AI
Add your answer

1. Find duplicate
2. 2,3 highest salary

Basic python coding, list, dict, generators etc

Q1. Salary negotiation
Add your answer

Q1. Conceptual questions
Add your answer

Q1. What is the difference between coalesce and repartition in data processing?
Ans. 
Coalesce reduces the number of partitions without shuffling data, while repartition reshuffles data to create a specific number of partitions.
• Coalesce is used to reduce the number of partitions without shuffling data

• Repartition is used to increase or decrease the number of partitions by shuffling data

• Coalesce is more efficient when reducing partitions as it avoids shuffling

• Repartition is useful when you need to explici...read more
Answered by AI
Add your answer
Q2. What is the difference between a DataFrame and an RDD (Resilient Distributed Dataset)?
Ans. 
DataFrame is a higher-level abstraction built on top of RDD, providing more structure and optimization capabilities.
• DataFrames are distributed collections of data organized into named columns, similar to tables in a relational database.

• RDDs are lower-level abstractions representing a collection of objects distributed across a cluster, with no inherent structure.

• DataFrames provide optimizations like query optimization a...read more
Answered by AI
Add your answer

Q1. Spark architecture
Add your answer
Q2. Word count program
Ans. 
A program to count the occurrences of each word in a text document.
• Use Spark RDD to read the text file and split the lines into words

• Apply transformations like map and reduceByKey to count the occurrences of each word

• Handle punctuation and case sensitivity to ensure accurate word count results
Answered by AI
Add your answer
Q3. Azure integration services
Add your answer
Q4. Azure linked services vs Azure dataset
Ans. 
Azure linked services are connections to external data sources, while Azure datasets are structured data objects within Azure Data Factory.
• Azure linked services are used to connect to external data sources such as databases, storage accounts, and SaaS applications.

• Azure datasets are structured data objects within Azure Data Factory that represent data from linked services or other sources.

• Linked services define the con...read more
Answered by AI
Add your answer