Celebal Technologies Big Data Engineer Interview Questions and Answers

It was WeCP based test

Q1. Explain Spark Architecture in detail
Ans. 
Spark Architecture is a distributed computing framework that provides high-level APIs for in-memory computing.
• Spark Architecture consists of a cluster manager, worker nodes, and a driver program.

• It uses Resilient Distributed Datasets (RDDs) for fault-tolerant distributed data processing.

• Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object.

• It supports various data ...read more
Answered by AI
Add your answer
Q2. Explain higher order function, closure, anonymous function, map, flatmap, tail recursion
Ans. 
Higher order functions, closures, anonymous functions, map, flatmap, and tail recursion are key concepts in functional programming.
• Higher order function: Functions that can take other functions as arguments or return functions as results.

• Closure: Functions that capture variables from their lexical scope, even when they are called outside that scope.

• Anonymous function: Functions without a specified name, often used as a...read more
Answered by AI
Add your answer

Q1. Explain pyspark architecture
Ans. 
PySpark architecture is based on the Apache Spark architecture, with additional components for Python integration.
• PySpark architecture includes Spark Core, Spark SQL, Spark Streaming, MLlib, and GraphX.

• It allows Python developers to interact with Spark using PySpark API.

• PySpark architecture enables distributed processing of large datasets using RDDs and DataFrames.

• It leverages the power of in-memory processing for fast...read more
Answered by AI
Add your answer

Q1. Scala , Hadoop Spark, SQL, sqoop
Add your answer
Q2. Scala Hadoop Spark SQL sqoop
Add your answer

Q1. Scala Hadoop Spark SQL sqoop
Add your answer

Questions on software and system designs

Helps employer identify particular personality traits like leadership, confidence, interpersonal and teamwork skills of potential employees

Q1. Why do you want this job? Again, companies want to hire people who are passionate about the job, so you should have a great answer about why you want the ...
Add your answer
Q2. This opportunity can help me in many ways to boost my confidence as well as drive my career forward
Add your answer

Q1. All type of joins with example
Ans. 
Different types of joins in SQL with examples
• Inner Join: Returns rows when there is a match in both tables

• Left Join: Returns all rows from the left table and the matched rows from the right table

• Right Join: Returns all rows from the right table and the matched rows from the left table

• Full Outer Join: Returns all rows when there is a match in either table
Answered by AI
Add your answer
Q2. How to handle large spark datasets
Ans. 
Large Spark datasets can be handled by partitioning, caching, optimizing transformations, and tuning resources.
• Partitioning data to distribute workload evenly across nodes

• Caching frequently accessed data to avoid recomputation

• Optimizing transformations to reduce unnecessary processing

• Tuning resources like memory allocation and parallelism for optimal performance
Answered by AI
Add your answer

Q1. How do you tune sparks configuration setting's to optimize query performance
Ans. 
Spark configuration settings can be tuned to optimize query performance by adjusting parameters like memory allocation, parallelism, and caching.
• Increase executor memory and cores to allow for more parallel processing

• Adjust shuffle partitions to optimize data shuffling during joins and aggregations

• Enable dynamic allocation to scale resources based on workload demands

• Utilize caching to store intermediate results and avo...read more
Answered by AI
Add your answer
Q2. What strategies do you use to handle data skew and partition imbalance in spark
Ans. 
To handle data skew and partition imbalance in Spark, strategies include using salting, bucketing, repartitioning, and optimizing join operations.
• Use salting to evenly distribute skewed keys across partitions

• Implement bucketing to pre-partition data based on a specific column

• Repartition data based on a specific key to balance partitions

• Optimize join operations by broadcasting small tables or using partitioning strategi
Answered by AI
Add your answer