State Street Corporation Data Engineer Interview Questions and Answers

Q1. What is lazy evaluation in spark.
Ans. 
Lazy evaluation in Spark delays the execution of transformations until an action is called.
• Lazy evaluation allows Spark to optimize the execution plan by combining multiple transformations into a single stage.

• Transformations are not executed immediately, but are stored as a directed acyclic graph (DAG) of operations.

• Actions trigger the execution of the DAG and produce results.

• Example: map() and filter() are transformat...read more
Answered by AI
Add your answer
Q2. What is mapreduce
Ans. 
MapReduce is a programming model and processing technique for parallel and distributed computing.
• MapReduce is used to process large datasets in parallel across a distributed cluster of computers.

• It consists of two main functions - Map function for processing key/value pairs and Reduce function for aggregating the results.

• Popularly used in big data processing frameworks like Hadoop for tasks like data sorting, searching...read more
Answered by AI
Add your answer

Q1. What is skewness and skewd tables
Ans. 
Skewness is a measure of asymmetry in a distribution. Skewed tables are tables with imbalanced data distribution.
• Skewness is a statistical measure that describes the asymmetry of the data distribution around the mean.

• Positive skewness indicates a longer tail on the right side of the distribution, while negative skewness indicates a longer tail on the left side.

• Skewed tables in data engineering refer to tables with imba...read more
Answered by AI
Add your answer

Q1. What is spark and explain working
Ans. 
Spark is a distributed computing framework designed for big data processing.
• Spark is built around the concept of Resilient Distributed Datasets (RDDs) which allow for fault-tolerant parallel processing of data.

• It provides high-level APIs in Java, Scala, Python, and R for ease of use.

• Spark can run on top of Hadoop, Mesos, Kubernetes, or in standalone mode.

• It includes modules for SQL, streaming, machine learning, and gra...read more
Answered by AI
Add your answer

Q1. PartitionBy vs bucketBy in spark
Add your answer

Q1. Advanced sql on ctl
Add your answer
Q2. Python data structure
Add your answer

Coding questions on sql python and spark

Q1. Hacker rank code pair
Ans. 
Implement a function to pair elements of an array based on a given sum.
• Iterate through the array and check if the current element plus any other element equals the given sum.

• Use a hash set to store elements already visited to avoid duplicate pairs.

• Return an array of arrays containing the pairs that sum up to the given value.
Answered by AI
Add your answer
Q2. Pyspark questions
Add your answer

Q1. Money you want for this position
Add your answer
Q2. Why to choose this position and not another
Add your answer

2 questions based on Algorithm and logic

Q1. Questions on basic python, hadoop, sql etc
Add your answer

Q1. About past experience
Add your answer
Q2. Python Coding question about list, dataframe etc
Add your answer
Q3. RDBMS Questions and SQL theory
Add your answer

Q1. Explain about projects completed during academics and role performed?
Ans. 
During my academics, I completed a project on analyzing customer behavior using machine learning techniques.
• Developed a predictive model to forecast customer churn based on historical data

• Used Python for data preprocessing, feature engineering, and model building

• Collaborated with a team to gather requirements and present findings to stakeholders
Answered by AI
Add your answer
Q2. SQL based questions
Add your answer