HiFx Data Engineer Interview Questions and Answers

Q1. What is lazy evaluation in spark.
Ans. 
Lazy evaluation in Spark delays the execution of transformations until an action is called.
• Lazy evaluation allows Spark to optimize the execution plan by combining multiple transformations into a single stage.

• Transformations are not executed immediately, but are stored as a directed acyclic graph (DAG) of operations.

• Actions trigger the execution of the DAG and produce results.

• Example: map() and filter() are transformat...read more
Answered by AI
Add your answer
Q2. What is mapreduce
Ans. 
MapReduce is a programming model and processing technique for parallel and distributed computing.
• MapReduce is used to process large datasets in parallel across a distributed cluster of computers.

• It consists of two main functions - Map function for processing key/value pairs and Reduce function for aggregating the results.

• Popularly used in big data processing frameworks like Hadoop for tasks like data sorting, searching...read more
Answered by AI
Add your answer

Q1. What is skewness and skewd tables
Ans. 
Skewness is a measure of asymmetry in a distribution. Skewed tables are tables with imbalanced data distribution.
• Skewness is a statistical measure that describes the asymmetry of the data distribution around the mean.

• Positive skewness indicates a longer tail on the right side of the distribution, while negative skewness indicates a longer tail on the left side.

• Skewed tables in data engineering refer to tables with imba...read more
Answered by AI
Add your answer

Q1. What is spark and explain working
Ans. 
Spark is a distributed computing framework designed for big data processing.
• Spark is built around the concept of Resilient Distributed Datasets (RDDs) which allow for fault-tolerant parallel processing of data.

• It provides high-level APIs in Java, Scala, Python, and R for ease of use.

• Spark can run on top of Hadoop, Mesos, Kubernetes, or in standalone mode.

• It includes modules for SQL, streaming, machine learning, and gra...read more
Answered by AI
Add your answer

Q1. PartitionBy vs bucketBy in spark
Add your answer

Q1. Explain error handling in PySpark
Ans. 
Error handling in PySpark involves using try-except blocks and logging to handle exceptions and errors.
• Use try-except blocks to catch and handle exceptions in PySpark code

• Utilize logging to record errors and exceptions for debugging purposes

• Consider using the .option('mode', 'PERMISSIVE') method to handle corrupt records in data processing
Answered by AI
Add your answer

Q1. Data Warehousing related questions
Add your answer

Q1. Data Modelling related questions
Add your answer

Coding questions on sql python and spark

Q1. Hacker rank code pair
Ans. 
Implement a function to pair elements of an array based on a given sum.
• Iterate through the array and check if the current element plus any other element equals the given sum.

• Use a hash set to store elements already visited to avoid duplicate pairs.

• Return an array of arrays containing the pairs that sum up to the given value.
Answered by AI
Add your answer
Q2. Pyspark questions
Add your answer

Q1. Money you want for this position
Add your answer
Q2. Why to choose this position and not another
Add your answer

2 questions based on Algorithm and logic

Q1. Questions on basic python, hadoop, sql etc
Add your answer

Q1. About past experience
Add your answer
Q2. Python Coding question about list, dataframe etc
Add your answer
Q3. RDBMS Questions and SQL theory
Add your answer

7 sql questions of easy to medium , 2 python questions, 1 pyspark question

Q1. Scenatio based, project questions
Add your answer