Epsilon Data Engineer Interview Questions and Answers

Q1. Basic Spark, Python, SQL Questions.
Add your answer
Q2. Some Data Modelling Questions.
Add your answer

Basic aptitude questions

Q1. SQL basics questions like joins
Add your answer
Q2. Pandas basics for data manipulation
Add your answer

Q1. SQL and tableau
Add your answer

Online test on sql join sub query

Q1. Final tech round
Add your answer

Q1. Focus more on AWS
Add your answer
Q2. Spark basic concept
Add your answer

Python coding and SQL questions.

Q1. Spark Concepts, architecture, tuning
Add your answer
Q2. Optimization Techniques
Add your answer

Aws, scala and cloud concept

Q1. Python question
Add your answer

Q1. Spark Context vs Spark Session
Ans. 
Spark Context is the entry point to any Spark functionality while Spark Session is a unified entry point for Spark 2.0+.
• Spark Context is the old entry point to Spark functionality.

• Spark Session is a unified entry point for Spark 2.0+.

• Spark Context is used to create RDDs, accumulators and broadcast variables.

• Spark Session is used to create DataFrames, execute SQL queries and read data from external sources.
Answered by AI
Add your answer
Q2. Repartitioning vs Coalesce
Ans. 
Repartitioning increases partitions while Coalesce reduces partitions.
• Repartitioning shuffles data across the cluster and can be used to increase parallelism.

• Coalesce merges partitions without shuffling data and can be used to reduce overhead.

• Repartitioning is expensive and should be used sparingly.

• Coalesce is faster but may not be as effective as repartitioning in increasing parallelism.

• Both can be used to optimize da
Answered by AI
Add your answer
Q3. Sql query to find Second Highest Salary
Ans. 
Sql query to find Second Highest Salary
• Use ORDER BY and LIMIT to select the second highest salary

• Use subquery to select the maximum salary and exclude it from the result set

• Handle cases where there are ties for the highest salary
Answered by AI
Add your answer
Q4. Internal Working of Spark
Ans. 
Spark is a distributed computing engine that processes large datasets in parallel across a cluster of computers.
• Spark uses a master-slave architecture with a driver program that coordinates tasks across worker nodes.

• Data is stored in Resilient Distributed Datasets (RDDs) that can be cached in memory for faster processing.

• Spark supports multiple programming languages including Java, Scala, and Python.

• Spark can be used f...read more
Answered by AI
Add your answer
Q5. What is Braodcast Join
Ans. 
Broadcast Join is a technique used in distributed computing to optimize join operations.
• Broadcast Join is used when one table is small enough to fit in memory of all nodes in a cluster.

• The smaller table is broadcasted to all nodes in the cluster, reducing network traffic.

• Broadcast Join is faster than other join techniques when used appropriately.

• Example: Joining a small reference table with a large fact table in a data
Answered by AI
Add your answer