Ernst & Young Senior Data Engineer Interview Questions, Process, and Tips

Q1. What is spark architecture
Ans. 
Spark architecture refers to the structure of Apache Spark, a distributed computing framework.
• Spark architecture consists of a cluster manager, worker nodes, and a driver program.

• The cluster manager allocates resources and schedules tasks across worker nodes.

• Worker nodes execute tasks in parallel and store data in memory or disk.

• The driver program coordinates the execution of tasks and manages the overall workflow.

• Spar...read more
Answered by AI
Add your answer
Q2. Optimizations of spark jobs
Ans. 
Optimizing Spark jobs involves tuning configurations, partitioning data, using appropriate data structures, and leveraging caching.
• Tune Spark configurations for optimal performance

• Partition data to distribute workload evenly

• Use appropriate data structures like DataFrames or Datasets

• Leverage caching to avoid recomputation

• Optimize shuffle operations to reduce data movement
Answered by AI
Add your answer

It was also very easy.

Q1. What is pyspark architecture
Ans. 
PySpark architecture is a distributed computing framework that combines Python and Spark to process big data.
• PySpark architecture consists of a driver program, cluster manager, and worker nodes.

• The driver program is responsible for creating SparkContext, which connects to the cluster manager.

• Cluster manager allocates resources and schedules tasks on worker nodes.

• Worker nodes execute the tasks and return results to the ...read more
Answered by AI
Add your answer
Q2. What is skewed partitioning
Ans. 
Skewed partitioning is when data is not evenly distributed across partitions, leading to performance issues.
• Skewed partitioning can occur when a key column has a few values that are much more common than others.

• It can lead to uneven processing and resource utilization in distributed systems like Hadoop or Spark.

• To address skewed partitioning, techniques like data skew detection, data skew handling, and data skew preven
Answered by AI
Add your answer

Q1. What are the various layers of mdm?
Ans. 
MDM (Master Data Management) typically consists of three layers: operational, analytical, and data governance.
• Operational layer: manages the day-to-day data operations and transactions.

• Analytical layer: focuses on data analysis and reporting for decision-making.

• Data governance layer: ensures data quality, security, and compliance.

• Example: In a retail company, the operational layer manages customer transactions, the ana...read more
Answered by AI
Add your answer

Q1. Tell me why do you wanna grab this oppurtunity
Add your answer

Mostly coding questions on Python, pyspark and SQL of medium difficulty.

Q1. Manager round with technical and details about the project.
Add your answer

Q1. Compensation and other questions Int he company.
Add your answer