When a spark job is submitted, what happens at backend. Explain the flow.

Question

Accepted Answer

When a spark job is submitted, various steps are executed at the backend to process the job.

The job is submitted to the Spark driver program.
The driver program communicates with the cluster manager to request resources.
The cluster manager allocates resources (CPU, memory) to the job.
The driver program creates DAG (Directed Acyclic Graph) of the job stages and tasks.
Tasks are then scheduled and executed on worker nodes in the cluster.
Intermediate results are stored in memory or disk depending on the job requirements.
Once all tasks are completed, the final result is returned to the driver program.

Accepted Answer

User submits a spark job through  Spark submit. The job is submitted to Spark driver.
SPark driver initialises and creates spark context. Spark context create the environment for execution. 
Then Spark driver parse the job into small tasks like map reduce and join etc 
SPark driver request resources from cluster manager. Then cluster manager allocate resources and assigns the container.
Now spark driver allocates tasks to executors which runs on containers.
Once all the tasks are completed. Executors will do any shuttle shift , repartition with subsequent tasks and provides the result to spark driver. And it will be reported to the user

When a spark job is submitted, what happens at backend. Explain the flow.

Top LTIMindtree Data Engineer interview questions & answers

Popular interview questions of Data Engineer

Top HR questions asked in LTIMindtree Data Engineer