How do u calculate the resources based on the number of cores and memory given..ex - 16 cores with 64 gb... What is overhead, driver memory etc

Question

Accepted Answer

Calculating resources based on cores and memory given with overhead and driver memory

Calculate the total memory available by multiplying the number of cores with memory per core
Deduct the overhead memory required for the operating system and other processes
Allocate driver memory for each executor based on the workload
Consider the memory requirements for other services like Hadoop, Spark, etc.
Example: For 16 cores with 64 GB memory, assuming 1 GB overhead and 2 GB driver memory per executor, we get 14 executors with 4 GB memory each

Top Infosys Senior Data Engineer interview questions & answers