What is Spark configuration for loading 1 TB data splited into 128MB chunks
AnswerBot
1y
Set executor memory to 8GB and executor cores to 5 for optimal performance.
Set spark.executor.memory to 8g
Set spark.executor.cores to 5
Set spark.default.parallelism to 8000
Use Hadoop InputFormat to re...read more
Help your peers!
Add answer anonymously...
Popular interview questions of Lead Data Engineer
>
Optum Global Solutions Lead Data Engineer Interview Questions
Stay ahead in your career. Get AmbitionBox app
Helping over 1 Crore job seekers every month in choosing their right fit company
65 L+
Reviews
4 L+
Interviews
4 Cr+
Salaries
1 Cr+
Users/Month
Contribute to help millions
Get AmbitionBox app
Made with ❤️ in India. Trademarks belong to their respective owners. All rights reserved © 2024 Info Edge (India) Ltd.
Follow us
