Sizing executors & the driver
Apache Spark · Performance Tuning
Theory
Sizing means choosing how many executors to run, how many cores each gets and how much memory each gets. A widely used rule of thumb for YARN-style clusters is: leave 1 core and about 1 GB per node for the operating system and daemons, give each executor about 5 cores (much more hurts I/O and GC; 1 core wastes memory on JVM overhead), and subtract about 10% of each executor's share for memoryOverhead. Total task slots are executors x cores, and a good partition count is 2 to 3 times the slots.
For example, 10 nodes with 16 cores and 64 GB each give 15 usable cores, so 3 executors of 5 cores per node; 63 GB / 3 = 21 GB per container, about 19 GB heap after overhead; 30 executors minus 1 for the YARN application master leaves 29, which is 145 task slots. The driver usually needs far less: a few GB, unless you collect() large results or broadcast big tables, which are also limited by spark.driver.maxResultSize.
Why we use it
Too few, oversized executors waste the cluster and suffer GC pauses; too many tiny ones waste memory on JVM overhead and cannot hold broadcasts. Sizing questions are a staple of Spark interviews and the first thing to check for slow or failing jobs.
When to use it
Use the rule of thumb as a starting point for self-managed YARN, Kubernetes or standalone clusters, then adjust from the Spark UI. On managed platforms you mostly choose node types and autoscaling instead. Revisit sizing when data volume changes a lot.
How it works
- Know nodes, cores per node and memory per node
- Subtract 1 core and ~1 GB per node for the OS
- Executors per node = usable cores / 5
- Memory per executor = usable memory / executors per node, minus ~10% overhead
- Total executors = nodes x per node, minus 1 for the application master
- Do tasks spill or hit GC often? Yes: Give more memory or more partitions. No: Keep the settings.
- Set shuffle partitions to ~2-3x total cores
Syntax
--num-executors 29 --executor-cores 5 --executor-memory 19g
--driver-memory 4g --conf spark.driver.maxResultSize=2g
--conf spark.sql.shuffle.partitions=290 # ~2-3x total cores
# spark.executor.instances / spark.executor.cores / spark.executor.memoryExample
from pyspark import SparkConf
from pyspark.sql import SparkSession
def size_executors(nodes, cores_per_node, mem_gb_per_node, cores_per_executor=5):
"""Classic rule of thumb for YARN-style clusters."""
usable_cores = cores_per_node - 1 # leave 1 core for OS / node daemons
usable_mem = mem_gb_per_node - 1 # leave ~1 GB for the OS
per_node = usable_cores // cores_per_executor # executors per node
mem_per_exec = usable_mem / per_node # memory per executor container
heap = int(mem_per_exec / 1.10) # remove ~10% memoryOverhead
executors = nodes * per_node - 1 # leave 1 slot for the YARN AM / driver
return {"executors": executors, "cores": cores_per_executor, "memory_gb": heap,
"total_slots": executors * cores_per_executor}
plan = size_executors(nodes=10, cores_per_node=16, mem_gb_per_node=64)
print(plan)
print("suggested shuffle partitions (2-3x slots):", plan["total_slots"] * 2, "to", plan["total_slots"] * 3)
# Turn the plan into Spark settings (what you would pass to spark-submit --conf)
cluster_conf = (SparkConf()
.set("spark.executor.instances", str(plan["executors"]))
.set("spark.executor.cores", str(plan["cores"]))
.set("spark.executor.memory", f"{plan['memory_gb']}g")
.set("spark.driver.memory", "4g")
.set("spark.driver.maxResultSize", "2g")
.set("spark.sql.shuffle.partitions", str(plan["total_slots"] * 2)))
for k, v in sorted(cluster_conf.getAll()):
print(f"--conf {k}={v}")
# Locally the same idea applies: local[2] = 1 JVM with 2 task slots
spark = (SparkSession.builder.master("local[2]")
.config("spark.ui.enabled", "false")
.config("spark.sql.shuffle.partitions", "4") # ~2x the 2 local slots
.getOrCreate())
sc = spark.sparkContext
print("local task slots:", sc.defaultParallelism)
spark.conf.set("spark.sql.adaptive.enabled", "false") # show the raw setting (AQE may merge)
df = spark.range(1000)
print("partitions after groupBy:", df.groupBy((df.id % 10).alias("k")).count().rdd.getNumPartitions())
# Output:
# {'executors': 29, 'cores': 5, 'memory_gb': 19, 'total_slots': 145}
# suggested shuffle partitions (2-3x slots): 290 to 435
# --conf spark.driver.maxResultSize=2g
# --conf spark.driver.memory=4g
# --conf spark.executor.cores=5
# --conf spark.executor.instances=29
# --conf spark.executor.memory=19g
# --conf spark.sql.shuffle.partitions=290
# local task slots: 2
# partitions after groupBy: 4
Key points
- Leave ~1 core and ~1 GB per node for the OS and daemons
- ~5 cores per executor is the usual sweet spot
- Executor heap = container share / 1.1 (overhead is ~10%)
- Task slots = executors x cores; partitions ~2-3x slots
- Driver: a few GB unless you collect or broadcast large data
- Treat the rule as a start; tune with the Spark UI