Memory management: unified memory, off-heap & spill
Apache Spark · Performance Tuning
Theory
An executor's JVM heap is spark.executor.memory. Spark first reserves 300 MB, then splits the rest: spark.memory.fraction (0.6 by default) becomes unified memory; the other 40% is user memory for your objects and metadata. Unified memory is shared by execution (shuffles, joins, sorts, aggregations) and storage (cached data and broadcasts). spark.memory.storageFraction (0.5) marks the part of storage that execution cannot evict, but each side can borrow free space from the other, and execution can evict cached blocks above that line.
Outside the heap, spark.executor.memoryOverhead (by default the larger of 384 MB and 10% of executor memory) covers native memory and Python workers; the container size is heap plus overhead. Optional off-heap memory (spark.memory.offHeap.*) keeps Tungsten data outside the garbage-collected heap. When execution memory runs out, Spark spills sorted or hashed data to local disk instead of failing. For example, an 8 GB executor gets (8192 - 300) x 0.6 = 4735 MB of unified memory.
Why we use it
Out-of-memory errors, containers killed for exceeding memory limits, long garbage-collection pauses and heavy spill are the most common Spark production problems. Knowing where each byte goes tells you which setting to change.
When to use it
Look here when the Spark UI shows spill, long GC time or lost executors. Raise memoryOverhead for heavy Python UDF or pandas work, add memory or partitions for spill, and cache only what is reused. Leave fraction settings at their defaults unless you have measured a reason.
How it works
- A task needs memory to sort, hash or join
- Free execution memory? Yes: Use it. No: Borrow storage or evict cached blocks.
- Still not enough memory? Yes: Spill sorted data to local disk. No: Continue in memory.
- Merge spilled files when the task finishes
- Container exceeds heap + overhead? Yes: Cluster manager kills the executor. No: Task completes.
- Task returns its result
Syntax
--executor-memory 8g # spark.executor.memory (heap)
--conf spark.executor.memoryOverhead=2g # off-heap overhead, Python workers
--conf spark.memory.fraction=0.6 # unified share of (heap - 300 MB)
--conf spark.memory.storageFraction=0.5 # storage protected from eviction
--conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=2g
df.persist(StorageLevel.MEMORY_AND_DISK)Example
from pyspark import StorageLevel
from pyspark.sql import SparkSession
spark = (SparkSession.builder.master("local[2]")
.config("spark.ui.enabled", "false")
.config("spark.memory.fraction", "0.6") # the default, set explicitly
.config("spark.memory.storageFraction", "0.5") # the default, set explicitly
.getOrCreate())
conf = spark.sparkContext.getConf()
# 1) Unified memory for one executor JVM heap (spark.executor.memory)
def unified_memory(heap_mb, fraction, storage_fraction, reserved_mb=300):
usable = heap_mb - reserved_mb # 300 MB reserved for Spark internals
unified = usable * fraction # shared by execution + storage
return {"user (your objects, UDF state)": round(usable - unified),
"unified (execution + storage)": round(unified),
" storage protected from eviction": round(unified * storage_fraction),
" execution can borrow the rest": round(unified * (1 - storage_fraction))}
fraction = float(conf.get("spark.memory.fraction"))
storage = float(conf.get("spark.memory.storageFraction"))
for part, mb in unified_memory(8 * 1024, fraction, storage).items(): # an 8g executor
print(f"{part:36s} {mb:>5} MB")
# 2) Outside the heap: overhead (Python workers, JVM native memory) and optional off-heap
heap_mb = 8 * 1024
overhead_mb = max(384, int(heap_mb * 0.10)) # default overhead rule
print("container size ~", heap_mb + overhead_mb, "MB (heap + overhead)")
print("off-heap enabled:", conf.get("spark.memory.offHeap.enabled", "false"))
# 3) Storage memory in action: cached blocks that do not fit spill to disk
orders = spark.range(100000).withColumnRenamed("id", "order_id")
orders.persist(StorageLevel.MEMORY_AND_DISK) # the DataFrame default level
orders.count()
print("storage level:", orders.storageLevel)
orders.unpersist()
# Output:
# user (your objects, UDF state) 3157 MB
# unified (execution + storage) 4735 MB
# storage protected from eviction 2368 MB
# execution can borrow the rest 2368 MB
# container size ~ 9011 MB (heap + overhead)
# off-heap enabled: false
# storage level: Disk Memory Serialized 1x Replicated
Key points
- Heap = 300 MB reserved + user memory + unified (execution + storage)
- Execution and storage borrow from each other inside unified memory
- Overhead = max(384 MB, 10% of heap) by default; Python workers live here
- Container killed for memory: raise memoryOverhead, not just heap
- Spill = data written to local disk when execution memory is full
- Off-heap memory avoids GC pressure but must be enabled and sized