Memory management: unified memory, off-heap & spill

Apache Spark · Performance Tuning

Theory

An executor's JVM heap is spark.executor.memory. Spark first reserves 300 MB, then splits the rest: spark.memory.fraction (0.6 by default) becomes unified memory; the other 40% is user memory for your objects and metadata. Unified memory is shared by execution (shuffles, joins, sorts, aggregations) and storage (cached data and broadcasts). spark.memory.storageFraction (0.5) marks the part of storage that execution cannot evict, but each side can borrow free space from the other, and execution can evict cached blocks above that line.

Outside the heap, spark.executor.memoryOverhead (by default the larger of 384 MB and 10% of executor memory) covers native memory and Python workers; the container size is heap plus overhead. Optional off-heap memory (spark.memory.offHeap.*) keeps Tungsten data outside the garbage-collected heap. When execution memory runs out, Spark spills sorted or hashed data to local disk instead of failing. For example, an 8 GB executor gets (8192 - 300) x 0.6 = 4735 MB of unified memory.

Why we use it

Out-of-memory errors, containers killed for exceeding memory limits, long garbage-collection pauses and heavy spill are the most common Spark production problems. Knowing where each byte goes tells you which setting to change.

When to use it

Look here when the Spark UI shows spill, long GC time or lost executors. Raise memoryOverhead for heavy Python UDF or pandas work, add memory or partitions for spill, and cache only what is reused. Leave fraction settings at their defaults unless you have measured a reason.

How it works

  1. A task needs memory to sort, hash or join
  2. Free execution memory? Yes: Use it. No: Borrow storage or evict cached blocks.
  3. Still not enough memory? Yes: Spill sorted data to local disk. No: Continue in memory.
  4. Merge spilled files when the task finishes
  5. Container exceeds heap + overhead? Yes: Cluster manager kills the executor. No: Task completes.
  6. Task returns its result

Syntax

--executor-memory 8g                       # spark.executor.memory (heap)
--conf spark.executor.memoryOverhead=2g    # off-heap overhead, Python workers
--conf spark.memory.fraction=0.6           # unified share of (heap - 300 MB)
--conf spark.memory.storageFraction=0.5    # storage protected from eviction
--conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=2g
df.persist(StorageLevel.MEMORY_AND_DISK)

Example

from pyspark import StorageLevel
from pyspark.sql import SparkSession

spark = (SparkSession.builder.master("local[2]")
         .config("spark.ui.enabled", "false")
         .config("spark.memory.fraction", "0.6")          # the default, set explicitly
         .config("spark.memory.storageFraction", "0.5")   # the default, set explicitly
         .getOrCreate())
conf = spark.sparkContext.getConf()

# 1) Unified memory for one executor JVM heap (spark.executor.memory)
def unified_memory(heap_mb, fraction, storage_fraction, reserved_mb=300):
    usable = heap_mb - reserved_mb                  # 300 MB reserved for Spark internals
    unified = usable * fraction                     # shared by execution + storage
    return {"user (your objects, UDF state)": round(usable - unified),
            "unified (execution + storage)": round(unified),
            "  storage protected from eviction": round(unified * storage_fraction),
            "  execution can borrow the rest": round(unified * (1 - storage_fraction))}

fraction = float(conf.get("spark.memory.fraction"))
storage = float(conf.get("spark.memory.storageFraction"))
for part, mb in unified_memory(8 * 1024, fraction, storage).items():   # an 8g executor
    print(f"{part:36s} {mb:>5} MB")

# 2) Outside the heap: overhead (Python workers, JVM native memory) and optional off-heap
heap_mb = 8 * 1024
overhead_mb = max(384, int(heap_mb * 0.10))         # default overhead rule
print("container size ~", heap_mb + overhead_mb, "MB (heap + overhead)")
print("off-heap enabled:", conf.get("spark.memory.offHeap.enabled", "false"))

# 3) Storage memory in action: cached blocks that do not fit spill to disk
orders = spark.range(100000).withColumnRenamed("id", "order_id")
orders.persist(StorageLevel.MEMORY_AND_DISK)        # the DataFrame default level
orders.count()
print("storage level:", orders.storageLevel)
orders.unpersist()

# Output:
# user (your objects, UDF state)        3157 MB
# unified (execution + storage)         4735 MB
#   storage protected from eviction     2368 MB
#   execution can borrow the rest       2368 MB
# container size ~ 9011 MB (heap + overhead)
# off-heap enabled: false
# storage level: Disk Memory Serialized 1x Replicated

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic