Caching & Persistence
Apache Spark · Performance Tuning
Theory
Spark is lazy: each action (count, show, write) re-runs the whole plan from the source. If you reuse the same expensive DataFrame several times, df.cache() tells Spark to keep it after it is first computed, so later actions read it from memory instead of recomputing joins and filters. For DataFrames, cache() is persist() with a memory-and-disk storage level (PySpark shows it as MEMORY_AND_DISK_DESER): partitions that do not fit in memory spill to local disk. persist(level) lets you pick another, such as DISK_ONLY.
Caching is itself lazy: nothing is stored until an action runs, so people often call count() once to fill the cache. Cached data is stored in a compressed columnar format and appears in the Storage tab of the Spark UI. Free it with df.unpersist() when you are done, because cached blocks use executor memory that joins and aggregations also need. spark.catalog.cacheTable(name) does the same for tables.
Why we use it
Iterative work, such as exploring a cleaned dataset, running several aggregations on one joined table or training ML models over many passes, would otherwise repeat the same expensive reads and shuffles every time. Caching turns those repeats into fast memory reads.
When to use it
Cache a DataFrame that is expensive to compute and reused by two or more actions in the same application. Do not cache data used once, data that is cheap to re-read from columnar files, or data larger than the cluster can hold. Always unpersist when finished.
How it works
- An expensive DataFrame (joins, filters)
- Will two or more actions reuse it? Yes: cache() or persist(level). No: Do not cache; let Spark recompute.
- First action computes the plan and stores partitions
- Later actions read from the cache (InMemoryTableScan)
- unpersist() when finished
- Memory is free for other work
Syntax
df.cache() # memory and disk for DataFrames
from pyspark import StorageLevel
df.persist(StorageLevel.DISK_ONLY)
df.count() # action fills the cache
df.is_cached; df.storageLevel
df.unpersist()
spark.catalog.cacheTable("orders"); spark.catalog.uncacheTable("orders")Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
orders = spark.createDataFrame([
(1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(2, 102, "West", "Phone", 2, 30000.0, "2026-01-06"),
(3, 101, "North", "Mouse", 3, 1500.0, "2026-01-07"),
(4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(5, 102, "West", "Tablet", 1, 25000.0, "2026-01-09"),
(6, 104, "North", "Phone", 1, 18000.0, "2026-01-10"),
(7, 103, "South", "Mouse", 2, 1000.0, "2026-01-11"),
(8, 105, "West", "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
customers = spark.createDataFrame([
(101, "Asha", "Jaipur", "Retail"),
(102, "Ravi", "Pune", "Corporate"),
(103, "Meera", "Delhi", "Retail"),
(104, "Karan", "Jaipur", "SMB"),
(105, "Priya", "Mumbai", "Corporate"),
], "customer_id INT, name STRING, city STRING, segment STRING")
from pyspark import StorageLevel
enriched = (orders.join(customers, "customer_id")
.withColumn("gst", F.col("amount") * 0.18))
enriched.cache() # lazy: marks the plan for caching
print("is_cached:", enriched.is_cached, "| level:", enriched.storageLevel)
enriched.count() # first action computes and stores it
# These reuse the cached data instead of re-running the join
enriched.groupBy("city").agg(F.sum("amount").alias("sales")).orderBy("city").show()
enriched.groupBy("segment").agg(F.count("*").alias("orders")).orderBy("segment").show()
enriched.explain() # plan now starts from InMemoryTableScan
enriched.unpersist()
print("after unpersist:", enriched.is_cached)
# Choosing a storage level explicitly
by_disk = orders.persist(StorageLevel.DISK_ONLY)
print(by_disk.storageLevel)
by_disk.unpersist()
# Output:
# is_cached: True | level: Disk Memory Deserialized 1x Replicated
# +------+-------+
# | city| sales|
# +------+-------+
# | Delhi|53000.0|
# |Jaipur|74500.0|
# |Mumbai|60000.0|
# | Pune|55000.0|
# +------+-------+
#
# +---------+------+
# | segment|orders|
# +---------+------+
# |Corporate| 3|
# | Retail| 4|
# | SMB| 1|
# +---------+------+
#
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=false
# +- InMemoryTableScan [customer_id#1, order_id#0, region#2, product#3, qty#4, amount#5, order_date#7, name#9, city#10, segment#11, gst#12]
# +- InMemoryRelation [customer_id#1, order_id#0, region#2, product#3, qty#4, amount#5, order_date#7, name#9, city#10, segment#11, gst#12], StorageLevel(disk, memory, deserialized, 1 replicas)
# +- AdaptiveSparkPlan isFinalPlan=true
# +- == Final Plan ==
# ResultQueryStage 2
# +- *(5) Project [customer_id#1, order_id#0, region#2, product#3, qty#4, amount#5, order_date#7, name#9, city#10, segment#11, (amount#5 * 0.18) AS gst#12]
# +- *(5) SortMergeJoin [customer_id#1], [customer_id#8], Inner
# :- *(3) Sort [customer_id#1 ASC NULLS FIRST], false, 0
# : +- ShuffleQueryStage 0
# : +- Exchange hashpartitioning(customer_id#1, 200), ENSURE_REQUIREMENTS, [plan_id=65]
# : +- *(1) Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7]
# : +- *(1) Filter isnotnull(customer_id#1)
# : +- *(1) Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
# +- *(4) Sort [customer_id#8 ASC NULLS FIRST], false, 0
# +- ShuffleQueryStage 1
# +- Exchange hashpartitioning(customer_id#8, 200), ENSURE_REQUIREMENTS, [plan_id=78]
# +- *(2) Filter isnotnull(customer_id#8)
# +- *(2) Scan ExistingRDD[customer_id#8,name#9,city#10,segment#11]
# +- == Initial Plan ==
# Project [customer_id#1, order_id#0, region#2, product#3, qty#4, amount#5, order_date#7, name#9, city#10, segment#11, (amount#5 * 0.18) AS gst#12]
# +- SortMergeJoin [customer_id#1], [customer_id#8], Inner
# :- Sort [customer_id#1 ASC NULLS FIRST], false, 0
# : +- Exchange hashpartitioning(customer_id#1, 200), ENSURE_REQUIREMENTS, [plan_id=29]
# : +- Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7]
# : +- Filter isnotnull(customer_id#1)
# : +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
# +- Sort [customer_id#8 ASC NULLS FIRST], false, 0
# +- Exchange hashpartitioning(customer_id#8, 200), ENSURE_REQUIREMENTS, [plan_id=30]
# +- Filter isnotnull(customer_id#8)
# +- Scan ExistingRDD[customer_id#8,name#9,city#10,segment#11]
#
#
# after unpersist: False
# Disk Serialized 1x Replicated
Key points
- DataFrame cache() = persist() with a memory-and-disk level; spills to disk if memory is full
- Caching is lazy: the first action fills it
- Only worth it when the same DataFrame is reused by several actions
- Cached data competes with execution memory: unpersist when done
- Check the Storage tab or explain() (InMemoryTableScan) to confirm it is used