Caching & Persistence

Apache Spark · Performance Tuning

Theory

Spark is lazy: each action (count, show, write) re-runs the whole plan from the source. If you reuse the same expensive DataFrame several times, df.cache() tells Spark to keep it after it is first computed, so later actions read it from memory instead of recomputing joins and filters. For DataFrames, cache() is persist() with a memory-and-disk storage level (PySpark shows it as MEMORY_AND_DISK_DESER): partitions that do not fit in memory spill to local disk. persist(level) lets you pick another, such as DISK_ONLY.

Caching is itself lazy: nothing is stored until an action runs, so people often call count() once to fill the cache. Cached data is stored in a compressed columnar format and appears in the Storage tab of the Spark UI. Free it with df.unpersist() when you are done, because cached blocks use executor memory that joins and aggregations also need. spark.catalog.cacheTable(name) does the same for tables.

Why we use it

Iterative work, such as exploring a cleaned dataset, running several aggregations on one joined table or training ML models over many passes, would otherwise repeat the same expensive reads and shuffles every time. Caching turns those repeats into fast memory reads.

When to use it

Cache a DataFrame that is expensive to compute and reused by two or more actions in the same application. Do not cache data used once, data that is cheap to re-read from columnar files, or data larger than the cluster can hold. Always unpersist when finished.

How it works

  1. An expensive DataFrame (joins, filters)
  2. Will two or more actions reuse it? Yes: cache() or persist(level). No: Do not cache; let Spark recompute.
  3. First action computes the plan and stores partitions
  4. Later actions read from the cache (InMemoryTableScan)
  5. unpersist() when finished
  6. Memory is free for other work

Syntax

df.cache()                               # memory and disk for DataFrames
from pyspark import StorageLevel
df.persist(StorageLevel.DISK_ONLY)
df.count()                               # action fills the cache
df.is_cached; df.storageLevel
df.unpersist()
spark.catalog.cacheTable("orders"); spark.catalog.uncacheTable("orders")

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

orders = spark.createDataFrame([
    (1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (2, 102, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (3, 101, "North", "Mouse",  3,  1500.0, "2026-01-07"),
    (4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (5, 102, "West",  "Tablet", 1, 25000.0, "2026-01-09"),
    (6, 104, "North", "Phone",  1, 18000.0, "2026-01-10"),
    (7, 103, "South", "Mouse",  2,  1000.0, "2026-01-11"),
    (8, 105, "West",  "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

customers = spark.createDataFrame([
    (101, "Asha",  "Jaipur", "Retail"),
    (102, "Ravi",  "Pune",   "Corporate"),
    (103, "Meera", "Delhi",  "Retail"),
    (104, "Karan", "Jaipur", "SMB"),
    (105, "Priya", "Mumbai", "Corporate"),
], "customer_id INT, name STRING, city STRING, segment STRING")

from pyspark import StorageLevel

enriched = (orders.join(customers, "customer_id")
                  .withColumn("gst", F.col("amount") * 0.18))

enriched.cache()                     # lazy: marks the plan for caching
print("is_cached:", enriched.is_cached, "| level:", enriched.storageLevel)
enriched.count()                     # first action computes and stores it

# These reuse the cached data instead of re-running the join
enriched.groupBy("city").agg(F.sum("amount").alias("sales")).orderBy("city").show()
enriched.groupBy("segment").agg(F.count("*").alias("orders")).orderBy("segment").show()
enriched.explain()                   # plan now starts from InMemoryTableScan

enriched.unpersist()
print("after unpersist:", enriched.is_cached)

# Choosing a storage level explicitly
by_disk = orders.persist(StorageLevel.DISK_ONLY)
print(by_disk.storageLevel)
by_disk.unpersist()

# Output:
# is_cached: True | level: Disk Memory Deserialized 1x Replicated
# +------+-------+
# |  city|  sales|
# +------+-------+
# | Delhi|53000.0|
# |Jaipur|74500.0|
# |Mumbai|60000.0|
# |  Pune|55000.0|
# +------+-------+
#
# +---------+------+
# |  segment|orders|
# +---------+------+
# |Corporate|     3|
# |   Retail|     4|
# |      SMB|     1|
# +---------+------+
#
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=false
# +- InMemoryTableScan [customer_id#1, order_id#0, region#2, product#3, qty#4, amount#5, order_date#7, name#9, city#10, segment#11, gst#12]
#       +- InMemoryRelation [customer_id#1, order_id#0, region#2, product#3, qty#4, amount#5, order_date#7, name#9, city#10, segment#11, gst#12], StorageLevel(disk, memory, deserialized, 1 replicas)
#             +- AdaptiveSparkPlan isFinalPlan=true
#                +- == Final Plan ==
#                   ResultQueryStage 2
#                   +- *(5) Project [customer_id#1, order_id#0, region#2, product#3, qty#4, amount#5, order_date#7, name#9, city#10, segment#11, (amount#5 * 0.18) AS gst#12]
#                      +- *(5) SortMergeJoin [customer_id#1], [customer_id#8], Inner
#                         :- *(3) Sort [customer_id#1 ASC NULLS FIRST], false, 0
#                         :  +- ShuffleQueryStage 0
#                         :     +- Exchange hashpartitioning(customer_id#1, 200), ENSURE_REQUIREMENTS, [plan_id=65]
#                         :        +- *(1) Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7]
#                         :           +- *(1) Filter isnotnull(customer_id#1)
#                         :              +- *(1) Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#                         +- *(4) Sort [customer_id#8 ASC NULLS FIRST], false, 0
#                            +- ShuffleQueryStage 1
#                               +- Exchange hashpartitioning(customer_id#8, 200), ENSURE_REQUIREMENTS, [plan_id=78]
#                                  +- *(2) Filter isnotnull(customer_id#8)
#                                     +- *(2) Scan ExistingRDD[customer_id#8,name#9,city#10,segment#11]
#                +- == Initial Plan ==
#                   Project [customer_id#1, order_id#0, region#2, product#3, qty#4, amount#5, order_date#7, name#9, city#10, segment#11, (amount#5 * 0.18) AS gst#12]
#                   +- SortMergeJoin [customer_id#1], [customer_id#8], Inner
#                      :- Sort [customer_id#1 ASC NULLS FIRST], false, 0
#                      :  +- Exchange hashpartitioning(customer_id#1, 200), ENSURE_REQUIREMENTS, [plan_id=29]
#                      :     +- Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7]
#                      :        +- Filter isnotnull(customer_id#1)
#                      :           +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#                      +- Sort [customer_id#8 ASC NULLS FIRST], false, 0
#                         +- Exchange hashpartitioning(customer_id#8, 200), ENSURE_REQUIREMENTS, [plan_id=30]
#                            +- Filter isnotnull(customer_id#8)
#                               +- Scan ExistingRDD[customer_id#8,name#9,city#10,segment#11]
#
#
# after unpersist: False
# Disk Serialized 1x Replicated

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic