Lazy evaluation & the DAG
Apache Spark · Spark Architecture
Theory
Spark transformations are lazy: calling filter, select, withColumn, join or groupBy does not process any data. Spark only records the step and returns a new DataFrame that describes the result. The recorded steps form a DAG (directed acyclic graph) of operations, also called the lineage. Work starts only when you call an action such as show(), count(), collect() or write, and at that moment Spark optimizes the whole DAG and runs it.
Because Spark sees the full chain before running it, it can push filters down to the file scan, read only the needed columns, combine steps and skip work whose result is never used. The lineage also gives fault tolerance: if an executor is lost, Spark recomputes only the missing partitions from the source. For example, orders.filter(amount > 20000).select('region') returns immediately even on a terabyte of data; nothing is read until you call .count().
Why we use it
Laziness lets Spark optimize the whole pipeline instead of one step at a time, and recover lost data by replaying lineage. It also explains surprises: errors in data appear at the action, and the same DataFrame used twice is computed twice unless you cache it.
When to use it
Keep it in mind when debugging (the failing line may be the action, not the transformation), when timing code (time the action) and when reusing a DataFrame in several actions, where cache() or persist() avoids recomputing the DAG.
How it works
- Write transformations (filter, select, join)
- Spark records each step in the logical plan (DAG)
- Was an action called? Yes: Optimize the whole DAG and run it. No: Keep waiting; no data is read.
- Executors read only the needed data and run tasks
- Return or write the result
Syntax
df2 = df.filter(...).select(...) # lazy: only builds the plan
df2.explain() # look at the plan, no job runs
df2.count() # action: the DAG executes now
df2.cache() # keep results for reuseExample
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.master("local[1]")
.config("spark.ui.enabled", "false")
.getOrCreate())
orders = spark.createDataFrame([
(101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(102, 2, "West", "Phone", 2, 30000.0, "2026-01-06"),
(103, 1, "North", "Mouse", 3, 1500.0, "2026-01-06"),
(104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(105, 4, "West", "Tablet", 1, 25000.0, "2026-01-08"),
(106, 2, "East", "Phone", 1, 15000.0, "2026-01-08"),
(107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
(108, 3, "South", "Mouse", 4, 2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
"qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
calls = spark.sparkContext.accumulator(0)
@F.udf("double")
def with_gst(amount):
calls.add(1) # counts how many times the UDF really runs
return amount * 1.18
big = (orders.filter(F.col("amount") > 20000)
.withColumn("amount_gst", with_gst("amount"))
.select("order_id", "amount_gst"))
print("after transformations, UDF calls:", calls.value) # still 0: nothing ran
print("rows:", big.count()) # action runs the DAG...
print("after count(), UDF calls:", calls.value) # ...but amount_gst was never needed
result = big.collect() # this action needs amount_gst
print("after collect(), UDF calls:", calls.value)
print(result[0])
# Output:
# after transformations, UDF calls: 0
# rows: 5
# after count(), UDF calls: 0
# after collect(), UDF calls: 5
# Row(order_id=101, amount_gst=64900.0)
Key points
- Transformations only build a plan; actions execute it
- The DAG/lineage lets Spark optimize the whole chain and recompute lost partitions
- Each action re-runs the DAG from the source unless the DataFrame is cached
- Runtime data errors surface at the action, not where the transformation is written
- explain() shows the plan without running a job
- Spark skips unused work: count() never ran the UDF because amount_gst was not needed