Lazy evaluation & the DAG

Apache Spark · Spark Architecture

Theory

Spark transformations are lazy: calling filter, select, withColumn, join or groupBy does not process any data. Spark only records the step and returns a new DataFrame that describes the result. The recorded steps form a DAG (directed acyclic graph) of operations, also called the lineage. Work starts only when you call an action such as show(), count(), collect() or write, and at that moment Spark optimizes the whole DAG and runs it.

Because Spark sees the full chain before running it, it can push filters down to the file scan, read only the needed columns, combine steps and skip work whose result is never used. The lineage also gives fault tolerance: if an executor is lost, Spark recomputes only the missing partitions from the source. For example, orders.filter(amount > 20000).select('region') returns immediately even on a terabyte of data; nothing is read until you call .count().

Why we use it

Laziness lets Spark optimize the whole pipeline instead of one step at a time, and recover lost data by replaying lineage. It also explains surprises: errors in data appear at the action, and the same DataFrame used twice is computed twice unless you cache it.

When to use it

Keep it in mind when debugging (the failing line may be the action, not the transformation), when timing code (time the action) and when reusing a DataFrame in several actions, where cache() or persist() avoids recomputing the DAG.

How it works

  1. Write transformations (filter, select, join)
  2. Spark records each step in the logical plan (DAG)
  3. Was an action called? Yes: Optimize the whole DAG and run it. No: Keep waiting; no data is read.
  4. Executors read only the needed data and run tasks
  5. Return or write the result

Syntax

df2 = df.filter(...).select(...)   # lazy: only builds the plan
df2.explain()                      # look at the plan, no job runs
df2.count()                        # action: the DAG executes now
df2.cache()                        # keep results for reuse

Example

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[1]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())

orders = spark.createDataFrame([
    (101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (102, 2, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (103, 1, "North", "Mouse",  3,  1500.0, "2026-01-06"),
    (104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (105, 4, "West",  "Tablet", 1, 25000.0, "2026-01-08"),
    (106, 2, "East",  "Phone",  1, 15000.0, "2026-01-08"),
    (107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
    (108, 3, "South", "Mouse",  4,  2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
   "qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

calls = spark.sparkContext.accumulator(0)

@F.udf("double")
def with_gst(amount):
    calls.add(1)                 # counts how many times the UDF really runs
    return amount * 1.18

big = (orders.filter(F.col("amount") > 20000)
             .withColumn("amount_gst", with_gst("amount"))
             .select("order_id", "amount_gst"))
print("after transformations, UDF calls:", calls.value)   # still 0: nothing ran

print("rows:", big.count())                               # action runs the DAG...
print("after count(), UDF calls:", calls.value)           # ...but amount_gst was never needed

result = big.collect()                                    # this action needs amount_gst
print("after collect(), UDF calls:", calls.value)
print(result[0])

# Output:
# after transformations, UDF calls: 0
# rows: 5
# after count(), UDF calls: 0
# after collect(), UDF calls: 5
# Row(order_id=101, amount_gst=64900.0)

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic