Transformations vs actions
Apache Spark · Spark Architecture
Theory
Every DataFrame or RDD operation is either a transformation or an action. A transformation takes a DataFrame and returns a new DataFrame without running anything: select, filter/where, withColumn, join, groupBy().agg, orderBy, distinct, union and limit are all transformations. DataFrames are immutable, so a transformation never changes the original; it creates a new plan on top of it.
An action asks for a real result, so it starts a Spark job: show(), count(), collect(), take(n), first(), toPandas(), foreach and every df.write...save() are actions. A quick test: if the method returns a DataFrame it is a transformation; if it returns a Python value, prints, or writes to storage, it is an action. For example, orders.filter('region = "North"') returns a DataFrame instantly, while .count() on it scans the data and returns the number 3.
Why we use it
Telling them apart tells you when Spark will do work and how much. Too many actions in a loop re-run the same pipeline again and again, and actions like collect() can overload the driver. It is one of the most common Spark interview questions.
When to use it
Chain as many transformations as you need and call as few actions as possible, ideally one write at the end of a pipeline. Use show(), take(n) or limit for quick looks, and avoid collect() or toPandas() on large data.
How it works
- Call a method on a DataFrame
- Does it return a new DataFrame? Yes: Transformation: add a step to the plan, no job. No: Action: Spark starts a job.
- Optimizer plans the job
- Executors compute the result
- Value returned, printed or written
Syntax
Transformations: select, filter, where, withColumn, join, groupBy().agg, orderBy, distinct, union, limit
Actions: show, count, collect, take, first, head, toPandas, foreach, write.save / saveAsTableExample
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.master("local[1]")
.config("spark.ui.enabled", "false")
.getOrCreate())
orders = spark.createDataFrame([
(101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(102, 2, "West", "Phone", 2, 30000.0, "2026-01-06"),
(103, 1, "North", "Mouse", 3, 1500.0, "2026-01-06"),
(104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(105, 4, "West", "Tablet", 1, 25000.0, "2026-01-08"),
(106, 2, "East", "Phone", 1, 15000.0, "2026-01-08"),
(107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
(108, 3, "South", "Mouse", 4, 2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
"qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
north = orders.filter(F.col("region") == "North") # transformation
cols = north.select("order_id", "amount") # transformation
print(type(cols).__name__) # still just a DataFrame
print(cols.count()) # action -> 3
print(cols.take(1)) # action -> list of Row
print(cols.agg(F.sum("amount")).first()[0]) # action -> a Python value
cols.orderBy("order_id").show() # action -> prints
# Output:
# DataFrame
# 3
# [Row(order_id=101, amount=55000.0)]
# 104500.0
# +--------+-------+
# |order_id| amount|
# +--------+-------+
# | 101|55000.0|
# | 103| 1500.0|
# | 107|48000.0|
# +--------+-------+
Key points
- Transformations return a new DataFrame and are lazy
- Actions return values, print or write, and trigger a job
- DataFrames are immutable: transformations never change the original
- Reusing a DataFrame in many actions recomputes it unless cached
- collect() and toPandas() pull everything to the driver; prefer show(), take() or a write