Narrow vs wide transformations (shuffle)
Apache Spark · Spark Architecture
Theory
A narrow transformation computes each output partition from just one input partition, so every task works on its own data with no network traffic: select, filter, withColumn, map and union are narrow. A wide transformation needs rows from many input partitions to build one output partition, so Spark must redistribute the data across the cluster by key. That redistribution is the shuffle: every task writes its rows to local shuffle files grouped by target partition, and tasks in the next stage fetch their pieces over the network. groupBy, join (except broadcast joins), distinct, orderBy and repartition are wide.
For example, filter(amount > 20000) on 100 partitions just runs 100 independent tasks, but groupBy('region').sum('amount') must first send all North rows to the same partition. In explain() output a shuffle appears as an Exchange node, and the number of partitions after a DataFrame shuffle is set by spark.sql.shuffle.partitions (default 200, adjusted at runtime by adaptive query execution).
Why we use it
Shuffles are usually the most expensive part of a Spark job: disk writes, network transfer, serialization and a new stage. Recognising wide transformations lets you reduce them, filter before shuffling, broadcast small tables and spot skew.
When to use it
Filter and select columns before wide operations, combine aggregations into one groupBy, and broadcast small join tables to avoid shuffles. Use repartition only when you really need a different partitioning, and coalesce to reduce partitions without a full shuffle.
How it works
- Transformation in the plan
- Does each output partition need data from only one input partition? Yes: Narrow: pipeline it in the current stage. No: Wide: plan a shuffle (Exchange).
- Map side writes shuffle files grouped by key
- Next stage fetches its partitions over the network
- Continue with the new partitioning
Syntax
Narrow: select, filter, withColumn, map, union, coalesce
Wide: groupBy, join, distinct, orderBy, repartition
df.explain() # look for Exchange = shuffle
spark.conf.set("spark.sql.shuffle.partitions", "64")Example
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.master("local[1]")
.config("spark.ui.enabled", "false")
.getOrCreate())
orders = spark.createDataFrame([
(101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(102, 2, "West", "Phone", 2, 30000.0, "2026-01-06"),
(103, 1, "North", "Mouse", 3, 1500.0, "2026-01-06"),
(104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(105, 4, "West", "Tablet", 1, 25000.0, "2026-01-08"),
(106, 2, "East", "Phone", 1, 15000.0, "2026-01-08"),
(107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
(108, 3, "South", "Mouse", 4, 2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
"qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
narrow = orders.filter(F.col("amount") > 20000).select("region", "amount")
wide = narrow.groupBy("region").agg(F.sum("amount").alias("total"))
print("--- narrow only: no Exchange ---")
narrow.explain()
print("--- with groupBy: Exchange = shuffle ---")
wide.explain()
# Output:
# --- narrow only: no Exchange ---
# == Physical Plan ==
# *(1) Project [region#2, amount#5]
# +- *(1) Filter (isnotnull(amount#5) AND (amount#5 > 20000.0))
# +- *(1) Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#
#
# --- with groupBy: Exchange = shuffle ---
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=false
# +- HashAggregate(keys=[region#2], functions=[sum(amount#5)])
# +- Exchange hashpartitioning(region#2, 200), ENSURE_REQUIREMENTS, [plan_id=31]
# +- HashAggregate(keys=[region#2], functions=[partial_sum(amount#5)])
# +- Project [region#2, amount#5]
# +- Filter (isnotnull(amount#5) AND (amount#5 > 20000.0))
# +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
Key points
- Narrow: one input partition per output partition, no data movement
- Wide: needs data from many partitions, causes a shuffle and a new stage
- In explain(), Exchange marks a shuffle
- Filter early and select only needed columns to shrink shuffles
- coalesce(n) reduces partitions without a full shuffle; repartition(n) always shuffles