Narrow vs wide transformations (shuffle)

Apache Spark · Spark Architecture

Theory

A narrow transformation computes each output partition from just one input partition, so every task works on its own data with no network traffic: select, filter, withColumn, map and union are narrow. A wide transformation needs rows from many input partitions to build one output partition, so Spark must redistribute the data across the cluster by key. That redistribution is the shuffle: every task writes its rows to local shuffle files grouped by target partition, and tasks in the next stage fetch their pieces over the network. groupBy, join (except broadcast joins), distinct, orderBy and repartition are wide.

For example, filter(amount > 20000) on 100 partitions just runs 100 independent tasks, but groupBy('region').sum('amount') must first send all North rows to the same partition. In explain() output a shuffle appears as an Exchange node, and the number of partitions after a DataFrame shuffle is set by spark.sql.shuffle.partitions (default 200, adjusted at runtime by adaptive query execution).

Why we use it

Shuffles are usually the most expensive part of a Spark job: disk writes, network transfer, serialization and a new stage. Recognising wide transformations lets you reduce them, filter before shuffling, broadcast small tables and spot skew.

When to use it

Filter and select columns before wide operations, combine aggregations into one groupBy, and broadcast small join tables to avoid shuffles. Use repartition only when you really need a different partitioning, and coalesce to reduce partitions without a full shuffle.

How it works

  1. Transformation in the plan
  2. Does each output partition need data from only one input partition? Yes: Narrow: pipeline it in the current stage. No: Wide: plan a shuffle (Exchange).
  3. Map side writes shuffle files grouped by key
  4. Next stage fetches its partitions over the network
  5. Continue with the new partitioning

Syntax

Narrow: select, filter, withColumn, map, union, coalesce
Wide:   groupBy, join, distinct, orderBy, repartition
df.explain()  # look for Exchange = shuffle
spark.conf.set("spark.sql.shuffle.partitions", "64")

Example

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[1]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())

orders = spark.createDataFrame([
    (101, 1, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (102, 2, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (103, 1, "North", "Mouse",  3,  1500.0, "2026-01-06"),
    (104, 3, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (105, 4, "West",  "Tablet", 1, 25000.0, "2026-01-08"),
    (106, 2, "East",  "Phone",  1, 15000.0, "2026-01-08"),
    (107, 5, "North", "Tablet", 2, 48000.0, "2026-01-09"),
    (108, 3, "South", "Mouse",  4,  2000.0, "2026-01-10"),
], "order_id INT, customer_id INT, region STRING, product STRING, "
   "qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

narrow = orders.filter(F.col("amount") > 20000).select("region", "amount")
wide = narrow.groupBy("region").agg(F.sum("amount").alias("total"))

print("--- narrow only: no Exchange ---")
narrow.explain()
print("--- with groupBy: Exchange = shuffle ---")
wide.explain()

# Output:
# --- narrow only: no Exchange ---
# == Physical Plan ==
# *(1) Project [region#2, amount#5]
# +- *(1) Filter (isnotnull(amount#5) AND (amount#5 > 20000.0))
#    +- *(1) Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#
#
# --- with groupBy: Exchange = shuffle ---
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=false
# +- HashAggregate(keys=[region#2], functions=[sum(amount#5)])
#    +- Exchange hashpartitioning(region#2, 200), ENSURE_REQUIREMENTS, [plan_id=31]
#       +- HashAggregate(keys=[region#2], functions=[partial_sum(amount#5)])
#          +- Project [region#2, amount#5]
#             +- Filter (isnotnull(amount#5) AND (amount#5 > 20000.0))
#                +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic