Partitions, repartition & coalesce
Apache Spark · Performance Tuning
Theory
A DataFrame is split into partitions, and each partition is processed by one task on one core. The number of partitions therefore sets the parallelism: too few and cores sit idle or tasks run out of memory, too many and scheduling overhead and tiny output files dominate. Reads produce partitions based on file sizes (spark.sql.files.maxPartitionBytes, 128 MB by default), and shuffles (groupBy, join, window) produce spark.sql.shuffle.partitions partitions (200 by default), which AQE can then merge.
df.repartition(n) or repartition(n, "col") does a full shuffle to create n evenly sized partitions, optionally hash-partitioned by a column so equal keys land together. df.coalesce(n) only reduces the count by merging existing partitions without a shuffle, so it is cheap but can leave uneven partitions. Use df.rdd.getNumPartitions() to check the count, and F.spark_partition_id() to see where rows live.
Why we use it
Partition count is the main dial for parallelism, memory per task and the number of output files. Getting it right fixes slow stages with idle cores, out-of-memory tasks and the small-files problem when writing tables.
When to use it
Use repartition to increase partitions, rebalance skewed data, or co-locate keys before heavy work or a partitioned write. Use coalesce to reduce partitions cheaply just before writing, without a shuffle. Avoid coalesce(1) on big data: one task then does all the work.
How it works
- Check df.rdd.getNumPartitions()
- Do you need more partitions or even sizes? Yes: repartition(n) or repartition(n, col): full shuffle. No: Go on.
- Do you only need fewer partitions (e.g. before a write)? Yes: coalesce(n): merge without a shuffle. No: Keep the current count.
- Tune spark.sql.shuffle.partitions for shuffles, or rely on AQE coalescing
- Balanced tasks and sensible file counts
Syntax
df.rdd.getNumPartitions()
df.repartition(8) # full shuffle, even sizes
df.repartition(8, "region") # hash by column
df.coalesce(2) # merge, no shuffle
df.repartitionByRange(4, "order_date")
spark.conf.set("spark.sql.shuffle.partitions", "64")Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
orders = spark.createDataFrame([
(1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(2, 102, "West", "Phone", 2, 30000.0, "2026-01-06"),
(3, 101, "North", "Mouse", 3, 1500.0, "2026-01-07"),
(4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(5, 102, "West", "Tablet", 1, 25000.0, "2026-01-09"),
(6, 104, "North", "Phone", 1, 18000.0, "2026-01-10"),
(7, 103, "South", "Mouse", 2, 1000.0, "2026-01-11"),
(8, 105, "West", "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
print("default partitions:", orders.rdd.getNumPartitions())
by_region = orders.repartition(3, "region") # shuffle: equal regions land together
print("after repartition(3, region):", by_region.rdd.getNumPartitions())
(by_region.withColumn("partition", F.spark_partition_id())
.groupBy("partition").agg(F.collect_set("region").alias("regions"), F.count("*").alias("rows"))
.orderBy("partition").show())
fewer = by_region.coalesce(1) # merge without a shuffle
print("after coalesce(1):", fewer.rdd.getNumPartitions())
# The plans show the difference: Exchange (shuffle) vs Coalesce (no shuffle)
by_region.explain()
fewer.explain()
spark.conf.set("spark.sql.shuffle.partitions", "8") # partitions produced by shuffles
print("shuffle partitions:", spark.conf.get("spark.sql.shuffle.partitions"))
# Output:
# default partitions: 2
# after repartition(3, region): 3
# +---------+-------------+----+
# |partition| regions|rows|
# +---------+-------------+----+
# | 0| [South]| 2|
# | 1|[West, North]| 6|
# +---------+-------------+----+
#
# after coalesce(1): 1
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=true
# +- == Final Plan ==
# ResultQueryStage 1
# +- ShuffleQueryStage 0
# +- Exchange hashpartitioning(region#2, 3), REPARTITION_BY_NUM, [plan_id=27]
# +- *(1) Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7]
# +- *(1) Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
# +- == Initial Plan ==
# Exchange hashpartitioning(region#2, 3), REPARTITION_BY_NUM, [plan_id=18]
# +- Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7]
# +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#
#
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=true
# +- == Final Plan ==
# ResultQueryStage 1
# +- Coalesce 1
# +- ShuffleQueryStage 0
# +- Exchange hashpartitioning(region#2, 3), REPARTITION_BY_NUM, [plan_id=166]
# +- *(1) Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7]
# +- *(1) Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
# +- == Initial Plan ==
# Coalesce 1
# +- Exchange hashpartitioning(region#2, 3), REPARTITION_BY_NUM, [plan_id=156]
# +- Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7]
# +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#
#
# shuffle partitions: 8
Key points
- One partition = one task = one core at a time
- repartition always shuffles; coalesce merges without a shuffle and can only reduce
- repartition(n, col) puts equal keys in the same partition
- spark.sql.shuffle.partitions sets the partition count after shuffles; AQE can merge small ones
- coalesce(1) forces all work through a single task: use only for tiny outputs