Partitions, repartition & coalesce

Apache Spark · Performance Tuning

Theory

A DataFrame is split into partitions, and each partition is processed by one task on one core. The number of partitions therefore sets the parallelism: too few and cores sit idle or tasks run out of memory, too many and scheduling overhead and tiny output files dominate. Reads produce partitions based on file sizes (spark.sql.files.maxPartitionBytes, 128 MB by default), and shuffles (groupBy, join, window) produce spark.sql.shuffle.partitions partitions (200 by default), which AQE can then merge.

df.repartition(n) or repartition(n, "col") does a full shuffle to create n evenly sized partitions, optionally hash-partitioned by a column so equal keys land together. df.coalesce(n) only reduces the count by merging existing partitions without a shuffle, so it is cheap but can leave uneven partitions. Use df.rdd.getNumPartitions() to check the count, and F.spark_partition_id() to see where rows live.

Why we use it

Partition count is the main dial for parallelism, memory per task and the number of output files. Getting it right fixes slow stages with idle cores, out-of-memory tasks and the small-files problem when writing tables.

When to use it

Use repartition to increase partitions, rebalance skewed data, or co-locate keys before heavy work or a partitioned write. Use coalesce to reduce partitions cheaply just before writing, without a shuffle. Avoid coalesce(1) on big data: one task then does all the work.

How it works

  1. Check df.rdd.getNumPartitions()
  2. Do you need more partitions or even sizes? Yes: repartition(n) or repartition(n, col): full shuffle. No: Go on.
  3. Do you only need fewer partitions (e.g. before a write)? Yes: coalesce(n): merge without a shuffle. No: Keep the current count.
  4. Tune spark.sql.shuffle.partitions for shuffles, or rely on AQE coalescing
  5. Balanced tasks and sensible file counts

Syntax

df.rdd.getNumPartitions()
df.repartition(8)                 # full shuffle, even sizes
df.repartition(8, "region")       # hash by column
df.coalesce(2)                    # merge, no shuffle
df.repartitionByRange(4, "order_date")
spark.conf.set("spark.sql.shuffle.partitions", "64")

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

orders = spark.createDataFrame([
    (1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (2, 102, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (3, 101, "North", "Mouse",  3,  1500.0, "2026-01-07"),
    (4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (5, 102, "West",  "Tablet", 1, 25000.0, "2026-01-09"),
    (6, 104, "North", "Phone",  1, 18000.0, "2026-01-10"),
    (7, 103, "South", "Mouse",  2,  1000.0, "2026-01-11"),
    (8, 105, "West",  "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

print("default partitions:", orders.rdd.getNumPartitions())

by_region = orders.repartition(3, "region")            # shuffle: equal regions land together
print("after repartition(3, region):", by_region.rdd.getNumPartitions())
(by_region.withColumn("partition", F.spark_partition_id())
          .groupBy("partition").agg(F.collect_set("region").alias("regions"), F.count("*").alias("rows"))
          .orderBy("partition").show())

fewer = by_region.coalesce(1)                           # merge without a shuffle
print("after coalesce(1):", fewer.rdd.getNumPartitions())

# The plans show the difference: Exchange (shuffle) vs Coalesce (no shuffle)
by_region.explain()
fewer.explain()

spark.conf.set("spark.sql.shuffle.partitions", "8")    # partitions produced by shuffles
print("shuffle partitions:", spark.conf.get("spark.sql.shuffle.partitions"))

# Output:
# default partitions: 2
# after repartition(3, region): 3
# +---------+-------------+----+
# |partition|      regions|rows|
# +---------+-------------+----+
# |        0|      [South]|   2|
# |        1|[West, North]|   6|
# +---------+-------------+----+
#
# after coalesce(1): 1
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=true
# +- == Final Plan ==
#    ResultQueryStage 1
#    +- ShuffleQueryStage 0
#       +- Exchange hashpartitioning(region#2, 3), REPARTITION_BY_NUM, [plan_id=27]
#          +- *(1) Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7]
#             +- *(1) Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
# +- == Initial Plan ==
#    Exchange hashpartitioning(region#2, 3), REPARTITION_BY_NUM, [plan_id=18]
#    +- Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7]
#       +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#
#
# == Physical Plan ==
# AdaptiveSparkPlan isFinalPlan=true
# +- == Final Plan ==
#    ResultQueryStage 1
#    +- Coalesce 1
#       +- ShuffleQueryStage 0
#          +- Exchange hashpartitioning(region#2, 3), REPARTITION_BY_NUM, [plan_id=166]
#             +- *(1) Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7]
#                +- *(1) Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
# +- == Initial Plan ==
#    Coalesce 1
#    +- Exchange hashpartitioning(region#2, 3), REPARTITION_BY_NUM, [plan_id=156]
#       +- Project [order_id#0, customer_id#1, region#2, product#3, qty#4, amount#5, cast(order_date#6 as date) AS order_date#7]
#          +- Scan ExistingRDD[order_id#0,customer_id#1,region#2,product#3,qty#4,amount#5,order_date#6]
#
#
# shuffle partitions: 8

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic