Hash vs range partitioning

Apache Spark · Performance Tuning

Theory

When Spark shuffles data it must decide which partition each row goes to. Hash partitioning computes hash(key) modulo the number of partitions, so all rows with the same key land in the same partition. It is what groupBy, joins, reduceByKey, df.repartition(n, 'col') and rdd.partitionBy(n) use. It is fast, but each partition holds an unrelated mix of keys, and with few distinct keys some partitions can be empty or overloaded.

Range partitioning first samples the keys, picks boundary values, and sends each contiguous range of keys to one partition, so partition 0 holds the smallest values and the last partition the largest. orderBy, sortByKey and df.repartitionByRange(n, 'col') use it. It is ideal for sorted output and range-based files, but boundaries come from a sample, so sizes are approximate and skewed values still create big partitions. For example, repartitionByRange(3, 'amount') on six orders puts 1500-15000, 25000-30000 and 52000-55000 into three partitions.

Why we use it

Partitioning decides how evenly work is spread and whether later steps can skip a shuffle. Knowing which operations hash and which range-partition helps you explain skew, empty partitions and why sorted writes produce nicely ordered files.

When to use it

Use hash partitioning (repartition by key) before repeated joins or aggregations on that key. Use range partitioning when you need globally sorted output or files organised by ranges of a column such as a date or id. Avoid partitioning on low-cardinality columns with many partitions.

How it works

  1. A shuffle needs a target partition for each row
  2. Sort or range layout needed? Yes: Range: sample keys, pick bounds. No: Hash: hash(key) % n.
  3. Send each row to its target partition
  4. Range: each partition holds one sorted key range
  5. Hash: all rows of a key are together, keys mixed
  6. Next stage reads its partition

Syntax

df.repartition(8, "region")              # hash
df.repartitionByRange(8, "order_date")    # range (sampled boundaries)
df.orderBy("amount")                      # range partitioning + sort
pairs.partitionBy(8)                      # RDD hash partitioner
pairs.sortByKey(numPartitions=8)          # RDD range partitioner
F.spark_partition_id()                    # see where rows went

Example

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[2]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())

orders = spark.createDataFrame([
    (101, "North", 55000.0), (102, "West", 30000.0), (103, "North", 1500.0),
    (104, "South", 52000.0), (105, "West", 25000.0), (106, "East", 15000.0),
], "order_id INT, region STRING, amount DOUBLE")

def show_parts(df, col):
    (df.groupBy(F.spark_partition_id().alias("partition"))
       .agg(F.count("*").alias("rows"), F.min(col).alias("min"), F.max(col).alias("max"))
       .orderBy("partition").show())

# 1) Hash partitioning: partition = hash(key) % n. Same key -> same partition,
#    but each partition holds a mix of unrelated keys.
show_parts(orders.repartition(3, "region"), "region")

# 2) Range partitioning: Spark samples the keys, picks boundaries, and each
#    partition holds one contiguous, sorted range of values.
show_parts(orders.repartitionByRange(3, "amount"), "amount")

# 3) RDD API: partitionBy(n) uses a hash partitioner on the key of a pair RDD
pairs = orders.rdd.map(lambda r: (r.region, r.amount)).partitionBy(2)
print([sorted({k for k, _ in part}) for part in pairs.glom().collect()])

# 4) sortByKey uses a range partitioner; it keeps the key order across partitions
print([[k for k, _ in part] for part in
       orders.rdd.map(lambda r: (r.amount, r.order_id)).sortByKey(numPartitions=2).glom().collect()])

# Output:
# +---------+----+-----+-----+
# |partition|rows|  min|  max|
# +---------+----+-----+-----+
# |        0|   1|South|South|
# |        1|   5| East| West|
# +---------+----+-----+-----+
#
# +---------+----+-------+-------+
# |partition|rows|    min|    max|
# +---------+----+-------+-------+
# |        0|   2| 1500.0|15000.0|
# |        1|   2|25000.0|30000.0|
# |        2|   2|52000.0|55000.0|
# +---------+----+-------+-------+
#
# [['North', 'West'], ['East', 'South']]
# [[1500.0, 15000.0, 25000.0, 30000.0], [52000.0, 55000.0]]

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic