Hash vs range partitioning
Apache Spark · Performance Tuning
Theory
When Spark shuffles data it must decide which partition each row goes to. Hash partitioning computes hash(key) modulo the number of partitions, so all rows with the same key land in the same partition. It is what groupBy, joins, reduceByKey, df.repartition(n, 'col') and rdd.partitionBy(n) use. It is fast, but each partition holds an unrelated mix of keys, and with few distinct keys some partitions can be empty or overloaded.
Range partitioning first samples the keys, picks boundary values, and sends each contiguous range of keys to one partition, so partition 0 holds the smallest values and the last partition the largest. orderBy, sortByKey and df.repartitionByRange(n, 'col') use it. It is ideal for sorted output and range-based files, but boundaries come from a sample, so sizes are approximate and skewed values still create big partitions. For example, repartitionByRange(3, 'amount') on six orders puts 1500-15000, 25000-30000 and 52000-55000 into three partitions.
Why we use it
Partitioning decides how evenly work is spread and whether later steps can skip a shuffle. Knowing which operations hash and which range-partition helps you explain skew, empty partitions and why sorted writes produce nicely ordered files.
When to use it
Use hash partitioning (repartition by key) before repeated joins or aggregations on that key. Use range partitioning when you need globally sorted output or files organised by ranges of a column such as a date or id. Avoid partitioning on low-cardinality columns with many partitions.
How it works
- A shuffle needs a target partition for each row
- Sort or range layout needed? Yes: Range: sample keys, pick bounds. No: Hash: hash(key) % n.
- Send each row to its target partition
- Range: each partition holds one sorted key range
- Hash: all rows of a key are together, keys mixed
- Next stage reads its partition
Syntax
df.repartition(8, "region") # hash
df.repartitionByRange(8, "order_date") # range (sampled boundaries)
df.orderBy("amount") # range partitioning + sort
pairs.partitionBy(8) # RDD hash partitioner
pairs.sortByKey(numPartitions=8) # RDD range partitioner
F.spark_partition_id() # see where rows wentExample
from pyspark.sql import SparkSession, functions as F
spark = (SparkSession.builder.master("local[2]")
.config("spark.ui.enabled", "false")
.getOrCreate())
orders = spark.createDataFrame([
(101, "North", 55000.0), (102, "West", 30000.0), (103, "North", 1500.0),
(104, "South", 52000.0), (105, "West", 25000.0), (106, "East", 15000.0),
], "order_id INT, region STRING, amount DOUBLE")
def show_parts(df, col):
(df.groupBy(F.spark_partition_id().alias("partition"))
.agg(F.count("*").alias("rows"), F.min(col).alias("min"), F.max(col).alias("max"))
.orderBy("partition").show())
# 1) Hash partitioning: partition = hash(key) % n. Same key -> same partition,
# but each partition holds a mix of unrelated keys.
show_parts(orders.repartition(3, "region"), "region")
# 2) Range partitioning: Spark samples the keys, picks boundaries, and each
# partition holds one contiguous, sorted range of values.
show_parts(orders.repartitionByRange(3, "amount"), "amount")
# 3) RDD API: partitionBy(n) uses a hash partitioner on the key of a pair RDD
pairs = orders.rdd.map(lambda r: (r.region, r.amount)).partitionBy(2)
print([sorted({k for k, _ in part}) for part in pairs.glom().collect()])
# 4) sortByKey uses a range partitioner; it keeps the key order across partitions
print([[k for k, _ in part] for part in
orders.rdd.map(lambda r: (r.amount, r.order_id)).sortByKey(numPartitions=2).glom().collect()])
# Output:
# +---------+----+-----+-----+
# |partition|rows| min| max|
# +---------+----+-----+-----+
# | 0| 1|South|South|
# | 1| 5| East| West|
# +---------+----+-----+-----+
#
# +---------+----+-------+-------+
# |partition|rows| min| max|
# +---------+----+-------+-------+
# | 0| 2| 1500.0|15000.0|
# | 1| 2|25000.0|30000.0|
# | 2| 2|52000.0|55000.0|
# +---------+----+-------+-------+
#
# [['North', 'West'], ['East', 'South']]
# [[1500.0, 15000.0, 25000.0, 30000.0], [52000.0, 55000.0]]
Key points
- Hash: partition = hash(key) % n; same key -> same partition
- Range: sampled boundaries; each partition holds a contiguous key range
- groupBy, joins, repartition(col) hash; orderBy, repartitionByRange range
- Few distinct keys + hash = empty or overloaded partitions
- Range boundaries come from a sample, so sizes are approximate
- Neither fixes a single hot key: use salting or AQE skew joins