partitionBy vs bucketBy (bucketing)
Apache Spark · Performance Tuning
Theory
Both organise data when you write it, but in different ways. partitionBy('region') creates one folder per distinct value (region=North/, region=West/), so a query that filters on region reads only the matching folders; this is partition pruning. It suits low-cardinality filter columns such as date; high-cardinality columns create thousands of tiny files.
bucketBy(4, 'customer_id') instead hashes the column into a fixed number of buckets (files) per table, optionally sorted with sortBy, and records the bucket spec in the catalog, so it only works with saveAsTable. When two tables are bucketed the same way on the join key, Spark can join or aggregate them without a shuffle, because rows with the same key are already in matching buckets. For example, joining plain DataFrames on customer_id needs 2 shuffle exchanges, while joining two tables bucketed into 4 buckets on customer_id needs none.
Why we use it
Choosing the right layout at write time makes every later read faster: pruning skips whole folders, and bucketing removes repeated shuffles for tables that are joined on the same key again and again. Interviewers often ask for the difference.
When to use it
Use partitionBy on date-like or low-cardinality filter columns. Use bucketBy for large tables repeatedly joined or aggregated on the same high-cardinality key, with the same bucket count on both sides. On Delta tables, liquid clustering or Z-ordering is the usual alternative to bucketing.
How it works
- Choose a layout for a table you are writing
- Filters on a low-cardinality column (date)? Yes: partitionBy it. No: No folder partitioning.
- Often joined on the same key? Yes: bucketBy(n, key) + saveAsTable. No: No bucketing.
- Use the same bucket count and key on both join sides
- Later reads prune folders and skip join shuffles
Syntax
df.write.partitionBy("region").parquet(path)
df.write.bucketBy(4, "customer_id").sortBy("customer_id") \
.mode("overwrite").saveAsTable("orders_b") # must be saveAsTable
spark.table("orders_b").join(spark.table("customers_b"), "customer_id")
spark.sql.sources.bucketing.enabled = true (default)Example
import os, tempfile
from pyspark.sql import SparkSession, functions as F
wh = tempfile.mkdtemp()
spark = (SparkSession.builder.master("local[2]")
.config("spark.ui.enabled", "false")
.config("spark.sql.warehouse.dir", wh)
.config("spark.sql.autoBroadcastJoinThreshold", "-1") # force a sort-merge join
.getOrCreate())
orders = spark.createDataFrame([
(101, 1, "North", 55000.0), (102, 2, "West", 30000.0), (103, 1, "North", 1500.0),
(104, 3, "South", 52000.0), (105, 4, "West", 25000.0), (106, 2, "East", 15000.0),
], "order_id INT, customer_id INT, region STRING, amount DOUBLE")
customers = spark.createDataFrame([(1, "Asha"), (2, "Ravi"), (3, "Meera"), (4, "Karan")],
"customer_id INT, name STRING")
# 1) partitionBy: one FOLDER per value -> partition pruning for filters on region
path = os.path.join(tempfile.mkdtemp(), "orders_by_region")
orders.write.mode("overwrite").partitionBy("region").parquet(path)
print(sorted(d for d in os.listdir(path) if "=" in d))
north = spark.read.parquet(path).filter("region = 'North'")
print("partition filter:", "PartitionFilters: [isnotnull(region" in north._jdf.queryExecution().executedPlan().toString())
# 2) bucketBy: a fixed number of FILES per table, rows placed by hash(customer_id).
# Needs saveAsTable, because the bucket spec is stored in the catalog.
orders.write.mode("overwrite").bucketBy(4, "customer_id").sortBy("customer_id").saveAsTable("orders_b")
customers.write.mode("overwrite").bucketBy(4, "customer_id").sortBy("customer_id").saveAsTable("customers_b")
print(spark.sql("DESCRIBE EXTENDED orders_b").filter("col_name IN ('Num Buckets', 'Bucket Columns')").collect())
def shuffles(df):
return df._jdf.queryExecution().executedPlan().toString().count("Exchange hashpartitioning")
# 3) Join on the bucket column: both sides are already hash-partitioned the same way
bucketed = spark.table("orders_b").join(spark.table("customers_b"), "customer_id")
plain = orders.join(customers, "customer_id")
print("shuffles, plain DataFrames:", shuffles(plain))
print("shuffles, bucketed tables :", shuffles(bucketed))
bucketed.groupBy("name").agg(F.sum("amount").alias("sales")).orderBy("name").show()
# Output:
# ['region=East', 'region=North', 'region=South', 'region=West']
# partition filter: True
# [Row(col_name='Num Buckets', data_type='4', comment=''), Row(col_name='Bucket Columns', data_type='[`customer_id`]', comment='')]
# shuffles, plain DataFrames: 2
# shuffles, bucketed tables : 0
# +-----+-------+
# | name| sales|
# +-----+-------+
# | Asha|56500.0|
# |Karan|25000.0|
# |Meera|52000.0|
# | Ravi|45000.0|
# +-----+-------+
Key points
- partitionBy: one folder per value -> partition pruning on filters
- bucketBy: fixed number of hash buckets per table, stored in the catalog
- bucketBy requires saveAsTable; plain .parquet(path) cannot bucket
- Same key and bucket count on both tables = join without a shuffle
- Never partitionBy a high-cardinality column like customer_id
- You can combine both: partitionBy(date) + bucketBy(n, id)