partitionBy vs bucketBy (bucketing)

Apache Spark · Performance Tuning

Theory

Both organise data when you write it, but in different ways. partitionBy('region') creates one folder per distinct value (region=North/, region=West/), so a query that filters on region reads only the matching folders; this is partition pruning. It suits low-cardinality filter columns such as date; high-cardinality columns create thousands of tiny files.

bucketBy(4, 'customer_id') instead hashes the column into a fixed number of buckets (files) per table, optionally sorted with sortBy, and records the bucket spec in the catalog, so it only works with saveAsTable. When two tables are bucketed the same way on the join key, Spark can join or aggregate them without a shuffle, because rows with the same key are already in matching buckets. For example, joining plain DataFrames on customer_id needs 2 shuffle exchanges, while joining two tables bucketed into 4 buckets on customer_id needs none.

Why we use it

Choosing the right layout at write time makes every later read faster: pruning skips whole folders, and bucketing removes repeated shuffles for tables that are joined on the same key again and again. Interviewers often ask for the difference.

When to use it

Use partitionBy on date-like or low-cardinality filter columns. Use bucketBy for large tables repeatedly joined or aggregated on the same high-cardinality key, with the same bucket count on both sides. On Delta tables, liquid clustering or Z-ordering is the usual alternative to bucketing.

How it works

  1. Choose a layout for a table you are writing
  2. Filters on a low-cardinality column (date)? Yes: partitionBy it. No: No folder partitioning.
  3. Often joined on the same key? Yes: bucketBy(n, key) + saveAsTable. No: No bucketing.
  4. Use the same bucket count and key on both join sides
  5. Later reads prune folders and skip join shuffles

Syntax

df.write.partitionBy("region").parquet(path)
df.write.bucketBy(4, "customer_id").sortBy("customer_id") \
  .mode("overwrite").saveAsTable("orders_b")      # must be saveAsTable
spark.table("orders_b").join(spark.table("customers_b"), "customer_id")
spark.sql.sources.bucketing.enabled = true (default)

Example

import os, tempfile
from pyspark.sql import SparkSession, functions as F

wh = tempfile.mkdtemp()
spark = (SparkSession.builder.master("local[2]")
         .config("spark.ui.enabled", "false")
         .config("spark.sql.warehouse.dir", wh)
         .config("spark.sql.autoBroadcastJoinThreshold", "-1")   # force a sort-merge join
         .getOrCreate())

orders = spark.createDataFrame([
    (101, 1, "North", 55000.0), (102, 2, "West", 30000.0), (103, 1, "North", 1500.0),
    (104, 3, "South", 52000.0), (105, 4, "West", 25000.0), (106, 2, "East", 15000.0),
], "order_id INT, customer_id INT, region STRING, amount DOUBLE")
customers = spark.createDataFrame([(1, "Asha"), (2, "Ravi"), (3, "Meera"), (4, "Karan")],
                                  "customer_id INT, name STRING")

# 1) partitionBy: one FOLDER per value -> partition pruning for filters on region
path = os.path.join(tempfile.mkdtemp(), "orders_by_region")
orders.write.mode("overwrite").partitionBy("region").parquet(path)
print(sorted(d for d in os.listdir(path) if "=" in d))
north = spark.read.parquet(path).filter("region = 'North'")
print("partition filter:", "PartitionFilters: [isnotnull(region" in north._jdf.queryExecution().executedPlan().toString())

# 2) bucketBy: a fixed number of FILES per table, rows placed by hash(customer_id).
#    Needs saveAsTable, because the bucket spec is stored in the catalog.
orders.write.mode("overwrite").bucketBy(4, "customer_id").sortBy("customer_id").saveAsTable("orders_b")
customers.write.mode("overwrite").bucketBy(4, "customer_id").sortBy("customer_id").saveAsTable("customers_b")
print(spark.sql("DESCRIBE EXTENDED orders_b").filter("col_name IN ('Num Buckets', 'Bucket Columns')").collect())

def shuffles(df):
    return df._jdf.queryExecution().executedPlan().toString().count("Exchange hashpartitioning")

# 3) Join on the bucket column: both sides are already hash-partitioned the same way
bucketed = spark.table("orders_b").join(spark.table("customers_b"), "customer_id")
plain = orders.join(customers, "customer_id")
print("shuffles, plain DataFrames:", shuffles(plain))
print("shuffles, bucketed tables :", shuffles(bucketed))
bucketed.groupBy("name").agg(F.sum("amount").alias("sales")).orderBy("name").show()

# Output:
# ['region=East', 'region=North', 'region=South', 'region=West']
# partition filter: True
# [Row(col_name='Num Buckets', data_type='4', comment=''), Row(col_name='Bucket Columns', data_type='[`customer_id`]', comment='')]
# shuffles, plain DataFrames: 2
# shuffles, bucketed tables : 0
# +-----+-------+
# | name|  sales|
# +-----+-------+
# | Asha|56500.0|
# |Karan|25000.0|
# |Meera|52000.0|
# | Ravi|45000.0|
# +-----+-------+

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic