File Formats, Partition Pruning & Predicate Pushdown
Apache Spark · Performance Tuning
Theory
How data is stored decides how much Spark must read. Row formats like CSV and JSON must be read and parsed in full, carry no reliable types and cannot skip data. Columnar formats like Parquet and ORC store each column separately with types, compression and min/max statistics per block, so Spark reads only the columns a query selects (column pruning) and skips blocks whose statistics cannot match a filter (predicate pushdown). Delta Lake and Iceberg add transactions and file statistics on top of Parquet.
Partitioning on disk adds another level. df.write.partitionBy("order_date").parquet(path) creates one folder per value, such as order_date=2026-01-05. A filter on that column lets Spark list and read only the matching folders: partition pruning. Check explain() for PartitionFilters and PushedFilters. Choose a low-to-medium cardinality partition column that queries filter on; partitioning by order_id would create millions of tiny files.
Why we use it
Reading less data is the biggest performance win there is. Columnar formats, partition pruning and pushdown can turn a full scan of every file into reading a few columns from a few folders, cutting both run time and cloud cost.
When to use it
Store analytical data as Parquet or a table format on Parquet (Delta, Iceberg), not CSV or JSON. Partition by a column most queries filter on with a moderate number of values, often a date. Keep filters as simple column comparisons so they can be pushed down; wrapping the column in a UDF blocks pushdown.
How it works
- Query with a filter on stored data
- Does the filter use the partition column? Yes: Partition pruning: read only matching folders. No: List all folders.
- Is the format columnar with statistics (Parquet/ORC)? Yes: Read only selected columns; skip blocks via pushed filters. No: Read and parse whole files.
- Apply any remaining filters in Spark
- Much less data read from storage
Syntax
df.write.mode("overwrite").partitionBy("order_date").parquet(path)
spark.read.parquet(path).filter(F.col("order_date") == "2026-01-07")
.explain() # look for PartitionFilters and PushedFilters
spark.read.format("csv").option("header", True).schema(ddl).load(p)Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
orders = spark.createDataFrame([
(1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
(2, 102, "West", "Phone", 2, 30000.0, "2026-01-06"),
(3, 101, "North", "Mouse", 3, 1500.0, "2026-01-07"),
(4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
(5, 102, "West", "Tablet", 1, 25000.0, "2026-01-09"),
(6, 104, "North", "Phone", 1, 18000.0, "2026-01-10"),
(7, 103, "South", "Mouse", 2, 1000.0, "2026-01-11"),
(8, 105, "West", "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))
import tempfile, os
path = os.path.join(tempfile.mkdtemp(), "orders_parquet")
orders.write.mode("overwrite").partitionBy("region").parquet(path)
print(sorted(d for d in os.listdir(path) if not d.startswith(("_", ".")))) # one folder per region
q = (spark.read.parquet(path)
.filter((F.col("region") == "West") & (F.col("amount") > 26000)) # partition + data filter
.select("order_id", "product", "amount"))
q.show()
# PartitionFilters -> only the region=West folder is read
# PushedFilters -> amount > 26000 is checked using Parquet statistics
# ReadSchema -> only the selected columns are read
q.explain()
# Output:
# ['region=North', 'region=South', 'region=West']
# +--------+-------+-------+
# |order_id|product| amount|
# +--------+-------+-------+
# | 8| Laptop|60000.0|
# | 2| Phone|30000.0|
# +--------+-------+-------+
#
# == Physical Plan ==
# *(1) Project [order_id#9, product#11, amount#13]
# +- *(1) Filter (isnotnull(amount#13) AND (amount#13 > 26000.0))
# +- *(1) ColumnarToRow
# +- FileScan parquet [order_id#9,product#11,amount#13,region#15] Batched: true, DataFilters: [isnotnull(amount#13), (amount#13 > 26000.0)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/tmpnjho9ccy/orders_parquet], PartitionFilters: [isnotnull(region#15), (region#15 = West)], PushedFilters: [IsNotNull(amount), GreaterThan(amount,26000.0)], ReadSchema: struct<order_id:int,product:string,amount:double>
Key points
- Columnar formats (Parquet, ORC) read only needed columns and skip blocks using statistics
- Partition pruning reads only folders matching a filter on the partition column
- Look for PartitionFilters and PushedFilters in explain() to confirm
- Partition by a moderate-cardinality column queries filter on, often a date
- Over-partitioning creates many small files that slow every read
- Functions or UDFs wrapped around a column can stop filters being pushed down