File Formats, Partition Pruning & Predicate Pushdown

Apache Spark · Performance Tuning

Theory

How data is stored decides how much Spark must read. Row formats like CSV and JSON must be read and parsed in full, carry no reliable types and cannot skip data. Columnar formats like Parquet and ORC store each column separately with types, compression and min/max statistics per block, so Spark reads only the columns a query selects (column pruning) and skips blocks whose statistics cannot match a filter (predicate pushdown). Delta Lake and Iceberg add transactions and file statistics on top of Parquet.

Partitioning on disk adds another level. df.write.partitionBy("order_date").parquet(path) creates one folder per value, such as order_date=2026-01-05. A filter on that column lets Spark list and read only the matching folders: partition pruning. Check explain() for PartitionFilters and PushedFilters. Choose a low-to-medium cardinality partition column that queries filter on; partitioning by order_id would create millions of tiny files.

Why we use it

Reading less data is the biggest performance win there is. Columnar formats, partition pruning and pushdown can turn a full scan of every file into reading a few columns from a few folders, cutting both run time and cloud cost.

When to use it

Store analytical data as Parquet or a table format on Parquet (Delta, Iceberg), not CSV or JSON. Partition by a column most queries filter on with a moderate number of values, often a date. Keep filters as simple column comparisons so they can be pushed down; wrapping the column in a UDF blocks pushdown.

How it works

  1. Query with a filter on stored data
  2. Does the filter use the partition column? Yes: Partition pruning: read only matching folders. No: List all folders.
  3. Is the format columnar with statistics (Parquet/ORC)? Yes: Read only selected columns; skip blocks via pushed filters. No: Read and parse whole files.
  4. Apply any remaining filters in Spark
  5. Much less data read from storage

Syntax

df.write.mode("overwrite").partitionBy("order_date").parquet(path)
spark.read.parquet(path).filter(F.col("order_date") == "2026-01-07")
.explain()   # look for PartitionFilters and PushedFilters
spark.read.format("csv").option("header", True).schema(ddl).load(p)

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

orders = spark.createDataFrame([
    (1, 101, "North", "Laptop", 1, 55000.0, "2026-01-05"),
    (2, 102, "West",  "Phone",  2, 30000.0, "2026-01-06"),
    (3, 101, "North", "Mouse",  3,  1500.0, "2026-01-07"),
    (4, 103, "South", "Laptop", 1, 52000.0, "2026-01-07"),
    (5, 102, "West",  "Tablet", 1, 25000.0, "2026-01-09"),
    (6, 104, "North", "Phone",  1, 18000.0, "2026-01-10"),
    (7, 103, "South", "Mouse",  2,  1000.0, "2026-01-11"),
    (8, 105, "West",  "Laptop", 1, 60000.0, "2026-01-12"),
], "order_id INT, customer_id INT, region STRING, product STRING, qty INT, amount DOUBLE, order_date STRING"
).withColumn("order_date", F.to_date("order_date"))

import tempfile, os

path = os.path.join(tempfile.mkdtemp(), "orders_parquet")
orders.write.mode("overwrite").partitionBy("region").parquet(path)
print(sorted(d for d in os.listdir(path) if not d.startswith(("_", "."))))   # one folder per region

q = (spark.read.parquet(path)
        .filter((F.col("region") == "West") & (F.col("amount") > 26000))   # partition + data filter
        .select("order_id", "product", "amount"))
q.show()

# PartitionFilters -> only the region=West folder is read
# PushedFilters    -> amount > 26000 is checked using Parquet statistics
# ReadSchema       -> only the selected columns are read
q.explain()

# Output:
# ['region=North', 'region=South', 'region=West']
# +--------+-------+-------+
# |order_id|product| amount|
# +--------+-------+-------+
# |       8| Laptop|60000.0|
# |       2|  Phone|30000.0|
# +--------+-------+-------+
#
# == Physical Plan ==
# *(1) Project [order_id#9, product#11, amount#13]
# +- *(1) Filter (isnotnull(amount#13) AND (amount#13 > 26000.0))
#    +- *(1) ColumnarToRow
#       +- FileScan parquet [order_id#9,product#11,amount#13,region#15] Batched: true, DataFilters: [isnotnull(amount#13), (amount#13 > 26000.0)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/tmpnjho9ccy/orders_parquet], PartitionFilters: [isnotnull(region#15), (region#15 = West)], PushedFilters: [IsNotNull(amount), GreaterThan(amount,26000.0)], ReadSchema: struct<order_id:int,product:string,amount:double>

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic