Photon & Disk Cache

Databricks · Architecture, Performance & Cost

Theory

Photon is Databricks' native vectorised query engine, written in C++, that runs Spark SQL and DataFrame operations much faster than the standard JVM engine, especially scans, joins, aggregations and writes on Delta and Parquet. It needs no code changes: SQL warehouses and serverless compute always use it, and on classic clusters you tick Use Photon Acceleration. The disk cache (formerly called the Delta cache) keeps copies of remote Parquet/Delta data files on the local SSDs of worker nodes, so repeated reads of the same data skip cloud storage. It is enabled automatically on node types with local SSDs and stays consistent when files change. SQL warehouses also have a query result cache that returns identical recent queries instantly.

Example: a dashboard query on main.sales.orders takes 9 seconds cold, then about 2 seconds on the next refresh because the files are already cached locally.

Why we use it

Most data work is scanning and joining large tables. Photon makes that compute faster, which often lowers cost because jobs finish sooner, and caching avoids paying the cloud-storage read latency again and again for hot tables.

When to use it

Keep Photon on for SQL, ETL and BI workloads; it helps less for Python UDF-heavy or RDD code, which it cannot accelerate. The disk cache helps repeated reads on the same cluster, such as dashboards and interactive analysis; it does not help one-pass jobs that read data once.

How it works

  1. Query or job starts
  2. Is Photon enabled on this compute? Yes: Run supported operators in Photon. No: Use the standard Spark engine.
  3. Are the needed files in the local disk cache? Yes: Read them from SSD. No: Fetch from cloud storage and cache them.
  4. Return results (and fill the result cache on SQL warehouses)
  5. Next run of the same query is faster

Syntax

Classic compute UI: Compute → Create compute → Performance → Use Photon Acceleration
SQL warehouses and serverless: Photon always on
Disk cache: spark.conf.set("spark.databricks.io.cache.enabled", "true")  (auto on SSD node types)
Check: query profile shows Photon operators; Spark UI shows Photon in the plan

Example

# Check / enable disk cache on a classic cluster
spark.conf.get("spark.databricks.io.cache.enabled")
spark.conf.set("spark.databricks.io.cache.enabled", "true")

-- First run reads from cloud storage; next runs read the local SSD cache
SELECT c.city, SUM(o.amount) AS sales
FROM main.sales.orders o
JOIN main.sales.customers c ON o.customer_id = c.customer_id
WHERE o.order_date >= '2026-10-01'
GROUP BY c.city;

-- Prefer built-in functions over Python UDFs so Photon can run them
SELECT upper(name) AS name_uc FROM main.sales.customers;

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic