Photon & Disk Cache
Databricks · Architecture, Performance & Cost
Theory
Photon is Databricks' native vectorised query engine, written in C++, that runs Spark SQL and DataFrame operations much faster than the standard JVM engine, especially scans, joins, aggregations and writes on Delta and Parquet. It needs no code changes: SQL warehouses and serverless compute always use it, and on classic clusters you tick Use Photon Acceleration. The disk cache (formerly called the Delta cache) keeps copies of remote Parquet/Delta data files on the local SSDs of worker nodes, so repeated reads of the same data skip cloud storage. It is enabled automatically on node types with local SSDs and stays consistent when files change. SQL warehouses also have a query result cache that returns identical recent queries instantly.
Example: a dashboard query on main.sales.orders takes 9 seconds cold, then about 2 seconds on the next refresh because the files are already cached locally.
Why we use it
Most data work is scanning and joining large tables. Photon makes that compute faster, which often lowers cost because jobs finish sooner, and caching avoids paying the cloud-storage read latency again and again for hot tables.
When to use it
Keep Photon on for SQL, ETL and BI workloads; it helps less for Python UDF-heavy or RDD code, which it cannot accelerate. The disk cache helps repeated reads on the same cluster, such as dashboards and interactive analysis; it does not help one-pass jobs that read data once.
How it works
- Query or job starts
- Is Photon enabled on this compute? Yes: Run supported operators in Photon. No: Use the standard Spark engine.
- Are the needed files in the local disk cache? Yes: Read them from SSD. No: Fetch from cloud storage and cache them.
- Return results (and fill the result cache on SQL warehouses)
- Next run of the same query is faster
Syntax
Classic compute UI: Compute → Create compute → Performance → Use Photon Acceleration
SQL warehouses and serverless: Photon always on
Disk cache: spark.conf.set("spark.databricks.io.cache.enabled", "true") (auto on SSD node types)
Check: query profile shows Photon operators; Spark UI shows Photon in the planExample
# Check / enable disk cache on a classic cluster
spark.conf.get("spark.databricks.io.cache.enabled")
spark.conf.set("spark.databricks.io.cache.enabled", "true")
-- First run reads from cloud storage; next runs read the local SSD cache
SELECT c.city, SUM(o.amount) AS sales
FROM main.sales.orders o
JOIN main.sales.customers c ON o.customer_id = c.customer_id
WHERE o.order_date >= '2026-10-01'
GROUP BY c.city;
-- Prefer built-in functions over Python UDFs so Photon can run them
SELECT upper(name) AS name_uc FROM main.sales.customers;Key points
- Photon needs no code changes; SQL warehouses and serverless always use it
- On classic clusters tick Use Photon Acceleration when creating compute
- Python UDFs and RDD code fall back to the standard engine
- Disk cache stores remote files on worker SSDs for faster repeat reads
- The query result cache on SQL warehouses returns identical recent queries instantly
- Look for Photon operators in the query profile to confirm it was used