Control plane vs compute plane
Databricks · Databricks Fundamentals
Theory
Databricks splits its architecture into two parts. The control plane is run by Databricks: it hosts the web UI, notebooks, job scheduler, Unity Catalog service, REST APIs and cluster manager. The compute plane is where your data is actually processed by Spark and Photon.
There are two kinds of compute plane. With classic compute, the virtual machines run inside your own cloud account and network (your AWS account, Azure subscription or GCP project). With serverless compute (serverless notebooks, jobs, pipelines and serverless SQL warehouses), the machines run in Databricks' cloud account, in the same region, and Databricks manages startup, scaling and patching. In both cases your table data stays in your cloud object storage; for example, a serverless job reads main.sales.orders from your S3 bucket, processes it, and writes back there.
Why we use it
Security teams need to know where data is processed and who manages the machines. The split also explains why serverless starts in seconds (Databricks keeps warm capacity) while classic clusters take minutes (new VMs in your account).
When to use it
Choose serverless for most notebooks, jobs and SQL when it is enabled in your workspace: fast start, no tuning. Choose classic compute when you need custom networking, specific instance types, GPUs, init scripts, R, or features serverless does not support.
How it works
- User clicks Run in a notebook
- Control plane receives the command
- Is the notebook attached to serverless compute? Yes: Databricks-managed VMs in its account run the code. No: Your cluster's VMs in your cloud account run the code.
- Compute reads and writes data in your cloud storage
- Results are sent back to the notebook in the control plane
Syntax
Control plane: UI, APIs, scheduler, Unity Catalog, cluster manager (Databricks account)
Classic compute plane: VMs in your cloud account
Serverless compute plane: VMs in Databricks' account, same regionExample
# The same code runs on either plane; only the compute you attach changes.
# Serverless: pick 'Serverless' in the notebook compute selector.
# Classic: attach to a cluster you created on the Compute page.
df = spark.read.table("main.sales.orders")
print(df.count())
-- In SQL: classic and serverless warehouses run the same SQL
SELECT COUNT(*) FROM main.sales.orders;Key points
- The control plane never needs your raw data to render the UI; processing happens in the compute plane
- Classic compute = VMs in your cloud account; serverless = VMs in Databricks' account
- Serverless compute requires a Unity Catalog-enabled workspace
- Table data lives in your cloud storage for both classic and serverless
- Serverless starts in seconds; classic clusters usually take a few minutes