Control plane vs compute plane

Databricks · Databricks Fundamentals

Theory

Databricks splits its architecture into two parts. The control plane is run by Databricks: it hosts the web UI, notebooks, job scheduler, Unity Catalog service, REST APIs and cluster manager. The compute plane is where your data is actually processed by Spark and Photon.

There are two kinds of compute plane. With classic compute, the virtual machines run inside your own cloud account and network (your AWS account, Azure subscription or GCP project). With serverless compute (serverless notebooks, jobs, pipelines and serverless SQL warehouses), the machines run in Databricks' cloud account, in the same region, and Databricks manages startup, scaling and patching. In both cases your table data stays in your cloud object storage; for example, a serverless job reads main.sales.orders from your S3 bucket, processes it, and writes back there.

Why we use it

Security teams need to know where data is processed and who manages the machines. The split also explains why serverless starts in seconds (Databricks keeps warm capacity) while classic clusters take minutes (new VMs in your account).

When to use it

Choose serverless for most notebooks, jobs and SQL when it is enabled in your workspace: fast start, no tuning. Choose classic compute when you need custom networking, specific instance types, GPUs, init scripts, R, or features serverless does not support.

How it works

  1. User clicks Run in a notebook
  2. Control plane receives the command
  3. Is the notebook attached to serverless compute? Yes: Databricks-managed VMs in its account run the code. No: Your cluster's VMs in your cloud account run the code.
  4. Compute reads and writes data in your cloud storage
  5. Results are sent back to the notebook in the control plane

Syntax

Control plane: UI, APIs, scheduler, Unity Catalog, cluster manager (Databricks account)
Classic compute plane: VMs in your cloud account
Serverless compute plane: VMs in Databricks' account, same region

Example

# The same code runs on either plane; only the compute you attach changes.
# Serverless: pick 'Serverless' in the notebook compute selector.
# Classic: attach to a cluster you created on the Compute page.
df = spark.read.table("main.sales.orders")
print(df.count())

-- In SQL: classic and serverless warehouses run the same SQL
SELECT COUNT(*) FROM main.sales.orders;

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic