Serverless compute
Databricks · Compute
Theory
Serverless compute lets you run notebooks, Lakeflow Jobs and pipelines without creating or tuning a cluster. You pick Serverless in the compute selector, and Databricks starts capacity from a warm pool in its own cloud account, usually within seconds, scales it with the workload and shuts it down when idle. Software comes from a serverless environment version (a fixed Python version plus libraries), and you add your own packages in the notebook's Environment side panel or with %pip.
Serverless uses Spark Connect under the hood, so DataFrame and SQL code works, but some low-level features do not: RDD APIs, R, most Spark configs, global temp views, init scripts and the Spark UI (you use the query profile instead). For example, Ravi attaches his orders notebook to Serverless, adds pandas to the Environment panel, and the job version of it runs on serverless compute for jobs with no cluster JSON at all.
Why we use it
Configuring clusters (instance types, autoscaling, runtime upgrades) takes time and is easy to get wrong. Serverless removes that work, starts fast and only bills while work runs, so small and spiky workloads cost less.
When to use it
Use serverless by default for notebooks, jobs and pipelines when your workspace has it. Use classic compute when you need R, RDDs, GPUs, init scripts, custom Spark configs, specific instance types or private networking that serverless cannot reach.
How it works
- Open a notebook or job task
- Choose Serverless in the compute selector
- Databricks assigns warm capacity in its own account
- Environment version and your dependencies are loaded
- Does the code use unsupported APIs (RDD, R, custom Spark conf)? Yes: Switch to classic compute. No: Code runs and scales automatically.
- Capacity is released when the notebook is idle or the job ends
Syntax
Notebook: compute selector → Serverless
Environment side panel → Base environment / Dependencies → Apply
Job task: Compute → Serverless
%pip install <package>Example
# Notebook attached to Serverless compute
%pip install openpyxl
from pyspark.sql import functions as F
orders = spark.read.table("main.sales.orders")
daily = (orders.groupBy("order_date")
.agg(F.sum("amount").alias("revenue")))
daily.write.mode("overwrite").saveAsTable("main.sales.daily_revenue")Key points
- Serverless needs a Unity Catalog-enabled workspace
- Starts in seconds and scales automatically; no cluster settings to tune
- Libraries come from the environment version plus your Environment panel or %pip
- No RDD APIs, R, init scripts, global temp views or Spark UI on serverless
- Use the query profile instead of the Spark UI to see performance