Serverless compute

Databricks · Compute

Theory

Serverless compute lets you run notebooks, Lakeflow Jobs and pipelines without creating or tuning a cluster. You pick Serverless in the compute selector, and Databricks starts capacity from a warm pool in its own cloud account, usually within seconds, scales it with the workload and shuts it down when idle. Software comes from a serverless environment version (a fixed Python version plus libraries), and you add your own packages in the notebook's Environment side panel or with %pip.

Serverless uses Spark Connect under the hood, so DataFrame and SQL code works, but some low-level features do not: RDD APIs, R, most Spark configs, global temp views, init scripts and the Spark UI (you use the query profile instead). For example, Ravi attaches his orders notebook to Serverless, adds pandas to the Environment panel, and the job version of it runs on serverless compute for jobs with no cluster JSON at all.

Why we use it

Configuring clusters (instance types, autoscaling, runtime upgrades) takes time and is easy to get wrong. Serverless removes that work, starts fast and only bills while work runs, so small and spiky workloads cost less.

When to use it

Use serverless by default for notebooks, jobs and pipelines when your workspace has it. Use classic compute when you need R, RDDs, GPUs, init scripts, custom Spark configs, specific instance types or private networking that serverless cannot reach.

How it works

  1. Open a notebook or job task
  2. Choose Serverless in the compute selector
  3. Databricks assigns warm capacity in its own account
  4. Environment version and your dependencies are loaded
  5. Does the code use unsupported APIs (RDD, R, custom Spark conf)? Yes: Switch to classic compute. No: Code runs and scales automatically.
  6. Capacity is released when the notebook is idle or the job ends

Syntax

Notebook: compute selector → Serverless
Environment side panel → Base environment / Dependencies → Apply
Job task: Compute → Serverless
%pip install <package>

Example

# Notebook attached to Serverless compute
%pip install openpyxl

from pyspark.sql import functions as F

orders = spark.read.table("main.sales.orders")
daily = (orders.groupBy("order_date")
               .agg(F.sum("amount").alias("revenue")))
daily.write.mode("overwrite").saveAsTable("main.sales.daily_revenue")

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic