All-purpose vs job compute

Databricks · Compute

Theory

Classic compute in Databricks comes in two flavours. All-purpose compute (an interactive cluster) is created from the Compute page and shared by people while they develop in notebooks; it keeps running until someone stops it or it auto-terminates. Job compute is created by the job scheduler when a job run starts and is terminated as soon as the run ends, so each run gets a fresh, isolated cluster.

Job compute is billed at a lower DBU rate than all-purpose compute, which is why production pipelines should not run on interactive clusters. For example, Asha develops orders_etl on her all-purpose cluster during the day; when it is ready, she creates a Lakeflow job that runs the same notebook every night on new job compute (or on serverless compute for jobs).

Why we use it

Interactive work needs a cluster that stays up between commands. Scheduled production work needs isolation, repeatability and the lowest cost. Mixing them causes noisy-neighbour problems and higher bills.

When to use it

Use all-purpose compute for exploration, development and debugging. Use job compute (or serverless jobs) for anything scheduled or triggered. Avoid pointing production jobs at an existing all-purpose cluster except for quick tests.

How it works

  1. Code is written and tested in a notebook
  2. Is someone running it by hand right now? Yes: Use all-purpose compute. No: It is scheduled or triggered.
  3. Create a job in Jobs & Pipelines
  4. Choose job compute or serverless for the task
  5. Run starts: new cluster is created
  6. Run ends: the job cluster is terminated

Syntax

All-purpose: Compute → Create compute
Job compute: Jobs & Pipelines → job → task → Compute → New job cluster
Jobs API: "job_clusters": [{"job_cluster_key": ..., "new_cluster": {...}}]

Example

# Job definition (Jobs API / bundle YAML equivalent) using job compute
job = {
  "name": "orders_nightly",
  "job_clusters": [{
    "job_cluster_key": "etl",
    "new_cluster": {
      "spark_version": "<LTS runtime>",
      "node_type_id": "<worker type>",
      "autoscale": {"min_workers": 1, "max_workers": 4}
    }
  }],
  "tasks": [{
    "task_key": "load_orders",
    "job_cluster_key": "etl",
    "notebook_task": {"notebook_path": "/Workspace/Users/asha@techtown.in/orders_etl"}
  }]
}

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic