All-purpose vs job compute
Databricks · Compute
Theory
Classic compute in Databricks comes in two flavours. All-purpose compute (an interactive cluster) is created from the Compute page and shared by people while they develop in notebooks; it keeps running until someone stops it or it auto-terminates. Job compute is created by the job scheduler when a job run starts and is terminated as soon as the run ends, so each run gets a fresh, isolated cluster.
Job compute is billed at a lower DBU rate than all-purpose compute, which is why production pipelines should not run on interactive clusters. For example, Asha develops orders_etl on her all-purpose cluster during the day; when it is ready, she creates a Lakeflow job that runs the same notebook every night on new job compute (or on serverless compute for jobs).
Why we use it
Interactive work needs a cluster that stays up between commands. Scheduled production work needs isolation, repeatability and the lowest cost. Mixing them causes noisy-neighbour problems and higher bills.
When to use it
Use all-purpose compute for exploration, development and debugging. Use job compute (or serverless jobs) for anything scheduled or triggered. Avoid pointing production jobs at an existing all-purpose cluster except for quick tests.
How it works
- Code is written and tested in a notebook
- Is someone running it by hand right now? Yes: Use all-purpose compute. No: It is scheduled or triggered.
- Create a job in Jobs & Pipelines
- Choose job compute or serverless for the task
- Run starts: new cluster is created
- Run ends: the job cluster is terminated
Syntax
All-purpose: Compute → Create compute
Job compute: Jobs & Pipelines → job → task → Compute → New job cluster
Jobs API: "job_clusters": [{"job_cluster_key": ..., "new_cluster": {...}}]Example
# Job definition (Jobs API / bundle YAML equivalent) using job compute
job = {
"name": "orders_nightly",
"job_clusters": [{
"job_cluster_key": "etl",
"new_cluster": {
"spark_version": "<LTS runtime>",
"node_type_id": "<worker type>",
"autoscale": {"min_workers": 1, "max_workers": 4}
}
}],
"tasks": [{
"task_key": "load_orders",
"job_cluster_key": "etl",
"notebook_task": {"notebook_path": "/Workspace/Users/asha@techtown.in/orders_etl"}
}]
}Key points
- All-purpose compute is interactive and shared; job compute lives only for one job run
- Job compute uses a lower DBU rate than all-purpose compute
- Job compute is created and terminated by the job; you cannot restart it yourself
- Several tasks in one job can share one job cluster through job_cluster_key
- Serverless compute for jobs is often the simplest choice when available