Cluster configuration (Runtime, autoscaling, auto-termination, Photon)
Databricks · Compute
Theory
When you create classic compute you choose a Databricks Runtime version: a tested bundle of Apache Spark, Delta Lake, Python, Java/Scala and libraries. LTS (long-term support) versions are best for production, and the ML runtime (the Machine learning checkbox) adds ML libraries. Photon is Databricks' vectorized C++ query engine; the Use Photon Acceleration checkbox makes SQL and DataFrame work on Delta tables much faster without code changes.
Sizing is set with Worker type and Driver type, and either a fixed number of workers or Enable autoscaling with Min and Max workers, so Databricks adds workers when tasks pile up and removes them when idle. Single node runs everything on the driver for small work. Auto-termination stops the cluster after a number of idle minutes. For example, asha-dev-cluster uses an LTS runtime, Photon, autoscaling 1 to 4 workers and terminates after 30 minutes of inactivity.
Why we use it
The wrong settings waste money (idle clusters, too many workers) or make jobs slow (too few workers, old runtime, no Photon). A few settings control most of the cost and speed of classic compute.
When to use it
Tune these when you must use classic compute. For production jobs choose an LTS runtime and autoscaling with sensible limits. Use single node for small data or learning. If you do not need control, serverless compute avoids these choices.
How it works
- Compute → Create compute
- Choose a policy and Databricks Runtime Version (LTS)
- Tick Use Photon Acceleration
- Is the data small or just for learning? Yes: Tick Single node. No: Choose Worker type and Min/Max with autoscaling.
- Set Terminate after N minutes of inactivity
- Click Create compute
Syntax
Compute → Create compute
Policy | Databricks Runtime Version | Machine learning | Use Photon Acceleration
Worker type | Min / Max | Enable autoscaling | Single node
Terminate after N minutes of inactivity | Tags | Advanced → Access modeExample
# Create an all-purpose cluster with the Databricks SDK for Python
from databricks.sdk import WorkspaceClient
from databricks.sdk.service import compute
w = WorkspaceClient()
lts = w.clusters.select_spark_version(latest=True, long_term_support=True)
w.clusters.create(
cluster_name="asha-dev-cluster",
spark_version=lts,
node_type_id=w.clusters.select_node_type(local_disk=True),
autoscale=compute.AutoScale(min_workers=1, max_workers=4),
autotermination_minutes=30,
runtime_engine=compute.RuntimeEngine.PHOTON,
custom_tags={"team": "sales"},
)Key points
- Pick an LTS Databricks Runtime for production; tick Machine learning only if you need ML libraries
- Photon speeds up SQL and DataFrame work; UDF-heavy code gains less
- Autoscaling adds workers when tasks queue and removes them when idle
- Always set auto-termination on all-purpose compute
- Tags (e.g. team=sales) flow into billing data for chargeback