Cluster configuration (Runtime, autoscaling, auto-termination, Photon)

Databricks · Compute

Theory

When you create classic compute you choose a Databricks Runtime version: a tested bundle of Apache Spark, Delta Lake, Python, Java/Scala and libraries. LTS (long-term support) versions are best for production, and the ML runtime (the Machine learning checkbox) adds ML libraries. Photon is Databricks' vectorized C++ query engine; the Use Photon Acceleration checkbox makes SQL and DataFrame work on Delta tables much faster without code changes.

Sizing is set with Worker type and Driver type, and either a fixed number of workers or Enable autoscaling with Min and Max workers, so Databricks adds workers when tasks pile up and removes them when idle. Single node runs everything on the driver for small work. Auto-termination stops the cluster after a number of idle minutes. For example, asha-dev-cluster uses an LTS runtime, Photon, autoscaling 1 to 4 workers and terminates after 30 minutes of inactivity.

Why we use it

The wrong settings waste money (idle clusters, too many workers) or make jobs slow (too few workers, old runtime, no Photon). A few settings control most of the cost and speed of classic compute.

When to use it

Tune these when you must use classic compute. For production jobs choose an LTS runtime and autoscaling with sensible limits. Use single node for small data or learning. If you do not need control, serverless compute avoids these choices.

How it works

  1. Compute → Create compute
  2. Choose a policy and Databricks Runtime Version (LTS)
  3. Tick Use Photon Acceleration
  4. Is the data small or just for learning? Yes: Tick Single node. No: Choose Worker type and Min/Max with autoscaling.
  5. Set Terminate after N minutes of inactivity
  6. Click Create compute

Syntax

Compute → Create compute
Policy | Databricks Runtime Version | Machine learning | Use Photon Acceleration
Worker type | Min / Max | Enable autoscaling | Single node
Terminate after N minutes of inactivity | Tags | Advanced → Access mode

Example

# Create an all-purpose cluster with the Databricks SDK for Python
from databricks.sdk import WorkspaceClient
from databricks.sdk.service import compute

w = WorkspaceClient()
lts = w.clusters.select_spark_version(latest=True, long_term_support=True)

w.clusters.create(
    cluster_name="asha-dev-cluster",
    spark_version=lts,
    node_type_id=w.clusters.select_node_type(local_disk=True),
    autoscale=compute.AutoScale(min_workers=1, max_workers=4),
    autotermination_minutes=30,
    runtime_engine=compute.RuntimeEngine.PHOTON,
    custom_tags={"team": "sales"},
)

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic