dbutils & widgets

Databricks · Notebooks & Development

Theory

dbutils is a helper object available in every Databricks notebook. The most used parts are dbutils.fs (list, copy, move and remove files, for example in Unity Catalog volumes under /Volumes/...), dbutils.widgets (input boxes and drop-downs at the top of a notebook), dbutils.secrets (read secrets without printing them), dbutils.notebook (run another notebook with arguments and return a value) and dbutils.jobs.taskValues (pass small values between job tasks).

Widgets turn a notebook into a parameterised report or job task. dbutils.widgets.dropdown("region", "West", ["West", "North", "South"]) draws a drop-down, and dbutils.widgets.get("region") reads the current value as a string. When the notebook runs as a job task, the task's parameters fill the widgets, so the same notebook can process region West today and North tomorrow.

Why we use it

Notebooks often need file operations, parameters and secrets. dbutils gives one consistent API for these, and widgets let jobs pass parameters into notebooks without editing code.

When to use it

Use widgets for notebook parameters (dates, regions, catalog names) and dbutils.fs for file housekeeping. Use dbutils.secrets for passwords and tokens instead of hard-coding them. In SQL you can use named parameter markers like :region instead.

How it works

  1. Notebook starts
  2. Create the widget with a default value
  3. Is the notebook running as a job task with a parameter? Yes: Widget takes the job parameter value. No: Widget keeps the value chosen in the UI.
  4. dbutils.widgets.get reads the value as a string
  5. Filter or process data with that value
  6. Show or write the result

Syntax

dbutils.fs.ls(path) | cp | mv | rm | mkdirs
dbutils.widgets.text(name, default, label) | dropdown(name, default, choices) | get(name) | removeAll()
dbutils.secrets.get(scope, key)
dbutils.notebook.run(path, timeout_seconds, args)
dbutils.help()

Example

dbutils.widgets.dropdown("region", "West", ["West", "North", "South"], "Region")
region = dbutils.widgets.get("region")

orders = spark.read.table("main.sales.orders")
display(orders.filter(orders.region == region)
              .select("order_id", "region", "amount"))

# Files in a Unity Catalog volume
display(dbutils.fs.ls("/Volumes/main/sales/raw_files/"))

# A secret: the value is redacted if printed
api_key = dbutils.secrets.get(scope="sales", key="crm_api_key")

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic