dbutils & widgets
Databricks · Notebooks & Development
Theory
dbutils is a helper object available in every Databricks notebook. The most used parts are dbutils.fs (list, copy, move and remove files, for example in Unity Catalog volumes under /Volumes/...), dbutils.widgets (input boxes and drop-downs at the top of a notebook), dbutils.secrets (read secrets without printing them), dbutils.notebook (run another notebook with arguments and return a value) and dbutils.jobs.taskValues (pass small values between job tasks).
Widgets turn a notebook into a parameterised report or job task. dbutils.widgets.dropdown("region", "West", ["West", "North", "South"]) draws a drop-down, and dbutils.widgets.get("region") reads the current value as a string. When the notebook runs as a job task, the task's parameters fill the widgets, so the same notebook can process region West today and North tomorrow.
Why we use it
Notebooks often need file operations, parameters and secrets. dbutils gives one consistent API for these, and widgets let jobs pass parameters into notebooks without editing code.
When to use it
Use widgets for notebook parameters (dates, regions, catalog names) and dbutils.fs for file housekeeping. Use dbutils.secrets for passwords and tokens instead of hard-coding them. In SQL you can use named parameter markers like :region instead.
How it works
- Notebook starts
- Create the widget with a default value
- Is the notebook running as a job task with a parameter? Yes: Widget takes the job parameter value. No: Widget keeps the value chosen in the UI.
- dbutils.widgets.get reads the value as a string
- Filter or process data with that value
- Show or write the result
Syntax
dbutils.fs.ls(path) | cp | mv | rm | mkdirs
dbutils.widgets.text(name, default, label) | dropdown(name, default, choices) | get(name) | removeAll()
dbutils.secrets.get(scope, key)
dbutils.notebook.run(path, timeout_seconds, args)
dbutils.help()Example
dbutils.widgets.dropdown("region", "West", ["West", "North", "South"], "Region")
region = dbutils.widgets.get("region")
orders = spark.read.table("main.sales.orders")
display(orders.filter(orders.region == region)
.select("order_id", "region", "amount"))
# Files in a Unity Catalog volume
display(dbutils.fs.ls("/Volumes/main/sales/raw_files/"))
# A secret: the value is redacted if printed
api_key = dbutils.secrets.get(scope="sales", key="crm_api_key")Key points
- dbutils.widgets.get always returns a string; cast numbers and dates yourself
- Job task parameters with the same name fill the widgets automatically
- dbutils.fs works with /Volumes/<catalog>/<schema>/<volume>/ paths
- Secret values are shown as [REDACTED] in notebook output
- dbutils.help() and dbutils.fs.help() list every available method