Job Parameters & Task Values
Databricks · Lakeflow Jobs
Theory
Jobs pass information in two ways. Job parameters are key/value pairs defined once on the job (for example run_date and region) and pushed to every task; you can override them for a single run with Run now with different parameters. Notebooks read them as widgets with dbutils.widgets.get, and SQL tasks use :run_date style parameters. Dynamic value references inside double curly braces fill values at run time, such as {{job.start_time.iso_date}}, {{job.run_id}} or {{job.parameters.region}}. Task values let one task send a small result to later tasks: an upstream notebook calls dbutils.jobs.taskValues.set, and a downstream task reads it with taskValues.get or the reference {{tasks.ingest_orders.values.row_count}}.
Example: ingest_orders sets row_count = 840; an If/else condition task checks {{tasks.ingest_orders.values.row_count}} > 0 before running build_gold.
Why we use it
Hard-coded dates and paths make jobs impossible to rerun or reuse. Parameters let one job handle any date or region, and task values let tasks make decisions based on what earlier tasks found, without writing temp tables.
When to use it
Use job parameters for inputs that change per run (dates, regions, environment names). Use task values for small outputs like counts, flags or file names. Do not pass large data through task values; write a table and pass its name instead.
How it works
- Define job parameters run_date and region
- Run starts; references like {{job.start_time.iso_date}} are resolved
- ingest_orders reads widgets and sets task value row_count
- Is row_count > 0 (If/else condition)? Yes: Run build_gold with the value. No: Skip to a notify task.
- Downstream tasks finish using the passed values
Syntax
Job parameters: Job details → Parameters → Edit parameters (key, default value)
Notebook: dbutils.widgets.get("run_date")
SQL task: WHERE order_date = :run_date
Task values: dbutils.jobs.taskValues.set(key="k", value=v)
dbutils.jobs.taskValues.get(taskKey="task", key="k", default=0)
References: {{job.parameters.<name>}} {{tasks.<task>.values.<key>}} {{job.start_time.iso_date}} {{job.run_id}}Example
# Task ingest_orders (notebook)
run_date = dbutils.widgets.get("run_date") # job parameter
region = dbutils.widgets.get("region")
df = (spark.table("main.sales.orders")
.where(f"order_date = '{run_date}' AND region = '{region}'"))
dbutils.jobs.taskValues.set(key="row_count", value=df.count())
# Task build_gold (notebook) reads the upstream value
rows = dbutils.jobs.taskValues.get(taskKey="ingest_orders", key="row_count", default=0)
print(f"Building gold for {rows} orders")
# Job parameters (Job details → Parameters)
# run_date = {{job.start_time.iso_date}}
# region = North
# If/else condition task: {{tasks.ingest_orders.values.row_count}} > 0Key points
- Job parameters are pushed to every task in the job
- Run now with different parameters overrides values for one run only
- {{job.start_time.iso_date}} gives the run's start date, handy for daily loads
- Task values are for small JSON-serialisable values, not datasets
- If/else condition tasks compare references to branch the DAG
- Avoid string-building SQL from parameters in production; use SQL parameters