Job Parameters & Task Values

Databricks · Lakeflow Jobs

Theory

Jobs pass information in two ways. Job parameters are key/value pairs defined once on the job (for example run_date and region) and pushed to every task; you can override them for a single run with Run now with different parameters. Notebooks read them as widgets with dbutils.widgets.get, and SQL tasks use :run_date style parameters. Dynamic value references inside double curly braces fill values at run time, such as {{job.start_time.iso_date}}, {{job.run_id}} or {{job.parameters.region}}. Task values let one task send a small result to later tasks: an upstream notebook calls dbutils.jobs.taskValues.set, and a downstream task reads it with taskValues.get or the reference {{tasks.ingest_orders.values.row_count}}.

Example: ingest_orders sets row_count = 840; an If/else condition task checks {{tasks.ingest_orders.values.row_count}} > 0 before running build_gold.

Why we use it

Hard-coded dates and paths make jobs impossible to rerun or reuse. Parameters let one job handle any date or region, and task values let tasks make decisions based on what earlier tasks found, without writing temp tables.

When to use it

Use job parameters for inputs that change per run (dates, regions, environment names). Use task values for small outputs like counts, flags or file names. Do not pass large data through task values; write a table and pass its name instead.

How it works

  1. Define job parameters run_date and region
  2. Run starts; references like {{job.start_time.iso_date}} are resolved
  3. ingest_orders reads widgets and sets task value row_count
  4. Is row_count > 0 (If/else condition)? Yes: Run build_gold with the value. No: Skip to a notify task.
  5. Downstream tasks finish using the passed values

Syntax

Job parameters: Job details → Parameters → Edit parameters (key, default value)
Notebook: dbutils.widgets.get("run_date")
SQL task: WHERE order_date = :run_date
Task values: dbutils.jobs.taskValues.set(key="k", value=v)
             dbutils.jobs.taskValues.get(taskKey="task", key="k", default=0)
References: {{job.parameters.<name>}}  {{tasks.<task>.values.<key>}}  {{job.start_time.iso_date}}  {{job.run_id}}

Example

# Task ingest_orders (notebook)
run_date = dbutils.widgets.get("run_date")          # job parameter
region   = dbutils.widgets.get("region")

df = (spark.table("main.sales.orders")
        .where(f"order_date = '{run_date}' AND region = '{region}'"))
dbutils.jobs.taskValues.set(key="row_count", value=df.count())

# Task build_gold (notebook) reads the upstream value
rows = dbutils.jobs.taskValues.get(taskKey="ingest_orders", key="row_count", default=0)
print(f"Building gold for {rows} orders")

# Job parameters (Job details → Parameters)
#   run_date = {{job.start_time.iso_date}}
#   region   = North
# If/else condition task:  {{tasks.ingest_orders.values.row_count}}  >  0

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic