Jobs & Task DAGs
Databricks · Lakeflow Jobs
Theory
Lakeflow Jobs (formerly Databricks Workflows, and before that just Jobs) is the built-in orchestrator. A job is a set of tasks plus settings for compute, triggers, parameters and notifications. Each task runs one thing: a notebook, a Lakeflow pipeline, a SQL query or file, a Python script or wheel, a dbt project, a dashboard refresh, another job, or control-flow tasks such as If/else condition and For each. Tasks are linked with Depends on, forming a DAG (directed acyclic graph), so independent tasks run in parallel and a task starts only when its upstream tasks finish. Run if lets a task run on other outcomes, for example only when something failed.
Example: job orders_daily runs ingest_orders (pipeline), then clean_orders and load_customers in parallel, then build_gold (SQL), then refresh_dashboard.
Why we use it
Real data flows have many steps that must run in order, retry on failure and alert someone when they break. Lakeflow Jobs does this inside Databricks with no external scheduler, using serverless or job compute and Unity Catalog permissions.
When to use it
Use Lakeflow Jobs to schedule and chain anything that runs on Databricks. Inside one pipeline, let the pipeline handle table order; use a job to combine pipelines with notebooks, SQL, dashboards and alerts. External orchestrators like Airflow can still trigger jobs if your company already uses them.
How it works
- Jobs & Pipelines → Create → Job
- Add the first task: pick type, source and compute
- + Add task for each next step and set Depends on
- Should a step run only on failure or always? Yes: Set Run if dependencies (e.g. At least one failed). No: Keep All succeeded.
- Add triggers, parameters and notifications in Job details
- Click Run now and watch the task graph
Syntax
UI: Jobs & Pipelines → Create → Job → choose a task type tile → + Add task
Task fields: Task name, Type, Source/Path, Compute, Depends on, Run if dependencies, Parameters
As code: Declarative Automation Bundles (formerly Databricks Asset Bundles) YAML, or the Jobs APIExample
# resources/orders_daily.job.yml (Declarative Automation Bundle)
resources:
jobs:
orders_daily:
name: orders_daily
tasks:
- task_key: ingest_orders
pipeline_task:
pipeline_id: ${resources.pipelines.orders_pipeline.id}
- task_key: clean_orders
depends_on: [{ task_key: ingest_orders }]
notebook_task:
notebook_path: ../src/clean_orders.ipynb
- task_key: build_gold
depends_on: [{ task_key: clean_orders }]
sql_task:
file: { path: ../src/build_gold.sql }
warehouse_id: ${var.warehouse_id}Key points
- Former names: Databricks Workflows / Jobs; the sidebar entry is Jobs & Pipelines
- Depends on builds the DAG; tasks without dependencies between them run in parallel
- Run if supports All succeeded (default), All done, At least one failed and more
- Serverless compute for jobs is the simplest choice when available
- If/else condition and For each tasks add branching and loops
- Define jobs as code with Declarative Automation Bundles for CI/CD