Jobs & Task DAGs

Databricks · Lakeflow Jobs

Theory

Lakeflow Jobs (formerly Databricks Workflows, and before that just Jobs) is the built-in orchestrator. A job is a set of tasks plus settings for compute, triggers, parameters and notifications. Each task runs one thing: a notebook, a Lakeflow pipeline, a SQL query or file, a Python script or wheel, a dbt project, a dashboard refresh, another job, or control-flow tasks such as If/else condition and For each. Tasks are linked with Depends on, forming a DAG (directed acyclic graph), so independent tasks run in parallel and a task starts only when its upstream tasks finish. Run if lets a task run on other outcomes, for example only when something failed.

Example: job orders_daily runs ingest_orders (pipeline), then clean_orders and load_customers in parallel, then build_gold (SQL), then refresh_dashboard.

Why we use it

Real data flows have many steps that must run in order, retry on failure and alert someone when they break. Lakeflow Jobs does this inside Databricks with no external scheduler, using serverless or job compute and Unity Catalog permissions.

When to use it

Use Lakeflow Jobs to schedule and chain anything that runs on Databricks. Inside one pipeline, let the pipeline handle table order; use a job to combine pipelines with notebooks, SQL, dashboards and alerts. External orchestrators like Airflow can still trigger jobs if your company already uses them.

How it works

  1. Jobs & Pipelines → Create → Job
  2. Add the first task: pick type, source and compute
  3. + Add task for each next step and set Depends on
  4. Should a step run only on failure or always? Yes: Set Run if dependencies (e.g. At least one failed). No: Keep All succeeded.
  5. Add triggers, parameters and notifications in Job details
  6. Click Run now and watch the task graph

Syntax

UI: Jobs & Pipelines → Create → Job → choose a task type tile → + Add task
Task fields: Task name, Type, Source/Path, Compute, Depends on, Run if dependencies, Parameters
As code: Declarative Automation Bundles (formerly Databricks Asset Bundles) YAML, or the Jobs API

Example

# resources/orders_daily.job.yml  (Declarative Automation Bundle)
resources:
  jobs:
    orders_daily:
      name: orders_daily
      tasks:
        - task_key: ingest_orders
          pipeline_task:
            pipeline_id: ${resources.pipelines.orders_pipeline.id}
        - task_key: clean_orders
          depends_on: [{ task_key: ingest_orders }]
          notebook_task:
            notebook_path: ../src/clean_orders.ipynb
        - task_key: build_gold
          depends_on: [{ task_key: clean_orders }]
          sql_task:
            file: { path: ../src/build_gold.sql }
            warehouse_id: ${var.warehouse_id}

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic