Triggered vs Continuous & Run Behaviour

Databricks · Declarative Pipelines (Lakeflow)

Theory

A pipeline's Pipeline mode is either triggered or continuous. A triggered pipeline processes all data available when the update starts, refreshes every table once and then stops its compute; you run it manually, on a schedule or from a job. A continuous pipeline keeps running, processing new data as soon as it arrives for low latency, and keeps its compute up the whole time. Separately, how an update behaves depends on how it was started. Updates you start from the editor or the pipeline page use fast-start, debugging-friendly behaviour (this is what development mode used to mean): compute is reused for quick reruns and retries are off so you see errors immediately. Updates started from a job or the API, and all continuous updates, get production behaviour: automatic retries and restarts on recoverable errors, and compute shuts down after the run.

Why we use it

Latency and cost pull in opposite directions. Pipeline mode lets you choose fresh-every-few-seconds data or cheap scheduled refreshes, and the run behaviour keeps development fast while production runs stay resilient.

When to use it

Use triggered mode for most batch and incremental ETL, scheduled hourly or daily through a job. Use continuous only when the business needs data within seconds or minutes. While building, run from the editor; in production, run from a Lakeflow Job so updates get retries.

How it works

  1. Decide how fresh the tables must be
  2. Is seconds-to-minutes latency required? Yes: Set Pipeline mode to Continuous. No: Keep Triggered.
  3. Develop and test with Run pipeline from the editor (fast-start, no retries)
  4. Ready for production? Yes: Schedule it with a Lakeflow Job (retries enabled). No: Keep iterating in the editor.
  5. Monitor updates in Jobs & Pipelines

Syntax

UI: pipeline Settings → Pipeline mode: Triggered | Continuous
Run: Run pipeline | Run pipeline with full table refresh | Dry run
Schedule: Schedule button in the editor (creates a job)  or  a Pipeline task in Lakeflow Jobs
Settings JSON: "continuous": false, "development": false

Example

// Pipeline settings (JSON view) for a production, triggered pipeline
{
  "name": "orders_pipeline",
  "catalog": "main",
  "schema": "sales",
  "serverless": true,
  "continuous": false,
  "development": false,
  "libraries": [{ "glob": { "include": "/Workspace/Users/asha@techtown.in/orders_pipeline/transformations/**" } }]
}

-- Then schedule it hourly with a Lakeflow Job (Pipeline task)
-- or with the Schedule button in the Lakeflow Pipelines Editor.

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic