Pipeline Concepts & the Pipeline Editor

Databricks · Declarative Pipelines (Lakeflow)

Theory

Lakeflow Spark Declarative Pipelines, usually just called Lakeflow pipelines (formerly Delta Live Tables, or DLT, a name you will still see in older blogs and code), is the declarative way to build ETL on Databricks. Instead of writing the steps to run, you declare the datasets you want, streaming tables and materialized views, as SQL or Python queries. Databricks reads all the source files, works out the dependency graph, creates the tables in Unity Catalog, runs them in the right order, handles checkpoints and retries, and records everything in an event log.

Example: three SQL files declare orders_bronze (from read_files), orders_silver (cleaned from orders_bronze) and sales_by_region (aggregated from orders_silver). You never write "run bronze first": the pipeline sees the references and draws bronze → silver → gold in the pipeline graph.

Why we use it

Hand-written ETL needs code for ordering, incremental state, retries, data quality and monitoring. Declarative pipelines give you all of that from a few queries, so you focus on what the tables should contain rather than how to orchestrate them.

When to use it

Use a pipeline for multi-step ingestion and transformation, especially incremental or streaming data with quality rules. For one-off analysis use a notebook or SQL query; for orchestrating many different task types (notebooks, pipelines, dashboards) wrap the pipeline in a Lakeflow Job.

How it works

  1. + New → ETL pipeline, set default catalog main and schema sales
  2. Write SQL or Python files declaring streaming tables and materialized views
  3. Click Dry run to check syntax and the dependency graph
  4. Did the dry run report issues? Yes: Fix them in the Issues panel and dry run again. No: Click Run pipeline.
  5. Pipeline builds the graph and updates tables in dependency order
  6. Check Tables, Pipeline graph and Event log in the bottom panel

Syntax

UI: + New → ETL pipeline (opens the Lakeflow Pipelines Editor)
SQL: CREATE OR REFRESH STREAMING TABLE <name> AS SELECT ... FROM STREAM <source>;
     CREATE OR REFRESH MATERIALIZED VIEW <name> AS SELECT ...;
Python: from pyspark import pipelines as dp  →  @dp.table, @dp.materialized_view

Example

-- transformations/orders.sql  (pipeline default catalog main, schema sales)
CREATE OR REFRESH STREAMING TABLE orders_bronze AS
SELECT * FROM STREAM read_files('/Volumes/main/sales/raw/orders/', format => 'csv');

CREATE OR REFRESH STREAMING TABLE orders_silver AS
SELECT CAST(order_id AS BIGINT) AS order_id, customer_id, region,
       CAST(amount AS DECIMAL(10,2)) AS amount, CAST(order_date AS DATE) AS order_date
FROM STREAM(orders_bronze)
WHERE order_id IS NOT NULL;

CREATE OR REFRESH MATERIALIZED VIEW sales_by_region AS
SELECT region, SUM(amount) AS total_sales, COUNT(*) AS orders
FROM orders_silver
GROUP BY region;

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic