Pipeline Concepts & the Pipeline Editor
Databricks · Declarative Pipelines (Lakeflow)
Theory
Lakeflow Spark Declarative Pipelines, usually just called Lakeflow pipelines (formerly Delta Live Tables, or DLT, a name you will still see in older blogs and code), is the declarative way to build ETL on Databricks. Instead of writing the steps to run, you declare the datasets you want, streaming tables and materialized views, as SQL or Python queries. Databricks reads all the source files, works out the dependency graph, creates the tables in Unity Catalog, runs them in the right order, handles checkpoints and retries, and records everything in an event log.
Example: three SQL files declare orders_bronze (from read_files), orders_silver (cleaned from orders_bronze) and sales_by_region (aggregated from orders_silver). You never write "run bronze first": the pipeline sees the references and draws bronze → silver → gold in the pipeline graph.
Why we use it
Hand-written ETL needs code for ordering, incremental state, retries, data quality and monitoring. Declarative pipelines give you all of that from a few queries, so you focus on what the tables should contain rather than how to orchestrate them.
When to use it
Use a pipeline for multi-step ingestion and transformation, especially incremental or streaming data with quality rules. For one-off analysis use a notebook or SQL query; for orchestrating many different task types (notebooks, pipelines, dashboards) wrap the pipeline in a Lakeflow Job.
How it works
- + New → ETL pipeline, set default catalog main and schema sales
- Write SQL or Python files declaring streaming tables and materialized views
- Click Dry run to check syntax and the dependency graph
- Did the dry run report issues? Yes: Fix them in the Issues panel and dry run again. No: Click Run pipeline.
- Pipeline builds the graph and updates tables in dependency order
- Check Tables, Pipeline graph and Event log in the bottom panel
Syntax
UI: + New → ETL pipeline (opens the Lakeflow Pipelines Editor)
SQL: CREATE OR REFRESH STREAMING TABLE <name> AS SELECT ... FROM STREAM <source>;
CREATE OR REFRESH MATERIALIZED VIEW <name> AS SELECT ...;
Python: from pyspark import pipelines as dp → @dp.table, @dp.materialized_viewExample
-- transformations/orders.sql (pipeline default catalog main, schema sales)
CREATE OR REFRESH STREAMING TABLE orders_bronze AS
SELECT * FROM STREAM read_files('/Volumes/main/sales/raw/orders/', format => 'csv');
CREATE OR REFRESH STREAMING TABLE orders_silver AS
SELECT CAST(order_id AS BIGINT) AS order_id, customer_id, region,
CAST(amount AS DECIMAL(10,2)) AS amount, CAST(order_date AS DATE) AS order_date
FROM STREAM(orders_bronze)
WHERE order_id IS NOT NULL;
CREATE OR REFRESH MATERIALIZED VIEW sales_by_region AS
SELECT region, SUM(amount) AS total_sales, COUNT(*) AS orders
FROM orders_silver
GROUP BY region;Key points
- Former name: Delta Live Tables (DLT); Python module is now pyspark.pipelines (import as dp)
- You declare tables; the pipeline infers order from the references between them
- Run pipeline updates all tables; Dry run validates code without writing data
- Default catalog and schema are set in pipeline Settings
- Every update writes an event log with lineage, data quality and timing
- Serverless compute is the default for new pipelines