Change Data Feed
Databricks · Delta Lake
Theory
Change Data Feed (CDF) makes a Delta table record row-level changes, not just new versions. You enable it with the table property delta.enableChangeDataFeed = true. From then on, every INSERT, UPDATE, DELETE and MERGE also writes change records, and you can read them with the table_changes('main.sales.orders', start_version) SQL function or spark.read.option("readChangeFeed", "true").option("startingVersion", 2).table(...).
Each change row has the table's columns plus three metadata columns: _change_type (insert, update_preimage, update_postimage or delete), _commit_version and _commit_timestamp. An update produces two rows: the old values (preimage) and the new values (postimage). For example, if version 2 updates order 102 from 800 to 850 and version 3 deletes order 103, table_changes from version 2 returns three rows. Downstream jobs can then process only what changed instead of rereading the whole table.
Why we use it
Incremental pipelines need to know which rows changed so silver and gold tables, caches or external systems can be updated cheaply. CDF gives that list directly from the Delta log, including updates and deletes that plain appends cannot show.
When to use it
Enable CDF on tables whose changes feed downstream tables, audits or syncs. It costs some extra storage. For append-only tables, a normal streaming read is enough. CDF only covers changes made after it was enabled.
How it works
- Enable CDF on the table
- A job runs UPDATE, DELETE or MERGE
- Delta writes data files plus change records
- Was the row updated? Yes: Record update_preimage and update_postimage. No: Record insert or delete.
- Downstream job reads table_changes from its last processed version
Syntax
ALTER TABLE t SET TBLPROPERTIES (delta.enableChangeDataFeed = true)
SELECT * FROM table_changes('main.sales.orders', 2 [, end_version])
spark.read.option("readChangeFeed", "true").option("startingVersion", 2).table(t)
Metadata: _change_type, _commit_version, _commit_timestampExample
ALTER TABLE main.sales.orders
SET TBLPROPERTIES (delta.enableChangeDataFeed = true); -- version 1
UPDATE main.sales.orders SET amount = 850 WHERE order_id = 102; -- version 2
DELETE FROM main.sales.orders WHERE order_id = 103; -- version 3
SELECT order_id, amount, _change_type, _commit_version
FROM table_changes('main.sales.orders', 2)
ORDER BY _commit_version, order_id, _change_type;
# Python, incremental read
changes = (spark.read.option("readChangeFeed", "true")
.option("startingVersion", 2)
.table("main.sales.orders"))Key points
- Enable with delta.enableChangeDataFeed = true; changes before that are not recorded
- An update gives two rows: update_preimage and update_postimage
- _commit_version and _commit_timestamp tell you when each change happened
- Read with table_changes() in SQL or readChangeFeed in Python/streaming
- Change files are cleaned up by VACUUM like other files