What is Delta Lake (transaction log, ACID)
Databricks · Delta Lake
Theory
Delta Lake is the default table format on Databricks. A Delta table is a folder of Parquet data files plus a _delta_log folder, the transaction log. Every write (INSERT, UPDATE, MERGE, OPTIMIZE) adds a new numbered JSON commit file that lists which data files were added and removed; every few commits Spark writes a Parquet checkpoint so readers do not have to replay the whole log. A table's current state is simply the set of files the log says are live.
Because commits are atomic, Delta gives ACID transactions on cheap object storage: a reader never sees half a write, two writers cannot silently overwrite each other (optimistic concurrency), and failed jobs leave no partial data. The log also enables time travel, schema enforcement and DESCRIBE HISTORY. For example, CREATE TABLE main.sales.orders (...) creates version 0, an INSERT makes version 1 and an UPDATE makes version 2.
Why we use it
Plain Parquet folders have no transactions: a failed job can leave half-written files, concurrent writers corrupt data and you cannot update or delete rows safely. Delta adds database-style reliability on top of open files in your own storage.
When to use it
Use Delta for essentially every table you create on Databricks (it is the default). Choose Apache Iceberg with USING iceberg only when other engines in your stack require Iceberg. Keep raw landing files as CSV or JSON in volumes, then load them into Delta.
How it works
- Writer starts an INSERT, UPDATE or MERGE
- Write new Parquet data files
- Try to commit the next JSON entry to _delta_log
- Did another writer commit a conflicting change first? Yes: Retry or fail with a concurrency error; table stays consistent. No: Commit succeeds: new version is visible.
- Readers use the latest committed version
Syntax
CREATE TABLE main.sales.orders (...) -- Delta by default
DESCRIBE HISTORY main.sales.orders
DESCRIBE DETAIL main.sales.orders
<table location>/_delta_log/00000000000000000002.jsonExample
CREATE TABLE IF NOT EXISTS main.sales.orders (
order_id INT,
customer_id INT,
region STRING,
amount DECIMAL(10,2),
order_date DATE
); -- version 0
INSERT INTO main.sales.orders VALUES
(101, 1, 'West', 1200, '2026-01-05'),
(102, 2, 'North', 800, '2026-01-06'); -- version 1
UPDATE main.sales.orders SET amount = 850
WHERE order_id = 102; -- version 2
DESCRIBE HISTORY main.sales.orders;Key points
- Delta = Parquet data files + _delta_log transaction log
- Each commit is atomic, so readers see a complete version or nothing
- Optimistic concurrency detects conflicting writers instead of corrupting data
- Checkpoints let readers skip replaying every JSON commit
- Delta is the default format for tables created on Databricks