What is Delta Lake (transaction log, ACID)

Databricks · Delta Lake

Theory

Delta Lake is the default table format on Databricks. A Delta table is a folder of Parquet data files plus a _delta_log folder, the transaction log. Every write (INSERT, UPDATE, MERGE, OPTIMIZE) adds a new numbered JSON commit file that lists which data files were added and removed; every few commits Spark writes a Parquet checkpoint so readers do not have to replay the whole log. A table's current state is simply the set of files the log says are live.

Because commits are atomic, Delta gives ACID transactions on cheap object storage: a reader never sees half a write, two writers cannot silently overwrite each other (optimistic concurrency), and failed jobs leave no partial data. The log also enables time travel, schema enforcement and DESCRIBE HISTORY. For example, CREATE TABLE main.sales.orders (...) creates version 0, an INSERT makes version 1 and an UPDATE makes version 2.

Why we use it

Plain Parquet folders have no transactions: a failed job can leave half-written files, concurrent writers corrupt data and you cannot update or delete rows safely. Delta adds database-style reliability on top of open files in your own storage.

When to use it

Use Delta for essentially every table you create on Databricks (it is the default). Choose Apache Iceberg with USING iceberg only when other engines in your stack require Iceberg. Keep raw landing files as CSV or JSON in volumes, then load them into Delta.

How it works

  1. Writer starts an INSERT, UPDATE or MERGE
  2. Write new Parquet data files
  3. Try to commit the next JSON entry to _delta_log
  4. Did another writer commit a conflicting change first? Yes: Retry or fail with a concurrency error; table stays consistent. No: Commit succeeds: new version is visible.
  5. Readers use the latest committed version

Syntax

CREATE TABLE main.sales.orders (...)          -- Delta by default
DESCRIBE HISTORY main.sales.orders
DESCRIBE DETAIL main.sales.orders
<table location>/_delta_log/00000000000000000002.json

Example

CREATE TABLE IF NOT EXISTS main.sales.orders (
  order_id    INT,
  customer_id INT,
  region      STRING,
  amount      DECIMAL(10,2),
  order_date  DATE
);                                                 -- version 0

INSERT INTO main.sales.orders VALUES
  (101, 1, 'West', 1200, '2026-01-05'),
  (102, 2, 'North', 800, '2026-01-06');           -- version 1

UPDATE main.sales.orders SET amount = 850
WHERE order_id = 102;                            -- version 2

DESCRIBE HISTORY main.sales.orders;

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic