Delta Lake & Apache Iceberg in open-source Spark

Apache Spark · DataFrames

Theory

Plain Parquet folders have no idea of a transaction: an overwrite deletes old files, a failed job can leave half-written files behind, and you cannot UPDATE, DELETE or MERGE individual rows. Open table formats fix this by keeping metadata next to the Parquet data files. Delta Lake keeps a _delta_log folder of commit files; Apache Iceberg keeps metadata and manifest files. Every write becomes an atomic commit that creates a new table version.

That gives ACID transactions, time travel (read an older version), row-level UPDATE, DELETE and MERGE INTO, schema evolution. In open-source Spark both are add-on libraries: you add the package for your Spark and Scala versions and register a SQL extension and a catalog. Delta uses spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension and spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog; Iceberg uses IcebergSparkSessionExtensions and a named catalog such as spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog.

Why we use it

Lakehouse platforms (Databricks, EMR, Dataproc, Snowflake and others) store tables in these formats, so data engineers are expected to know why they exist and how to enable them. They turn a folder of files into a reliable table.

When to use it

Use a table format for any table that is updated, merged into, read while being written, or needs history and audits. Plain Parquet is fine for write-once outputs and scratch data. Pick the format your platform and other engines support best; on Databricks, Delta is built in.

How it works

  1. Job writes to a table
  2. Plain Parquet path? Yes: Files replaced; no versions. No: Writer adds new data files.
  3. Writer adds a commit to the metadata (Delta log / Iceberg snapshot)
  4. Another writer committed first? Yes: Check conflicts and retry. No: Commit lands atomically.
  5. Readers see either the old or the new version, never half
  6. Older versions remain readable for time travel

Syntax

# Delta (pip install delta-spark; versions must match your Spark)
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
df.write.format("delta").save(path)
SELECT * FROM delta.`/path` VERSION AS OF 0
# Iceberg (iceberg-spark-runtime package)
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.local.type=hadoop ; spark.sql.catalog.local.warehouse=/path

Example

import os, tempfile
from pyspark.sql import SparkSession

# ---- Delta Lake / Iceberg (labelled snippet: needs the package jars, NOT run here) ----
# Delta Lake:
#   pip install delta-spark   (version matching your Spark release)
#   from delta import configure_spark_with_delta_pip
#   builder = (SparkSession.builder
#       .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
#       .config("spark.sql.catalog.spark_catalog",
#               "org.apache.spark.sql.delta.catalog.DeltaCatalog"))
#   spark = configure_spark_with_delta_pip(builder).getOrCreate()
#   df.write.format("delta").mode("overwrite").save("/tmp/orders_delta")
#   spark.sql("SELECT * FROM delta.`/tmp/orders_delta` VERSION AS OF 0")
#
# Apache Iceberg (add iceberg-spark-runtime-<spark>_<scala>:<version> via --packages):
#   --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
#   --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog
#   --conf spark.sql.catalog.local.type=hadoop
#   --conf spark.sql.catalog.local.warehouse=/tmp/iceberg_wh
#   spark.sql("CREATE TABLE local.db.orders (id INT, amount DOUBLE) USING iceberg")
# ---------------------------------------------------------------------------------------

# Runnable part: what plain Parquet folders are missing (the problem table formats solve)
spark = (SparkSession.builder.master("local[2]")
         .config("spark.ui.enabled", "false")
         .getOrCreate())
path = os.path.join(tempfile.mkdtemp(), "orders_parquet")

v1 = spark.createDataFrame([(101, 55000.0), (102, 30000.0)], "order_id INT, amount DOUBLE")
v1.write.mode("overwrite").parquet(path)
files = [f for f in os.listdir(path) if not f.startswith(".")]
print("v1 folder:", sum(f.endswith(".parquet") for f in files), "parquet files +", [f for f in files if f.startswith("_")])

v2 = spark.createDataFrame([(101, 56000.0)], "order_id INT, amount DOUBLE")
v2.write.mode("overwrite").parquet(path)          # old files are deleted, no history kept
print("rows now:", spark.read.parquet(path).count())
print("transaction log present:", os.path.exists(os.path.join(path, "_delta_log")))

# No UPDATE/MERGE on a plain Parquet path: you must rewrite files yourself
try:
    spark.sql(f"UPDATE parquet.`{path}` SET amount = 0")
except Exception as e:
    print("UPDATE on plain Parquet ->", type(e).__name__)

# Table formats keep a log of versions next to the Parquet files, which gives
# ACID commits, time travel (VERSION AS OF), UPDATE/DELETE/MERGE and schema evolution.

# Output:
# v1 folder: 2 parquet files + ['_SUCCESS']
# rows now: 1
# transaction log present: False
# UPDATE on plain Parquet -> UnsupportedOperationException

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic