Delta Lake & Apache Iceberg in open-source Spark
Apache Spark · DataFrames
Theory
Plain Parquet folders have no idea of a transaction: an overwrite deletes old files, a failed job can leave half-written files behind, and you cannot UPDATE, DELETE or MERGE individual rows. Open table formats fix this by keeping metadata next to the Parquet data files. Delta Lake keeps a _delta_log folder of commit files; Apache Iceberg keeps metadata and manifest files. Every write becomes an atomic commit that creates a new table version.
That gives ACID transactions, time travel (read an older version), row-level UPDATE, DELETE and MERGE INTO, schema evolution. In open-source Spark both are add-on libraries: you add the package for your Spark and Scala versions and register a SQL extension and a catalog. Delta uses spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension and spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog; Iceberg uses IcebergSparkSessionExtensions and a named catalog such as spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog.
Why we use it
Lakehouse platforms (Databricks, EMR, Dataproc, Snowflake and others) store tables in these formats, so data engineers are expected to know why they exist and how to enable them. They turn a folder of files into a reliable table.
When to use it
Use a table format for any table that is updated, merged into, read while being written, or needs history and audits. Plain Parquet is fine for write-once outputs and scratch data. Pick the format your platform and other engines support best; on Databricks, Delta is built in.
How it works
- Job writes to a table
- Plain Parquet path? Yes: Files replaced; no versions. No: Writer adds new data files.
- Writer adds a commit to the metadata (Delta log / Iceberg snapshot)
- Another writer committed first? Yes: Check conflicts and retry. No: Commit lands atomically.
- Readers see either the old or the new version, never half
- Older versions remain readable for time travel
Syntax
# Delta (pip install delta-spark; versions must match your Spark)
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
df.write.format("delta").save(path)
SELECT * FROM delta.`/path` VERSION AS OF 0
# Iceberg (iceberg-spark-runtime package)
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.local.type=hadoop ; spark.sql.catalog.local.warehouse=/pathExample
import os, tempfile
from pyspark.sql import SparkSession
# ---- Delta Lake / Iceberg (labelled snippet: needs the package jars, NOT run here) ----
# Delta Lake:
# pip install delta-spark (version matching your Spark release)
# from delta import configure_spark_with_delta_pip
# builder = (SparkSession.builder
# .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
# .config("spark.sql.catalog.spark_catalog",
# "org.apache.spark.sql.delta.catalog.DeltaCatalog"))
# spark = configure_spark_with_delta_pip(builder).getOrCreate()
# df.write.format("delta").mode("overwrite").save("/tmp/orders_delta")
# spark.sql("SELECT * FROM delta.`/tmp/orders_delta` VERSION AS OF 0")
#
# Apache Iceberg (add iceberg-spark-runtime-<spark>_<scala>:<version> via --packages):
# --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
# --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog
# --conf spark.sql.catalog.local.type=hadoop
# --conf spark.sql.catalog.local.warehouse=/tmp/iceberg_wh
# spark.sql("CREATE TABLE local.db.orders (id INT, amount DOUBLE) USING iceberg")
# ---------------------------------------------------------------------------------------
# Runnable part: what plain Parquet folders are missing (the problem table formats solve)
spark = (SparkSession.builder.master("local[2]")
.config("spark.ui.enabled", "false")
.getOrCreate())
path = os.path.join(tempfile.mkdtemp(), "orders_parquet")
v1 = spark.createDataFrame([(101, 55000.0), (102, 30000.0)], "order_id INT, amount DOUBLE")
v1.write.mode("overwrite").parquet(path)
files = [f for f in os.listdir(path) if not f.startswith(".")]
print("v1 folder:", sum(f.endswith(".parquet") for f in files), "parquet files +", [f for f in files if f.startswith("_")])
v2 = spark.createDataFrame([(101, 56000.0)], "order_id INT, amount DOUBLE")
v2.write.mode("overwrite").parquet(path) # old files are deleted, no history kept
print("rows now:", spark.read.parquet(path).count())
print("transaction log present:", os.path.exists(os.path.join(path, "_delta_log")))
# No UPDATE/MERGE on a plain Parquet path: you must rewrite files yourself
try:
spark.sql(f"UPDATE parquet.`{path}` SET amount = 0")
except Exception as e:
print("UPDATE on plain Parquet ->", type(e).__name__)
# Table formats keep a log of versions next to the Parquet files, which gives
# ACID commits, time travel (VERSION AS OF), UPDATE/DELETE/MERGE and schema evolution.
# Output:
# v1 folder: 2 parquet files + ['_SUCCESS']
# rows now: 1
# transaction log present: False
# UPDATE on plain Parquet -> UnsupportedOperationException
Key points
- Both store data as Parquet plus a metadata layer that records commits
- Each write is an atomic commit and a new version (time travel)
- Enable row-level UPDATE, DELETE and MERGE INTO
- Open-source Spark needs the package jars plus extension and catalog configs
- Package versions must match your Spark and Scala versions
- Plain Parquet overwrite deletes history; there is no rollback