Structured Streaming Basics
Databricks · Data Ingestion
Theory
Structured Streaming is Spark's engine for processing data that keeps arriving. You write the same DataFrame code you use for batch, but read with spark.readStream and write with writeStream. Spark runs it as a series of small micro-batches. Each run records progress (offsets, and state for aggregations) in a checkpoint location, which is what lets a stream restart after a failure and continue exactly where it left off without losing or duplicating data. The trigger decides when micro-batches run: availableNow=True processes everything available then stops, processingTime='1 minute' runs a batch every minute forever, and the default runs batches back to back.
Example: a Kafka or Auto Loader stream of orders feeds orders_bronze; a job runs it hourly with availableNow, so it behaves like an incremental batch.
Why we use it
It lets you process only new data incrementally with exactly-once guarantees, instead of re-reading everything. The same code can run as a cheap scheduled job or as a low-latency always-on stream just by changing the trigger.
When to use it
Use availableNow in scheduled jobs when minutes-to-hours latency is fine and you want compute to shut down. Use processingTime or continuous pipelines when the business needs near real-time data. For most ETL, Lakeflow pipelines manage streams and checkpoints for you.
How it works
- spark.readStream reads the source
- Read the last committed offsets from the checkpoint
- Process new data as a micro-batch with normal DataFrame logic
- Write results to the Delta table and commit offsets to the checkpoint
- Trigger is availableNow and backlog is empty? Yes: Stop the query. No: Wait for the next trigger and repeat.
- Restart later: resume from the checkpoint
Syntax
df = spark.readStream.format("delta"|"cloudFiles"|"kafka").load(...) | spark.readStream.table("t")
(df.writeStream
.option("checkpointLocation", "<path>")
.outputMode("append"|"update"|"complete")
.trigger(availableNow=True | processingTime="1 minute")
.toTable("<catalog.schema.table>"))Example
from pyspark.sql import functions as F
bronze = spark.readStream.table("main.sales.orders_bronze")
silver = (bronze
.filter(F.col("amount") > 0)
.withColumn("order_date", F.to_date("order_date")))
q = (silver.writeStream
.option("checkpointLocation", "/Volumes/main/sales/raw/_checkpoints/orders_silver")
.outputMode("append")
.trigger(availableNow=True) # or processingTime="1 minute"
.toTable("main.sales.orders_silver"))
q.awaitTermination()Key points
- Each stream needs its own checkpointLocation; deleting it reprocesses from scratch
- availableNow = incremental batch: process the backlog, then stop
- processingTime = run a micro-batch on a fixed interval until stopped
- Append mode suits plain inserts; aggregations without watermarks need update or complete
- Use withWatermark for streaming aggregations and joins to bound state
- Changing the query logic may need a new checkpoint, so plan your changes