Structured Streaming Basics

Databricks · Data Ingestion

Theory

Structured Streaming is Spark's engine for processing data that keeps arriving. You write the same DataFrame code you use for batch, but read with spark.readStream and write with writeStream. Spark runs it as a series of small micro-batches. Each run records progress (offsets, and state for aggregations) in a checkpoint location, which is what lets a stream restart after a failure and continue exactly where it left off without losing or duplicating data. The trigger decides when micro-batches run: availableNow=True processes everything available then stops, processingTime='1 minute' runs a batch every minute forever, and the default runs batches back to back.

Example: a Kafka or Auto Loader stream of orders feeds orders_bronze; a job runs it hourly with availableNow, so it behaves like an incremental batch.

Why we use it

It lets you process only new data incrementally with exactly-once guarantees, instead of re-reading everything. The same code can run as a cheap scheduled job or as a low-latency always-on stream just by changing the trigger.

When to use it

Use availableNow in scheduled jobs when minutes-to-hours latency is fine and you want compute to shut down. Use processingTime or continuous pipelines when the business needs near real-time data. For most ETL, Lakeflow pipelines manage streams and checkpoints for you.

How it works

  1. spark.readStream reads the source
  2. Read the last committed offsets from the checkpoint
  3. Process new data as a micro-batch with normal DataFrame logic
  4. Write results to the Delta table and commit offsets to the checkpoint
  5. Trigger is availableNow and backlog is empty? Yes: Stop the query. No: Wait for the next trigger and repeat.
  6. Restart later: resume from the checkpoint

Syntax

df = spark.readStream.format("delta"|"cloudFiles"|"kafka").load(...) | spark.readStream.table("t")
(df.writeStream
   .option("checkpointLocation", "<path>")
   .outputMode("append"|"update"|"complete")
   .trigger(availableNow=True | processingTime="1 minute")
   .toTable("<catalog.schema.table>"))

Example

from pyspark.sql import functions as F

bronze = spark.readStream.table("main.sales.orders_bronze")

silver = (bronze
  .filter(F.col("amount") > 0)
  .withColumn("order_date", F.to_date("order_date")))

q = (silver.writeStream
  .option("checkpointLocation", "/Volumes/main/sales/raw/_checkpoints/orders_silver")
  .outputMode("append")
  .trigger(availableNow=True)      # or processingTime="1 minute"
  .toTable("main.sales.orders_silver"))

q.awaitTermination()

Key points

Databricks topics

Databricks Fundamentals

Compute

Notebooks & Development

Apache Spark on Databricks

Delta Lake

Unity Catalog & Governance

Data Ingestion

Declarative Pipelines (Lakeflow)

Lakeflow Jobs

Databricks SQL & BI

Architecture, Performance & Cost

Machine Learning & AI

Certification & Interview

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Databricks

The lakehouse platform from the ground up: Apache Spark, Delta Lake, Unity Catalog, Lakeflow pipelines and jobs, Databricks SQL and MLflow. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice SQL / Spark SQL Sharpen the SQL you use in Databricks SQL and Spark SQL: 150 query problems in a live editor with instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Databricks  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Databricks needs a real workspace — there's no in-browser engine for it. Click Copy above, then paste it into a notebook or the SQL editor in Databricks Free Edition (free, no credit card) and run it on serverless compute.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic