The Streaming Model: Unbounded Table & Micro-batches
Apache Spark · Structured Streaming
Theory
Structured Streaming treats a live data stream as a table that keeps growing: every new event is a new row appended to an unbounded input table. You write the same DataFrame code you would write for a batch table, such as events.groupBy("event_type").count(), and Spark runs it incrementally. Each time it triggers, it reads only the new rows, updates the result (and any state, such as running counts) and writes the changes to a sink.
By default the engine runs in micro-batch mode: it collects whatever new data has arrived, processes it as a small batch job, records progress in a checkpoint, and repeats. That gives exactly-once results with replayable sources and idempotent or transactional sinks, with end-to-end latency typically from about a second upwards. You start a stream with spark.readStream, define transformations, and call writeStream ... start(), which returns a StreamingQuery you can monitor (status, lastProgress) or stop.
Why we use it
Writing separate code for batch and real-time pipelines doubles the work and the bugs. Structured Streaming lets you reuse DataFrame logic for continuously arriving data, while Spark handles incremental processing, state and recovery after failures.
When to use it
Use it when data keeps arriving (Kafka topics, files landing in storage, CDC feeds) and results must stay fresh, from seconds to hours. If data arrives once a day and is processed once, a plain batch job is simpler. For incremental batch, use a stream with trigger(availableNow=True).
How it works
- New events arrive at the source
- Trigger fires: read only the new rows (offsets since last batch)
- Run the query on them and update state, e.g. running counts
- Write changed results to the sink
- Record progress in the checkpoint
- Is the query still running? Yes: Wait for the next trigger and repeat. No: Stop.
- Results stay up to date
Syntax
stream = spark.readStream.format("json").schema(schema).load(path)
q = (stream.groupBy("event_type").count()
.writeStream.outputMode("complete").format("memory").queryName("counts")
.start())
q.status; q.lastProgress; q.awaitTermination(); q.stop()Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
import tempfile, os, json
base = tempfile.mkdtemp() # a real job would use cloud storage paths
src = os.path.join(base, "events_in"); os.makedirs(src)
def drop_file(name, rows): # simulate a new file landing in the folder
with open(os.path.join(src, name), "w") as f:
for r in rows:
f.write(json.dumps(dict(zip(["user_id", "event_type", "event_ts"], r))) + "\n")
schema = "user_id INT, event_type STRING, event_ts TIMESTAMP" # streams need a schema up front
drop_file("batch1.json", [(101, "login", "2026-01-05 09:00:00"), (101, "view", "2026-01-05 09:02:00"),
(102, "login", "2026-01-05 09:04:00")])
events = spark.readStream.schema(schema).json(src) # unbounded input table
print("isStreaming:", events.isStreaming)
counts = events.groupBy("event_type").count() # same code as batch
q = (counts.writeStream
.outputMode("complete")
.format("memory").queryName("event_counts") # in-memory table, for testing
.option("checkpointLocation", os.path.join(base, "chk"))
.trigger(availableNow=True) # process what is there, then stop
.start())
q.awaitTermination()
spark.sql("SELECT * FROM event_counts ORDER BY event_type").show()
p = q.lastProgress
print("batch:", p["batchId"], "| rows read:", p["numInputRows"])
# Output:
# isStreaming: True
# +----------+-----+
# |event_type|count|
# +----------+-----+
# | login| 2|
# | view| 1|
# +----------+-----+
#
# batch: 0 | rows read: 3
Key points
- A stream is an unbounded table; your query runs incrementally on new rows
- Same DataFrame API as batch: select, filter, groupBy, join
- Default engine is micro-batch: small batch jobs one after another
- start() returns a StreamingQuery: use status, lastProgress, stop()
- File sources need a schema up front (no inference on streams by default)