The Streaming Model: Unbounded Table & Micro-batches

Apache Spark · Structured Streaming

Theory

Structured Streaming treats a live data stream as a table that keeps growing: every new event is a new row appended to an unbounded input table. You write the same DataFrame code you would write for a batch table, such as events.groupBy("event_type").count(), and Spark runs it incrementally. Each time it triggers, it reads only the new rows, updates the result (and any state, such as running counts) and writes the changes to a sink.

By default the engine runs in micro-batch mode: it collects whatever new data has arrived, processes it as a small batch job, records progress in a checkpoint, and repeats. That gives exactly-once results with replayable sources and idempotent or transactional sinks, with end-to-end latency typically from about a second upwards. You start a stream with spark.readStream, define transformations, and call writeStream ... start(), which returns a StreamingQuery you can monitor (status, lastProgress) or stop.

Why we use it

Writing separate code for batch and real-time pipelines doubles the work and the bugs. Structured Streaming lets you reuse DataFrame logic for continuously arriving data, while Spark handles incremental processing, state and recovery after failures.

When to use it

Use it when data keeps arriving (Kafka topics, files landing in storage, CDC feeds) and results must stay fresh, from seconds to hours. If data arrives once a day and is processed once, a plain batch job is simpler. For incremental batch, use a stream with trigger(availableNow=True).

How it works

  1. New events arrive at the source
  2. Trigger fires: read only the new rows (offsets since last batch)
  3. Run the query on them and update state, e.g. running counts
  4. Write changed results to the sink
  5. Record progress in the checkpoint
  6. Is the query still running? Yes: Wait for the next trigger and repeat. No: Stop.
  7. Results stay up to date

Syntax

stream = spark.readStream.format("json").schema(schema).load(path)
q = (stream.groupBy("event_type").count()
       .writeStream.outputMode("complete").format("memory").queryName("counts")
       .start())
q.status; q.lastProgress; q.awaitTermination(); q.stop()

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

import tempfile, os, json

base = tempfile.mkdtemp()                     # a real job would use cloud storage paths
src = os.path.join(base, "events_in"); os.makedirs(src)

def drop_file(name, rows):                    # simulate a new file landing in the folder
    with open(os.path.join(src, name), "w") as f:
        for r in rows:
            f.write(json.dumps(dict(zip(["user_id", "event_type", "event_ts"], r))) + "\n")

schema = "user_id INT, event_type STRING, event_ts TIMESTAMP"   # streams need a schema up front

drop_file("batch1.json", [(101, "login", "2026-01-05 09:00:00"), (101, "view", "2026-01-05 09:02:00"),
                          (102, "login", "2026-01-05 09:04:00")])

events = spark.readStream.schema(schema).json(src)        # unbounded input table
print("isStreaming:", events.isStreaming)

counts = events.groupBy("event_type").count()             # same code as batch

q = (counts.writeStream
        .outputMode("complete")
        .format("memory").queryName("event_counts")       # in-memory table, for testing
        .option("checkpointLocation", os.path.join(base, "chk"))
        .trigger(availableNow=True)                       # process what is there, then stop
        .start())
q.awaitTermination()

spark.sql("SELECT * FROM event_counts ORDER BY event_type").show()
p = q.lastProgress
print("batch:", p["batchId"], "| rows read:", p["numInputRows"])

# Output:
# isStreaming: True
# +----------+-----+
# |event_type|count|
# +----------+-----+
# |     login|    2|
# |      view|    1|
# +----------+-----+
#
# batch: 0 | rows read: 3

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic