Triggers & Checkpoints
Apache Spark · Structured Streaming
Theory
A trigger decides when the next micro-batch runs. With no trigger set, Spark starts the next batch as soon as the previous one finishes. trigger(processingTime="1 minute") runs a batch at fixed intervals. trigger(availableNow=True) processes everything available at start, in one or more batches, and then stops; it replaces the older trigger(once=True). That makes cheap scheduled incremental jobs. An experimental continuous trigger also exists for very low latency.
A checkpoint folder records the source offsets each batch read, which batches finished (commits) and any state, such as running counts. When the query restarts with the same checkpoint, it continues exactly where it stopped, with no data skipped or repeated for replayable sources and idempotent sinks. Each query needs its own checkpoint folder on reliable storage, and some query changes are incompatible with an existing one.
Why we use it
Triggers let you trade latency for cost: always-on streams for freshness, availableNow for scheduled incremental loads. Checkpoints are what make restarts, upgrades and failures safe, which is essential for any production pipeline.
When to use it
Use processingTime for always-on streams with a steady cadence, the default trigger when lowest micro-batch latency matters, and availableNow for jobs run hourly or daily that should pick up only new data. Always set a checkpointLocation in production, one per query, and never share or delete it casually.
How it works
- Query starts with a checkpointLocation
- Read the last committed offsets from the checkpoint
- Trigger fires: plan a batch from those offsets to the latest available
- Write the planned offsets, process the batch, write the sink, then commit
- Is it availableNow and the backlog is empty? Yes: Stop the query. No: Wait for the next trigger.
- On restart, resume from the last commit
Syntax
.trigger(processingTime="30 seconds")
.trigger(availableNow=True) # process backlog, then stop
.option("checkpointLocation", "s3://bucket/chk/orders_silver")
q.awaitTermination(); q.recentProgress
# checkpoint folder: offsets/ commits/ sources/ state/ metadataExample
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
import tempfile, os, json
base = tempfile.mkdtemp() # a real job would use cloud storage paths
src = os.path.join(base, "events_in"); os.makedirs(src)
def drop_file(name, rows): # simulate a new file landing in the folder
with open(os.path.join(src, name), "w") as f:
for r in rows:
f.write(json.dumps(dict(zip(["user_id", "event_type", "event_ts"], r))) + "\n")
schema = "user_id INT, event_type STRING, event_ts TIMESTAMP" # streams need a schema up front
chk = os.path.join(base, "chk_orders_events")
out = os.path.join(base, "events_out")
def run_incremental():
q = (spark.readStream.schema(schema).json(src)
.writeStream.format("parquet").option("path", out)
.option("checkpointLocation", chk)
.trigger(availableNow=True) # pick up only what is new, then stop
.start())
q.awaitTermination()
return q.lastProgress["numInputRows"] if q.lastProgress else 0
drop_file("day1.json", [(101, "login", "2026-01-05 09:00:00"), (102, "login", "2026-01-05 09:04:00")])
print("run 1 read rows:", run_incremental())
drop_file("day2.json", [(103, "login", "2026-01-06 10:00:00")])
print("run 2 read rows:", run_incremental()) # only the new file, thanks to the checkpoint
print("run 3 read rows:", run_incremental()) # nothing new
print("total rows in sink:", spark.read.parquet(out).count())
print("checkpoint contents:", sorted(os.listdir(chk)))
# Output:
# run 1 read rows: 2
# run 2 read rows: 1
# run 3 read rows: 0
# total rows in sink: 3
# checkpoint contents: ['.metadata.crc', 'commits', 'metadata', 'offsets', 'sources']
Key points
- No trigger: next micro-batch starts as soon as the last one ends
- processingTime='1 minute': fixed-interval micro-batches
- availableNow=True: process the backlog in batches, then stop (preferred over once=True)
- Checkpoint stores offsets, commits and state so restarts resume exactly
- One checkpoint folder per query, on reliable storage
- Some query changes are incompatible with an old checkpoint