Triggers & Checkpoints

Apache Spark · Structured Streaming

Theory

A trigger decides when the next micro-batch runs. With no trigger set, Spark starts the next batch as soon as the previous one finishes. trigger(processingTime="1 minute") runs a batch at fixed intervals. trigger(availableNow=True) processes everything available at start, in one or more batches, and then stops; it replaces the older trigger(once=True). That makes cheap scheduled incremental jobs. An experimental continuous trigger also exists for very low latency.

A checkpoint folder records the source offsets each batch read, which batches finished (commits) and any state, such as running counts. When the query restarts with the same checkpoint, it continues exactly where it stopped, with no data skipped or repeated for replayable sources and idempotent sinks. Each query needs its own checkpoint folder on reliable storage, and some query changes are incompatible with an existing one.

Why we use it

Triggers let you trade latency for cost: always-on streams for freshness, availableNow for scheduled incremental loads. Checkpoints are what make restarts, upgrades and failures safe, which is essential for any production pipeline.

When to use it

Use processingTime for always-on streams with a steady cadence, the default trigger when lowest micro-batch latency matters, and availableNow for jobs run hourly or daily that should pick up only new data. Always set a checkpointLocation in production, one per query, and never share or delete it casually.

How it works

  1. Query starts with a checkpointLocation
  2. Read the last committed offsets from the checkpoint
  3. Trigger fires: plan a batch from those offsets to the latest available
  4. Write the planned offsets, process the batch, write the sink, then commit
  5. Is it availableNow and the backlog is empty? Yes: Stop the query. No: Wait for the next trigger.
  6. On restart, resume from the last commit

Syntax

.trigger(processingTime="30 seconds")
.trigger(availableNow=True)             # process backlog, then stop
.option("checkpointLocation", "s3://bucket/chk/orders_silver")
q.awaitTermination(); q.recentProgress
# checkpoint folder: offsets/  commits/  sources/  state/  metadata

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

import tempfile, os, json

base = tempfile.mkdtemp()                     # a real job would use cloud storage paths
src = os.path.join(base, "events_in"); os.makedirs(src)

def drop_file(name, rows):                    # simulate a new file landing in the folder
    with open(os.path.join(src, name), "w") as f:
        for r in rows:
            f.write(json.dumps(dict(zip(["user_id", "event_type", "event_ts"], r))) + "\n")

schema = "user_id INT, event_type STRING, event_ts TIMESTAMP"   # streams need a schema up front

chk = os.path.join(base, "chk_orders_events")
out = os.path.join(base, "events_out")

def run_incremental():
    q = (spark.readStream.schema(schema).json(src)
            .writeStream.format("parquet").option("path", out)
            .option("checkpointLocation", chk)
            .trigger(availableNow=True)          # pick up only what is new, then stop
            .start())
    q.awaitTermination()
    return q.lastProgress["numInputRows"] if q.lastProgress else 0

drop_file("day1.json", [(101, "login", "2026-01-05 09:00:00"), (102, "login", "2026-01-05 09:04:00")])
print("run 1 read rows:", run_incremental())

drop_file("day2.json", [(103, "login", "2026-01-06 10:00:00")])
print("run 2 read rows:", run_incremental())     # only the new file, thanks to the checkpoint

print("run 3 read rows:", run_incremental())     # nothing new
print("total rows in sink:", spark.read.parquet(out).count())
print("checkpoint contents:", sorted(os.listdir(chk)))

# Output:
# run 1 read rows: 2
# run 2 read rows: 1
# run 3 read rows: 0
# total rows in sink: 3
# checkpoint contents: ['.metadata.crc', 'commits', 'metadata', 'offsets', 'sources']

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic