Sources, Sinks & Output Modes
Apache Spark · Structured Streaming
Theory
A streaming query reads from a source and writes to a sink. Built-in sources include files in a folder (json, csv, parquet, text; each new file is read once), Kafka, and the test sources rate and socket. Built-in sinks include files, Kafka, foreach and foreachBatch, plus console and memory for debugging. foreachBatch(func) hands you each micro-batch as a normal DataFrame plus a batch id, so you can use any batch writer, such as a MERGE upsert or a JDBC write.
The output mode says what is written each trigger. Append writes only new result rows and is the default; it works for queries without aggregation, and for aggregations only with a watermark (rows are emitted once final). Update writes only rows that changed since the last trigger. Complete rewrites the whole result table every time and is only allowed for aggregations. Not every sink supports every mode: the file sink supports only append.
Why we use it
Choosing the right source, sink and output mode decides whether results are correct, complete and efficient. A wrong mode either fails at start (for example append on an aggregate without a watermark) or rewrites far more data than needed.
When to use it
Use append for event-level pipelines (clean and land events). Use update for aggregates feeding a key-value store or a MERGE in foreachBatch. Use complete only for small aggregate tables such as dashboards. Use console and memory sinks only for testing.
How it works
- Pick a source: files, Kafka or a table format
- Does the query aggregate? Yes: Choose update, complete, or append with a watermark. No: Use append.
- Does the sink support that mode? Yes: Configure the sink and checkpoint. No: Use foreachBatch or another sink.
- Start the query with writeStream ... start()
- Each trigger writes rows according to the output mode
Syntax
spark.readStream.format("kafka").option("kafka.bootstrap.servers", hosts).option("subscribe", topic).load()
spark.readStream.schema(s).option("maxFilesPerTrigger", 1).json(path)
df.writeStream.outputMode("append" | "update" | "complete")
.format("parquet").option("path", out).option("checkpointLocation", chk)
df.writeStream.foreachBatch(lambda batch_df, batch_id: ...)Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
import tempfile, os, json
base = tempfile.mkdtemp() # a real job would use cloud storage paths
src = os.path.join(base, "events_in"); os.makedirs(src)
def drop_file(name, rows): # simulate a new file landing in the folder
with open(os.path.join(src, name), "w") as f:
for r in rows:
f.write(json.dumps(dict(zip(["user_id", "event_type", "event_ts"], r))) + "\n")
schema = "user_id INT, event_type STRING, event_ts TIMESTAMP" # streams need a schema up front
drop_file("f1.json", [(101, "login", "2026-01-05 09:00:00"), (101, "view", "2026-01-05 09:02:00")])
drop_file("f2.json", [(102, "login", "2026-01-05 09:04:00"), (101, "purchase", "2026-01-05 09:07:00")])
events = spark.readStream.schema(schema).option("maxFilesPerTrigger", 1).json(src) # one file per batch
# 1. Append mode, file sink: each event is written once
q1 = (events.withColumn("event_date", F.to_date("event_ts"))
.writeStream.outputMode("append").format("parquet")
.option("path", os.path.join(base, "events_out"))
.option("checkpointLocation", os.path.join(base, "chk_append"))
.trigger(availableNow=True).start())
q1.awaitTermination()
print("rows landed:", spark.read.parquet(os.path.join(base, "events_out")).count())
# 2. Update mode with foreachBatch: see what each micro-batch emits
def show_batch(batch_df, batch_id):
print(f"--- micro-batch {batch_id} (update mode: changed rows only)")
batch_df.orderBy("event_type").show()
q2 = (events.groupBy("event_type").count()
.writeStream.outputMode("update").foreachBatch(show_batch)
.option("checkpointLocation", os.path.join(base, "chk_update"))
.trigger(availableNow=True).start())
q2.awaitTermination()
# Output:
# rows landed: 4
# --- micro-batch 0 (update mode: changed rows only)
# +----------+-----+
# |event_type|count|
# +----------+-----+
# | login| 1|
# | view| 1|
# +----------+-----+
#
# --- micro-batch 1 (update mode: changed rows only)
# +----------+-----+
# |event_type|count|
# +----------+-----+
# | login| 2|
# | purchase| 1|
# +----------+-----+
Key points
- Append (default): only new rows; aggregations need a watermark to use it
- Update: only rows that changed this trigger
- Complete: whole result each trigger; aggregations only
- File sink supports append only; console and memory are for testing
- foreachBatch lets you reuse any batch writer (MERGE, JDBC) per micro-batch