Sources, Sinks & Output Modes

Apache Spark · Structured Streaming

Theory

A streaming query reads from a source and writes to a sink. Built-in sources include files in a folder (json, csv, parquet, text; each new file is read once), Kafka, and the test sources rate and socket. Built-in sinks include files, Kafka, foreach and foreachBatch, plus console and memory for debugging. foreachBatch(func) hands you each micro-batch as a normal DataFrame plus a batch id, so you can use any batch writer, such as a MERGE upsert or a JDBC write.

The output mode says what is written each trigger. Append writes only new result rows and is the default; it works for queries without aggregation, and for aggregations only with a watermark (rows are emitted once final). Update writes only rows that changed since the last trigger. Complete rewrites the whole result table every time and is only allowed for aggregations. Not every sink supports every mode: the file sink supports only append.

Why we use it

Choosing the right source, sink and output mode decides whether results are correct, complete and efficient. A wrong mode either fails at start (for example append on an aggregate without a watermark) or rewrites far more data than needed.

When to use it

Use append for event-level pipelines (clean and land events). Use update for aggregates feeding a key-value store or a MERGE in foreachBatch. Use complete only for small aggregate tables such as dashboards. Use console and memory sinks only for testing.

How it works

  1. Pick a source: files, Kafka or a table format
  2. Does the query aggregate? Yes: Choose update, complete, or append with a watermark. No: Use append.
  3. Does the sink support that mode? Yes: Configure the sink and checkpoint. No: Use foreachBatch or another sink.
  4. Start the query with writeStream ... start()
  5. Each trigger writes rows according to the output mode

Syntax

spark.readStream.format("kafka").option("kafka.bootstrap.servers", hosts).option("subscribe", topic).load()
spark.readStream.schema(s).option("maxFilesPerTrigger", 1).json(path)
df.writeStream.outputMode("append" | "update" | "complete")
  .format("parquet").option("path", out).option("checkpointLocation", chk)
df.writeStream.foreachBatch(lambda batch_df, batch_id: ...)

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

import tempfile, os, json

base = tempfile.mkdtemp()                     # a real job would use cloud storage paths
src = os.path.join(base, "events_in"); os.makedirs(src)

def drop_file(name, rows):                    # simulate a new file landing in the folder
    with open(os.path.join(src, name), "w") as f:
        for r in rows:
            f.write(json.dumps(dict(zip(["user_id", "event_type", "event_ts"], r))) + "\n")

schema = "user_id INT, event_type STRING, event_ts TIMESTAMP"   # streams need a schema up front

drop_file("f1.json", [(101, "login", "2026-01-05 09:00:00"), (101, "view", "2026-01-05 09:02:00")])
drop_file("f2.json", [(102, "login", "2026-01-05 09:04:00"), (101, "purchase", "2026-01-05 09:07:00")])

events = spark.readStream.schema(schema).option("maxFilesPerTrigger", 1).json(src)  # one file per batch

# 1. Append mode, file sink: each event is written once
q1 = (events.withColumn("event_date", F.to_date("event_ts"))
        .writeStream.outputMode("append").format("parquet")
        .option("path", os.path.join(base, "events_out"))
        .option("checkpointLocation", os.path.join(base, "chk_append"))
        .trigger(availableNow=True).start())
q1.awaitTermination()
print("rows landed:", spark.read.parquet(os.path.join(base, "events_out")).count())

# 2. Update mode with foreachBatch: see what each micro-batch emits
def show_batch(batch_df, batch_id):
    print(f"--- micro-batch {batch_id} (update mode: changed rows only)")
    batch_df.orderBy("event_type").show()

q2 = (events.groupBy("event_type").count()
        .writeStream.outputMode("update").foreachBatch(show_batch)
        .option("checkpointLocation", os.path.join(base, "chk_update"))
        .trigger(availableNow=True).start())
q2.awaitTermination()

# Output:
# rows landed: 4
# --- micro-batch 0 (update mode: changed rows only)
# +----------+-----+
# |event_type|count|
# +----------+-----+
# |     login|    1|
# |      view|    1|
# +----------+-----+
#
# --- micro-batch 1 (update mode: changed rows only)
# +----------+-----+
# |event_type|count|
# +----------+-----+
# |     login|    2|
# |  purchase|    1|
# +----------+-----+

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic