Event Time, Watermarks & Window Aggregations

Apache Spark · Structured Streaming

Theory

Event time is when something actually happened (the event_ts inside the record), not when Spark sees it. Grouping by F.window("event_ts", "10 minutes") puts each event in a fixed, non-overlapping 10-minute bucket (tumbling); adding a slide argument makes overlapping sliding windows, and F.session_window(ts, "5 minutes") groups events separated by less than a gap.

Events can arrive late, so Spark cannot know when a window is finished. withWatermark("event_ts", "10 minutes") tells it: the watermark is the maximum event time seen so far minus 10 minutes. Windows that end before the watermark are final: in append mode they are emitted once, their state is dropped and older events may be ignored. A late event may still be counted if its window is still in state, but that is not guaranteed. For custom per-key state, the transformWithState API (transformWithStateInPandas in Python) offers value, list and map state with timers and TTL.

Why we use it

Real streams are out of order: mobile apps buffer events, networks lag. Event-time windows give correct per-period results regardless of arrival order, and watermarks stop state from growing forever, which would eventually crash a long-running query.

When to use it

Use event-time windows for per-minute or per-hour metrics, and always add a watermark to streaming aggregations, dedup and stream-stream joins that run long. Pick the watermark delay from how late real data arrives. Use transformWithState only for custom stateful logic that windows cannot express.

How it works

  1. Events arrive, possibly out of order
  2. Assign each event to window(event_ts, size)
  3. Update the window's running aggregate in state
  4. Advance watermark = max event_ts seen - delay
  5. Does a window end before the watermark? Yes: Emit it (append) and drop its state. No: Keep it open for late events.
  6. Correct per-window results with bounded state

Syntax

(events.withWatermark("event_ts", "10 minutes")
   .groupBy(F.window("event_ts", "10 minutes"), "event_type")
   .count())
F.window("event_ts", "10 minutes", "5 minutes")     # sliding
F.session_window("event_ts", "5 minutes")            # session
events.withWatermark("event_ts", "1 hour").dropDuplicatesWithinWatermark(["event_id"])

Example

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("techtown-spark").getOrCreate()

import tempfile, os, json

base = tempfile.mkdtemp()                     # a real job would use cloud storage paths
src = os.path.join(base, "events_in"); os.makedirs(src)

def drop_file(name, rows):                    # simulate a new file landing in the folder
    with open(os.path.join(src, name), "w") as f:
        for r in rows:
            f.write(json.dumps(dict(zip(["user_id", "event_type", "event_ts"], r))) + "\n")

schema = "user_id INT, event_type STRING, event_ts TIMESTAMP"   # streams need a schema up front

drop_file("a.json", [(101, "login", "2026-01-05 09:01:00"), (102, "login", "2026-01-05 09:04:00"),
                     (101, "view",  "2026-01-05 09:12:00")])
drop_file("b.json", [(103, "login", "2026-01-05 09:35:00")])     # watermark becomes 09:35 - 10 min = 09:25
drop_file("c.json", [(104, "login", "2026-01-05 09:03:00")])     # late, but its window is still in state

def show_batch(df, batch_id):
    if not df.isEmpty():
        print(f"--- batch {batch_id}: windows finalised by the watermark")
        df.orderBy("start").show()

def run_once():                                   # e.g. a job scheduled every 15 minutes
    events = spark.readStream.schema(schema).option("maxFilesPerTrigger", 1).json(src)
    per_window = (events
        .withWatermark("event_ts", "10 minutes")
        .groupBy(F.window("event_ts", "10 minutes").alias("w"))
        .agg(F.count("*").alias("events"), F.collect_set("user_id").alias("users"))
        .select(F.date_format("w.start", "HH:mm").alias("start"),
                F.date_format("w.end", "HH:mm").alias("end"), "events", "users"))
    q = (per_window.writeStream.outputMode("append").foreachBatch(show_batch)
            .option("checkpointLocation", os.path.join(base, "chk_win"))
            .trigger(availableNow=True).start())
    q.awaitTermination()
    dropped = sum(p["stateOperators"][0]["numRowsDroppedByWatermark"] for p in q.recentProgress)
    print("watermark:", q.lastProgress["eventTime"].get("watermark"), "| late rows dropped:", dropped)

run_once()
drop_file("d.json", [(105, "login", "2026-01-05 09:06:00")])     # late, and 09:00-09:10 was already emitted
run_once()                                                       # this row is dropped

# Output:
# --- batch 2: windows finalised by the watermark
# +-----+-----+------+---------------+
# |start|  end|events|          users|
# +-----+-----+------+---------------+
# |09:00|09:10|     3|[101, 102, 104]|
# |09:10|09:20|     1|          [101]|
# +-----+-----+------+---------------+
#
# watermark: 2026-01-05T03:55:00.000Z | late rows dropped: 0
# watermark: 2026-01-05T03:55:00.000Z | late rows dropped: 1

Key points

Apache Spark topics

Spark Fundamentals

Spark Architecture

RDDs

DataFrames

Aggregations & Joins

Spark SQL

Window Functions

Cleaning & Complex Types

UDFs

Performance Tuning

Structured Streaming

MLlib & Ecosystem

Interview Prep

T
Tech Town
Log InSign Up Free
Home SQL Python DSA Power BI Snowflake Databricks Spark Gen AI Practice Problems Contest Blog

Master Apache Spark

PySpark and Spark SQL from the ground up: DataFrames, joins, window functions, partitions and performance tuning, Structured Streaming and more. Click any topic in the sidebar to open it right here.

{{ sectionCount }}
Sections
{{ topicCount }}
Topics
Free
Forever
Hands-on practice Practice PySpark Write real PySpark — DataFrames, joins, window functions and Spark SQL — in a live editor with hidden tests and instant checking. Start practising →
Section {{ sec.num }}

{{ sec.label }}

Spark  /  {{ current.sectionLabel }}  /  {{ current.name }}

{{ current.name }}

Theory

{{ current.def }}

Why we use it

{{ rich.why }}

When to use it

{{ rich.when }}

{{ sec.h }}

{{ sec.body }}

{{ sec.body }}

Diagram

{{ rich.dg.caption }}

{{ tb.title }}
{{ c.v }}
{{ rich.dg.op }}
➜
{{ rich.dg.output.title }}
{{ c.v }}
{{ nd.label }}
{{ nd.sub }}
{{ nd.label }}
{{ nd.sub }}
{{ rich.dg.left.title }}
{{ it }}
{{ rich.dg.right.title }}
{{ it }}
On screen — where to click

{{ rich.ui.title }}

{{ u.text }}
  1. {{ st.n }}{{ st.text }}
{{ rich.ui.title }}
{{ u.text }}
Flowchart — how it works
{{ nd.text }}
{{ nd.text }}
◇ {{ nd.text }}
Yes → {{ nd.yes }}
No → {{ nd.no }}
Syntax
{{ current.syntax }}
Example
{{ current.example }}
Want to run this yourself? Try the same API on Practice PySpark Online right in your browser, or run it locally after pip install pyspark (needs Java 17+), or paste it into a Databricks Free Edition notebook.
Key Points — click to check off as you learn
  • {{ pt.mark }} {{ pt.text }}

Share this topic