Event Time, Watermarks & Window Aggregations
Apache Spark · Structured Streaming
Theory
Event time is when something actually happened (the event_ts inside the record), not when Spark sees it. Grouping by F.window("event_ts", "10 minutes") puts each event in a fixed, non-overlapping 10-minute bucket (tumbling); adding a slide argument makes overlapping sliding windows, and F.session_window(ts, "5 minutes") groups events separated by less than a gap.
Events can arrive late, so Spark cannot know when a window is finished. withWatermark("event_ts", "10 minutes") tells it: the watermark is the maximum event time seen so far minus 10 minutes. Windows that end before the watermark are final: in append mode they are emitted once, their state is dropped and older events may be ignored. A late event may still be counted if its window is still in state, but that is not guaranteed. For custom per-key state, the transformWithState API (transformWithStateInPandas in Python) offers value, list and map state with timers and TTL.
Why we use it
Real streams are out of order: mobile apps buffer events, networks lag. Event-time windows give correct per-period results regardless of arrival order, and watermarks stop state from growing forever, which would eventually crash a long-running query.
When to use it
Use event-time windows for per-minute or per-hour metrics, and always add a watermark to streaming aggregations, dedup and stream-stream joins that run long. Pick the watermark delay from how late real data arrives. Use transformWithState only for custom stateful logic that windows cannot express.
How it works
- Events arrive, possibly out of order
- Assign each event to window(event_ts, size)
- Update the window's running aggregate in state
- Advance watermark = max event_ts seen - delay
- Does a window end before the watermark? Yes: Emit it (append) and drop its state. No: Keep it open for late events.
- Correct per-window results with bounded state
Syntax
(events.withWatermark("event_ts", "10 minutes")
.groupBy(F.window("event_ts", "10 minutes"), "event_type")
.count())
F.window("event_ts", "10 minutes", "5 minutes") # sliding
F.session_window("event_ts", "5 minutes") # session
events.withWatermark("event_ts", "1 hour").dropDuplicatesWithinWatermark(["event_id"])Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("techtown-spark").getOrCreate()
import tempfile, os, json
base = tempfile.mkdtemp() # a real job would use cloud storage paths
src = os.path.join(base, "events_in"); os.makedirs(src)
def drop_file(name, rows): # simulate a new file landing in the folder
with open(os.path.join(src, name), "w") as f:
for r in rows:
f.write(json.dumps(dict(zip(["user_id", "event_type", "event_ts"], r))) + "\n")
schema = "user_id INT, event_type STRING, event_ts TIMESTAMP" # streams need a schema up front
drop_file("a.json", [(101, "login", "2026-01-05 09:01:00"), (102, "login", "2026-01-05 09:04:00"),
(101, "view", "2026-01-05 09:12:00")])
drop_file("b.json", [(103, "login", "2026-01-05 09:35:00")]) # watermark becomes 09:35 - 10 min = 09:25
drop_file("c.json", [(104, "login", "2026-01-05 09:03:00")]) # late, but its window is still in state
def show_batch(df, batch_id):
if not df.isEmpty():
print(f"--- batch {batch_id}: windows finalised by the watermark")
df.orderBy("start").show()
def run_once(): # e.g. a job scheduled every 15 minutes
events = spark.readStream.schema(schema).option("maxFilesPerTrigger", 1).json(src)
per_window = (events
.withWatermark("event_ts", "10 minutes")
.groupBy(F.window("event_ts", "10 minutes").alias("w"))
.agg(F.count("*").alias("events"), F.collect_set("user_id").alias("users"))
.select(F.date_format("w.start", "HH:mm").alias("start"),
F.date_format("w.end", "HH:mm").alias("end"), "events", "users"))
q = (per_window.writeStream.outputMode("append").foreachBatch(show_batch)
.option("checkpointLocation", os.path.join(base, "chk_win"))
.trigger(availableNow=True).start())
q.awaitTermination()
dropped = sum(p["stateOperators"][0]["numRowsDroppedByWatermark"] for p in q.recentProgress)
print("watermark:", q.lastProgress["eventTime"].get("watermark"), "| late rows dropped:", dropped)
run_once()
drop_file("d.json", [(105, "login", "2026-01-05 09:06:00")]) # late, and 09:00-09:10 was already emitted
run_once() # this row is dropped
# Output:
# --- batch 2: windows finalised by the watermark
# +-----+-----+------+---------------+
# |start| end|events| users|
# +-----+-----+------+---------------+
# |09:00|09:10| 3|[101, 102, 104]|
# |09:10|09:20| 1| [101]|
# +-----+-----+------+---------------+
#
# watermark: 2026-01-05T03:55:00.000Z | late rows dropped: 0
# watermark: 2026-01-05T03:55:00.000Z | late rows dropped: 1
Key points
- Event time = when it happened; window() groups by event time, not arrival time
- Watermark = max event time seen - delay; older windows are final
- Append mode emits a window once, after the watermark passes its end
- Events older than the watermark may be dropped (no guarantee either way)
- Longer delay = more lateness tolerated but more state kept
- transformWithState handles custom state with timers and TTL