DT4: Streams, big data and file formats
Processing streams: windows and late events
🎯You'll be able to
Explain how stream processors group events into time windows, why event time differs from arrival time, and how watermarks decide when a window is complete.
🏠The everyday picture
A teacher collects homework due at 9 a.m. but accepts work up to 9:10 from students stuck in traffic. After 9:10, the pile is marked. A watermark is that 9:10.
📘What it really is
-
Event time is when something happened; processing time is when the processor saw it. Phones go offline, so events can arrive minutes or hours late.
-
Windows group events by event time:
- tumbling: fixed, non-overlapping (orders per 15 minutes);
- sliding: overlapping (orders in the last 15 minutes, updated every minute);
-
session: activity separated by a gap (a customer's browsing session).
-
A watermark says "events older than this have probably all arrived". When the watermark passes a window's end, the window's result is published. Events later than the watermark are dropped, sent to a side output, or used to update the result, depending on the setting.
-
Apache Flink is a common stream processor with event time, windows, watermarks and state. Spark Structured Streaming and Kafka Streams offer similar ideas.
-
Exactly-once results are possible inside some processors (Flink with checkpoints and a transactional sink), but the whole system end to end is still usually built to tolerate repeats.
📝A worked example
Input (illustrative, worked by hand): 15-minute tumbling windows by event time, watermark = newest event time seen minus 5 minutes. Events arrive in this order: 10:02, 10:09, 10:14, 10:21, 10:13 (late), 10:23, 10:31, 10:12 (very late).
Expected output: after 10:21 arrives, the watermark is 10:16, so the 10:00–10:15 window closes with 3 orders (10:02, 10:09, 10:14). The 10:13 event arrives after that and is late. By the time 10:31 arrives, the watermark is 10:26, still inside the 10:15–10:30 window, which holds 10:21 and 10:23 so far. The 10:12 event is late too. With a 10-minute allowed lateness, both late events would be added and the first window updated to 5.
🌍Where it's used
- Ride-hailing: demand per area per minute for surge pricing.
- Monitoring: error counts per 5 minutes from millions of devices.
- Gaming and streaming: live viewer counts.
📝Case scenario
A ticketing site counted sales per minute by processing time. When its message system slowed during a big release, a backlog of sales was processed together, and the chart showed a false spike two minutes after the real one. Switching to event time with a watermark put sales in the minute they happened.
🖱️Try it
Using the worked example's rules, add one more event at 10:36 after the 10:12 event. Say which window closes and what its count is. Success: the watermark becomes 10:31, so the 10:15–10:30 window closes with 2 orders.
⚠️Common mistakes
- Grouping by processing time. Sign: spikes after slowdowns. Fix: group by event time.
- A watermark that's too tight. Sign: many late events dropped. Fix: measure how late events really arrive and allow for it.
- Results without a correction plan. Sign: live numbers never match the daily batch. Fix: decide how late events update results, and which figure is trusted.
🛠️For your project
Not in the kit's tests. Use this lesson for the capstone's streaming design notes.
✅Quick check
What's the difference between event time and processing time?
Show the answer
Event time is when it happened; processing time is when the system saw it.
What does a watermark decide?
Show the answer
When a window is treated as complete and its result published.