Watermarks: what problem do they solve?

What it is

A watermark is an assertion that flows through a stream saying "I do not expect to see any more events with a timestamp earlier than T". When a watermark passes the end of a window, the window fires.

It exists because stream processing distinguishes event time (when the thing happened, carried in the event) from processing time (when your system saw it). Those diverge constantly: mobile clients buffer offline and flush hours later, retries reorder, partitions have different lags, and a backfill replays last week at full speed.

Any computation that groups by time (per-minute counts, sessionisation, windowed joins) must therefore answer a question that has no exact answer: when have I seen everything for this window? A watermark is a heuristic answer to that question, and everything below follows from it being a heuristic rather than a guarantee.

Commonly confused with a trigger. A watermark is an assertion about completeness; a trigger is a policy about when to emit. Watermark-based firing is the default trigger, and you can also fire early (speculative results) or late (updates), which is the distinction the Dataflow model makes precise.

The problem it solves

Without event time you compute on arrival time, and your per-minute counts are counts of when data arrived, which is a property of your infrastructure rather than of the world. A mobile client that reconnects and flushes an hour of events puts all of them in the current minute. Every dashboard is then wrong in a way that correlates with network conditions.

With event time you get correct grouping and a new problem: an event-time window can never be proven complete, because a straggler may always arrive. You must trade completeness against latency, and the watermark is where you set that dial.

Mechanics

Generating a watermark

The common strategy is bounded out-of-orderness: assume events arrive at most $d$ behind the maximum timestamp seen so far.

// The watermark is (max event time seen) - d. Choose d from measured lateness,
// not from taste: plot the distribution of (processing time - event time) and
// pick a percentile you are willing to be complete at.
WatermarkStrategy
    .<Event>forBoundedOutOfOrderness(Duration.ofSeconds(20))
    .withTimestampAssigner((event, recordTimestamp) -> event.getEventTimeMillis())
    // Without this, ONE quiet partition freezes the whole job's watermark.
    .withIdleness(Duration.ofMinutes(1));

The choice of $d$ is the entire completeness-latency tradeoff expressed as a number:

$d$	Windows close	Events dropped or late
0 s	Immediately	Everything out of order
20 s	20 s after the window ends	Whatever is more than 20 s late
5 min	5 min after	Very little

Derive it from data: measure the distribution of arrival delay and set $d$ at, say, p99. Then handle the remaining 1 percent explicitly rather than pretending it does not exist.

How it propagates, and the two ways it stalls

An operator's watermark is the minimum across all its inputs, because it can only assert completeness up to the least-advanced source. That minimum rule is correct and it creates the two classic failures.

Idle partitions. If one Kafka partition stops producing, its watermark stops advancing, so the minimum stops, so no window anywhere in the job fires. The symptom is a job that consumes normally, emits nothing, and reports no errors. withIdleness marks a quiet source idle so it is excluded from the minimum, and omitting it is the single most common watermark bug.

Skewed sources. One partition an hour behind holds the global watermark an hour back, so every window waits and state grows. Flink's watermark alignment lets you bound the spread by pausing consumption from partitions that have run ahead:

WatermarkStrategy.<Event>forBoundedOutOfOrderness(Duration.ofSeconds(20))
    .withWatermarkAlignment("group-1", Duration.ofMinutes(1), Duration.ofSeconds(5));
    // Sources may not diverge by more than a minute in event time; fast ones
    // are throttled so state stays bounded during a backfill.

Alignment matters most on replay, where one partition may be minutes ahead and memory blows up holding open windows for the laggards.

What happens to late data

Three tiers, and a good answer names all three:

stream.keyBy(Event::getUserId)
      .window(TumblingEventTimeWindows.of(Time.minutes(1)))
      .allowedLateness(Time.minutes(5))          // 2. keep state, re-fire on late events
      .sideOutputLateData(lateTag)               // 3. route the truly late somewhere
      .aggregate(new CountAggregate());

On time: arrives before the watermark passes the window end. Included.
Late but within allowed lateness: the window state is retained for a further period, and a late event triggers a re-fire with an updated result. Downstream sinks must therefore be idempotent or upsert-capable, or you double-count. This is the constraint people miss.
Beyond allowed lateness: dropped by default. Always route it to a side output and count it, because silently dropped data is the failure mode that destroys trust in a pipeline. A late_events_total metric with an alert is non-negotiable.

Allowed lateness is not free: window state is held for window + lateness, so generous lateness means proportionally more state and more checkpoint cost.

Watermarks and checkpoints are different things

They travel through the same stream and solve different problems, and interviewers sometimes probe whether you know that.

A watermark carries event-time completeness and drives when results are emitted.
A checkpoint barrier implements Chandy-Lamport distributed snapshots for fault tolerance: operators snapshot state when the barrier arrives, and on failure the job restarts from the last complete snapshot.

Exactly-once end-to-end then needs a sink that participates in a two-phase commit tied to checkpoints, which is how Flink's Kafka sink works: begin a transaction after each checkpoint, commit when it completes.

A worked example

Ad click aggregation. Count clicks per campaign per minute. Sources: web (arrives in under a second) and mobile SDK (batches every 30 seconds, buffers offline for up to hours). Requirement: dashboards within 2 minutes, billing correct to the cent.

Measure first. Plot arrival delay:

p50   0.8 s
p90   6 s
p99   38 s
p999  4 min
max   6 h        (offline mobile clients flushing)

The design that follows. One watermark cannot serve both requirements, so use two paths over the same stream:

                        .-- watermark d=45s, no lateness --> dashboard sink
                        |   (fires ~45s after window end; ~1% of events missing)
kafka --> assign ts ----|
                        '-- watermark d=45s, allowedLateness=6h --> billing sink
                            (fires at 45s, RE-FIRES on late events, upsert sink)

Dashboards take $d = 45$ s (just past p99), fire once, and accept roughly 1 percent incompleteness. Latency requirement met.
Billing uses the same watermark but 6 hours of allowed lateness and an upsert sink keyed by (campaign, window_start), so each re-fire overwrites rather than adds. Correct to the cent, at the cost of holding 6 hours of window state.
Beyond 6 hours, a side output to a dead-letter topic with an alert. Over a month this catches a handful of events, and each one is investigated because it means a client behaved unexpectedly.

State cost, which is the thing to compute out loud. 6 hours of lateness, 1 minute windows, 50,000 active campaigns:

open windows = 6 h x 60 = 360 per campaign
state        = 360 x 50,000 x ~120 bytes = ~2.2 GB

That is comfortable on RocksDB state backend, not on the heap. Deriving the number is what makes "6 hours of lateness" a decision rather than a wish.

The incident that this design prevents, and the one it does not. It prevents the classic "billing was 3 percent low every month and nobody knew", because late events are counted rather than dropped. It does not prevent a stalled watermark freezing both paths, which is why the idleness setting and a watermark_lag alert matter more than any of the above:

# The watermark is the pipeline's most important health metric. If it stops
# advancing, everything downstream silently stops producing, with no errors.
(time() * 1000 - flink_taskmanager_job_task_operator_currentOutputWatermark) / 1000 > 300

Production evidence

Akidau et al., "The Dataflow Model" (VLDB 2015) is the primary source. It separates the four questions a streaming system must answer (what is computed, where in event time, when results are emitted, how refinements relate) and defines watermarks, triggers and accumulation modes precisely. Akidau's "Streaming 101 and 102" articles are the readable version and are the standard reference.

Apache Flink implements this model, and its documentation on watermark strategies, withIdleness, watermark alignment, allowed lateness and side outputs is the operational source for everything above. Google Cloud Dataflow implements the same model, being the productisation of the paper's system.

Chandy and Lamport, "Distributed Snapshots" (1985) is the algorithm behind Flink's checkpoint barriers, and the reason checkpointing and watermarking are separate mechanisms travelling in the same stream.

Kafka Streams takes a deliberately different approach with a simpler stream-time model and grace periods rather than full watermark propagation, which is a good contrast to draw: less powerful, considerably simpler to operate.

The debate

The alternative is processing time, which is simpler, has no watermarks, no late data and no window state held open. It is correct when the question is genuinely about your system ("requests per second hitting this service") rather than about the world ("clicks per minute per campaign").

The other alternative is the Lambda architecture: an approximate streaming path for freshness plus a batch path that recomputes the truth nightly. It works, and it costs you two implementations of the same business logic that must agree, which they eventually will not.

My position: event time with watermarks for anything where the timestamp is a property of the world; processing time where it is a property of your infrastructure. Derive $d$ from the measured arrival-delay distribution rather than picking a round number. Use two paths from one stream when latency and completeness requirements genuinely differ, rather than compromising both. And always route late data to a side output with a metric, because silently dropped events are how a pipeline loses its users' trust.

Watermarks are the wrong machinery when the data is genuinely in order (a single partition, a single producer), when you have no windowing at all (stateless transformation), or when the correct answer really is "count what arrived", where introducing event time adds complexity for no benefit.

Follow-up Q&A

"Watermarks in Flink: what problem do they solve?" They tell the system when it is safe to close an event-time window. In stream processing you care about when something happened rather than when you saw it, because events arrive late and out of order, so a per-minute window needs an answer to "have I seen everything for this minute". A watermark asserts that no events earlier than T are expected. It is a heuristic, so it is a completeness-versus-latency dial, and the escape hatches for what arrives afterwards are allowed lateness (retain state and re-fire) and side outputs (route the truly late somewhere countable).

"Your job consumes normally and emits nothing. What happened?" Almost certainly a stalled watermark. An operator's watermark is the minimum across its inputs, so one idle partition freezes the whole job: no window fires, no error is raised, lag looks fine. The fix is an idleness timeout on the source so quiet partitions are excluded from the minimum. Confirm it by graphing currentOutputWatermark against wall clock; that gap is the metric to alert on, and it should be on the dashboard before the incident.

"How do you choose the out-of-orderness bound?" From measurement, not taste. Plot the distribution of processing time minus event time over a representative period, and set $d$ at a percentile you are willing to be complete at, typically p99. Then handle the remainder explicitly with allowed lateness and a side output. Picking a round number like "5 minutes" without looking at the distribution is how you end up either dropping real data or holding windows open far longer than necessary.

"What must be true of your sink if you use allowed lateness?" It must be idempotent or upsert-capable, because a late event causes the window to re-fire with an updated result. If the sink appends, you double-count, and the pipeline is wrong in a way that looks like inflated traffic rather than a bug. Key the upsert by the window identity, typically the grouping key plus window start, so the re-fire overwrites cleanly.

"How are watermarks different from checkpoint barriers?" Different problems, same stream. Watermarks carry event-time completeness and drive when results are emitted. Checkpoint barriers implement Chandy-Lamport distributed snapshots for fault tolerance: operators snapshot state when the barrier passes, and recovery restarts from the last complete snapshot. Exactly-once end to end then needs a sink that ties a two-phase commit to checkpoint completion, which is how Flink's Kafka sink achieves it.

Common misconceptions

The most common is that a watermark guarantees completeness. It is a heuristic assertion, which is precisely why allowed lateness and side outputs exist. Treating it as a guarantee leads to dropping late data without noticing.

The second is that late data is rare and can be ignored. On any pipeline with mobile clients it is a long tail measured in hours, and it is systematically biased toward users with poor connectivity, so dropping it skews the data rather than merely reducing it.

The third is that a bigger out-of-orderness bound is safer. It delays every window, holds more state, and increases checkpoint cost. The right structure is a modest bound plus explicit handling of the tail.

Interview delivery note

Say this: "They tell the system when it's safe to close an event-time window. You care about event time rather than processing time because events arrive late and out of order, so a watermark is an assertion that nothing earlier than T is still coming. It's a heuristic, so it's really a completeness-versus-latency dial, and I'd set the bound from the measured distribution of arrival delay rather than picking a round number. Then two escape hatches for the tail: allowed lateness, which keeps the window state and re-fires, and a side output so the truly late events are counted rather than silently dropped."

The depth signal is the operational failure: "the thing I'd watch for is a stalled watermark from an idle partition. The operator's watermark is the minimum across its inputs, so one quiet partition freezes the whole job and no windows fire at all, with no error and normal-looking lag. That's what the idleness setting is for, and watermark lag is the metric I'd alert on." And if allowed lateness comes up, add that the sink must be upsert-capable, because a re-fire double-counts otherwise.

The Staff & Team Lead Interview Reference