Reconnect, resume, and client-side dedupe
What it is
A long-lived stream will disconnect. The design question is what the client sees when it comes back: a gap it does not know about, a gap it does know about, or the events it missed.
DELIVERY GUARANTEE WHAT THE CLIENT GETS BACK
at-most-once whatever is happening now. Anything
during the disconnect is lost, and
the client does not know it was lost.
at-least-once everything after the last event it
acknowledged, possibly including some
it already has.
exactly-once EFFECT at-least-once plus client-side
deduplication, or idempotent
application, so the observable result
is correct.
Server-sent events has resume built in: the server sets an id: on each event, the browser's
EventSource reconnects automatically and sends the last one back in a Last-Event-ID header.
WebSocket has nothing, so you build the same mechanism yourself.
What this is confused with: reconnection and resumption. Reconnecting is trivial and every client library does it. Resuming means the server can answer "what did I miss since event 4711", which requires a replay buffer and a monotonic id, and neither exists by default.
Also confused: exactly-once delivery. It is not available. At-least-once plus deduplication is the achievable thing, and the dedupe belongs on the client because the server cannot know what the client already applied.
The problem it solves
The silent gap is the failure, because nothing reports it.
A dashboard streaming order events over WebSocket.
t+0 connection drops (a wifi handover, 4 seconds)
t+4s client reconnects, server starts sending current
events
t+4s the 11 orders that arrived during those 4 seconds
are never sent
The dashboard now shows a total that is 11 orders low, and it
stays wrong until the next full refresh, which may be never.
Nothing errored. No metric moved. The user sees a number and
believes it.
And the naive fix produces the opposite failure:
"On reconnect, resend everything from the last 5 minutes."
the client receives ~40 events it already applied
order totals double-count
a "new order" toast fires 40 times
and on a flapping connection this repeats
At-least-once without dedupe is a correctness bug in the
other direction.
The third failure, which appears at scale:
A deploy drops 100,000 connections. Every client reconnects
and asks to resume. The server now has to serve 100,000
replay queries against its buffer, simultaneously, on top of
the live stream.
The replay path, which is never load-tested, becomes the
bottleneck during exactly the event that triggers it.
Mechanics
SSE: what you get for free, and what you must add
The server's obligations:
event: order.created
id: 4711 <- REQUIRED for resume. Monotonic
data: {"orderId":"..."} <- per stream.
retry: 3000 <- reconnection delay hint, in ms
The browser's EventSource then:
- reconnects automatically on disconnect
- sends Last-Event-ID: 4711 on the reconnect request
- waits `retry` ms (with no jitter of its own, which is the
gap you must fill)
Your handler reads the header and replays from there.
@app.get("/stream")
async def stream(request: Request, last_event_id: str | None = Header(None)):
async def gen():
# Resume, or start from now. The distinction between
# "no header" (a new client) and "header we cannot
# satisfy" (a client that was away too long) is the
# important one.
if last_event_id is not None:
since = int(last_event_id)
if since < await buffer.oldest_id(stream_key):
# The gap is REAL and we cannot fill it. Say so
# explicitly rather than silently continuing,
# so the client can do a full resync.
yield "event: resync_required\ndata: {}\n\n"
else:
async for e in buffer.replay(stream_key, after=since):
yield sse(e)
async for e in live(stream_key):
yield sse(e)
return StreamingResponse(gen(), media_type="text/event-stream", headers={
"Cache-Control": "no-cache, no-transform",
"X-Accel-Buffering": "no", # or a proxy buffers it and it is
# not a stream at all
"Connection": "keep-alive",
})
The resync_required branch is the part that gets omitted, and omitting it converts a detectable
gap into a silent one, which is the failure this whole page exists to prevent.
The replay buffer
The buffer answers "give me everything after id X for this
stream". Three implementations, with different trade-offs:
IN-MEMORY RING BUFFER, per stream
+ zero infrastructure, microsecond replay
- lost on restart, which is exactly when everyone
reconnects
- does not work with more than one server unless the
client returns to the same one
Right for: short windows on a single-instance service.
REDIS STREAM
XADD with MAXLEN ~ N, XRANGE for replay.
+ survives an app restart, shared across instances, and
the id is already a monotonic stream id
+ MAXLEN gives you a bounded window for free
Right for: most systems. This is the default answer.
THE LOG ITSELF (Kafka and similar)
The topic is the buffer, the offset is the cursor, and
retention is the window.
+ no separate buffer, and the window can be days
- a consumer per connected client is not viable at high
connection counts, so you need a fan-out layer that
itself keeps a cursor per client
Right for: internal consumers, and as the source the
fan-out layer reads from.
Sizing the window is a product decision, not a technical one:
The window must exceed the longest disconnect you intend to
resume across.
a wifi handover seconds
a mobile tunnel or lift 30 to 120 seconds
a laptop lid closed minutes to hours
a deploy of the streaming
tier seconds, but for everyone at
once
A 30-second buffer does not survive a 90-second tunnel, and
the client will resync. Whether that is acceptable is a
question about what a resync costs: if it is a 40 KB snapshot,
a short window is fine; if it is a 4 MB rebuild, buy a longer
window.
Monotonic ids, and why not UUIDs
The id must be MONOTONIC PER STREAM, because two operations
depend on ordering:
RESUME: "everything after 4711" is a range query. With a
UUID it is a lookup followed by a scan, and if the id is
unknown you cannot tell whether it is too old or invalid.
GAP DETECTION: a client receiving 4711 then 4713 knows it
lost 4712 without asking anyone. With UUIDs it cannot.
Use a per-stream sequence, a Redis stream id, or a Kafka
offset. If you need global uniqueness as well, a composite
(stream_key, seq) or a UUIDv7 gives ordering and uniqueness
together.
Client-side dedupe
At-least-once means the boundary events arrive twice. The client must handle it.
// A bounded set of recently applied ids. Bounded matters: an
// unbounded Set is a memory leak on a stream that runs for
// days.
class Dedupe {
constructor(capacity = 1000) {
this.capacity = capacity
this.seen = new Set()
this.order = []
}
isNew(id) {
if (this.seen.has(id)) return false
this.seen.add(id)
this.order.push(id)
if (this.order.length > this.capacity) {
this.seen.delete(this.order.shift())
}
return true
}
}
// Capacity must exceed the largest replay you will ever
// receive, or you will re-apply an event that fell out of the
// window. Size it from the replay buffer's window times the
// peak event rate.
Better than dedupe, where the data model allows it: make application idempotent.
DEDUPE keep a set of ids, drop repeats. Works for
everything, costs memory, and the capacity is
a parameter that can be wrong.
IDEMPOTENT apply by key with a version:
APPLICATION state[e.orderId] = e if e.version >
current.version
Re-applying the same event is a no-op by
construction, and no id set is needed.
The second is strictly better where the events are state
updates rather than increments. It fails for events whose
effect is cumulative ("add 3 to the total"), and the fix
there is to send the total rather than the delta, which is
usually possible and usually better anyway.
"Send the state, not the delta" removes the dedupe problem entirely for a large class of streams, and it is worth checking before building an id set.
The operational details that break streams
IDLE TIMEOUTS. A load balancer with a 60-second idle timeout
kills an SSE connection that has had no events for 60
seconds. Send a heartbeat:
: ping\n\n (an SSE comment; the client
ignores it, the LB sees traffic)
every 15 to 30 seconds. This is the most common cause of
"the stream drops every minute".
PROXY BUFFERING. nginx and friends buffer responses by
default, so the stream is not a stream. `X-Accel-Buffering:
no`, `proxy_buffering off`, and `Cache-Control:
no-transform`. It works perfectly in local development,
which is why it reaches production.
HTTP/1.1 CONNECTION LIMIT. Browsers allow ~6 connections per
host, and an SSE stream holds one open indefinitely. Six
tabs and the seventh hangs. HTTP/2 removes this, and it is
a real reason to require it for SSE.
RECONNECT STORMS. The `retry:` hint has no jitter, so every
client reconnects at the same offset after a mass
disconnect. Add jitter client-side, and see the
connection-draining discussion for the server side.
COMPRESSION. Some intermediaries buffer to compress.
`no-transform` and, if necessary, disabling compression on
the stream endpoint.
A worked example: a dashboard that was quietly wrong
An operations dashboard: WebSocket, ~1,400 concurrent connections, streaming order and shipment events. A recurring, unreproducible report that "the totals are sometimes wrong".
The investigation:
Instrumented the client to log every gap in a sequence it did
not have (the events carried no sequence, so the first change
was adding one).
Over one week, across 1,400 clients:
reconnects 38,000
reconnects with a detectable gap 31,200 (82%)
median events missed per gap 3
p99 47
So roughly four out of five reconnects lost events, and
nobody had ever seen an error, because the client simply
continued from the next live event.
Cause: reconnection was implemented (a retry loop) and
resumption was not. The two had been treated as the same
thing.
Eighty-two percent of reconnects lost data silently, and the only reason it was noticed at all was that a human occasionally checked a total against the database.
The implementation:
1. MONOTONIC IDS. A per-stream sequence, assigned at publish
time, carried on every event.
2. REPLAY BUFFER: a Redis stream per tenant, XADD with
MAXLEN ~ 50000, which at the observed peak rate of ~120
events/sec per tenant is about 7 minutes of window.
Sized from the disconnect distribution: p99 disconnect
duration was 41 seconds, p99.9 was 6 minutes (laptop lids).
7 minutes covers p99.9.
3. RESUME PROTOCOL on the WebSocket, since it has none:
client -> {"type":"subscribe","stream":"orders",
"afterSeq": 4711}
server -> replays from the buffer, then switches to live
server -> {"type":"resync_required"} if 4711 is older
than the buffer
4. CLIENT DEDUPE: a bounded set of the last 2,000 sequence
numbers. 2,000 > the maximum replay (7 minutes x 120/sec
would be 50,400, so the set was NOT large enough and this
was caught in review).
Changed to idempotent application instead: events carry
the full order state and a version, and the client applies
by key if the version is newer. No id set at all.
5. HEARTBEAT every 20 seconds, because the load balancer's
idle timeout was 60.
6. JITTERED RECONNECT: full jitter, 0.5s base, 30s cap.
The dedupe-set sizing error caught in review is the instructive part: a bounded set must exceed the largest possible replay, and the largest possible replay is the buffer window times the peak rate, which was 25 times the proposed capacity. Switching to idempotent application removed the parameter entirely, which is why it is the better answer where the data model allows it.
Then the reconnect-storm problem, found in a load test rather than in production:
Simulated a deploy: dropped all 1,400 connections at once.
all 1,400 reconnected within 2 seconds (the retry hint had
no jitter)
each requested a replay of ~40 seconds of events
the Redis XRANGE calls: 1,400 in ~2 seconds, each returning
~4,800 events
Redis CPU: 22% -> 96%
p99 replay latency: 4ms -> 3.1 seconds
and 340 clients timed out and retried, making it worse
The replay path had never been load-tested, because it only
runs during the event that causes it.
Fixes:
- full jitter on the client reconnect: 1,400 reconnects
spread over 30 seconds instead of 2
- a server-side admission limit on concurrent replays, with
the excess told to wait rather than being served slowly
- replay results capped: if a client needs more than 5,000
events, send resync_required instead, because a snapshot is
cheaper than a 40,000-event replay
- and the snapshot endpoint was made cheap enough to be the
fallback: 38 KB for a full tenant state
Retest: 1,400 simultaneous disconnects, Redis peak 41% CPU,
p99 replay 22ms, zero timeouts.
"If the replay is bigger than the snapshot, send the snapshot" is the rule that fell out, and it bounds the replay path's cost by construction.
Measured after:
before after
reconnects with a
detectable gap 82% 0%
silent data loss ~31,000/wk 0
"totals are wrong" reports ~4/wk 0
stream drops from LB idle
timeout ~1,100/wk 0
Redis peak CPU during a
deploy (untested) 41%
p99 replay latency n/a 22ms
Production evidence
The HTML Living Standard specifies server-sent events, including the id field, the Last-Event-ID
request header on reconnection, and the retry field, which is why SSE resumption is a protocol feature
and WebSocket resumption is an application concern.
Redis Streams provide monotonic entry ids, XADD with MAXLEN for a bounded window, and XRANGE
for replay after an id, which is why they are the common implementation of a replay buffer for this
pattern.
Kafka's consumer offsets are the same idea at a different scale: the log is the buffer, the offset is the cursor, and retention is the window, which is the model most fan-out layers borrow.
nginx's X-Accel-Buffering: no and proxy_buffering off exist specifically because response
buffering defeats streaming, and it is documented as the mechanism for per-response opt-out.
Browser connection limits per host under HTTP/1.1 (commonly six) are documented behaviour and are the reason a long-lived SSE stream and multiple tabs interact badly without HTTP/2.
AWS's "Exponential Backoff and Jitter" article is the canonical treatment of why unjittered
reconnection re-synchronises a herd, which applies directly to the retry hint, since the specification
provides a delay and no jitter.
Idempotent state application over deltas is a standard technique in replicated state systems: sending the current value with a version, rather than an increment, makes re-delivery harmless by construction.
The debate
SSE or WebSocket? SSE for one-directional server-to-client streaming, and the resumption support is
a substantial part of the argument: id, Last-Event-ID and retry are protocol features you would
otherwise implement. WebSocket when you need bidirectional low-latency messaging, and then you build
the resume protocol yourself, which is the cost people do not count.
Should the server track what each client has acknowledged? Usually not. Client-supplied cursors scale better and survive server restarts, because the server holds a bounded buffer rather than per-client state, and 100,000 clients each with a server-side cursor is a state-management problem you did not need. Server-side tracking is right where the client cannot be trusted to be honest about its position, which is a different threat model.
Dedupe or idempotent application? Idempotent application where the events are state updates, because it removes a capacity parameter that can be wrong. In the worked example the proposed dedupe set was 25 times too small and would have re-applied events silently. Dedupe remains necessary for genuinely cumulative events, and the better fix there is usually to send the total rather than the delta.
How long should the replay window be? Long enough to cover the disconnect duration you intend to support, which you measure rather than guess, and no longer, because the window is memory. The resolution is a cheap resync path: if a snapshot is 38 KB, a short window plus a fast resync is better than a long window.
Is a silent gap ever acceptable? For a live metric that is refreshed anyway, arguably. For anything the user reads as a total or a list, no, and the distinction is whether the client's state is derived from the stream or merely displayed by it. The minimum acceptable behaviour is a detectable gap: if you cannot replay, say so, so the client can resync.
Should you load-test the replay path? Yes, and almost nobody does, because it only runs during the event that triggers it. In one case 1,400 simultaneous reconnects took Redis from 22 to 96 percent CPU and p99 replay from 4ms to 3.1 seconds, and the discovery was in a load test rather than in production only because someone thought to simulate a deploy.
Follow-up Q&A
"What is the difference between reconnection and resumption?"
Reconnection is re-establishing the transport, which every client library does. Resumption is the server being able to answer "what did I miss after event 4711", which requires a monotonic per-stream id and a bounded replay buffer, neither of which exists by default. Treating them as the same thing produces the silent gap: in one system 82 percent of reconnects lost events and nothing errored, because the client simply continued from the next live event and the totals were quietly wrong.
"What does SSE give you that WebSocket does not?"
Resumption as a protocol feature. The server sets id on each event and optionally retry; the browser
reconnects automatically and sends Last-Event-ID on the reconnect request. With WebSocket you build the
same thing: a subscribe message carrying the client's cursor, a replay from a buffer, and a
resync-required signal when the cursor is too old. That is not difficult, and it is a real cost that
should be counted when choosing the transport.
"What must the server do when it cannot satisfy a resume request?"
Say so explicitly, with a resync-required signal, rather than silently starting from the live position. That single branch is the difference between a detectable gap and a silent one, and it is the branch most commonly omitted. The client then fetches a snapshot, which is why the snapshot path needs to be cheap: if it is 38 kilobytes, a short buffer window plus a fast resync is a better design than a long window.
"Why must the event id be monotonic rather than a UUID?"
Two operations depend on ordering. Resume is a range query, "everything after 4711", which with a UUID becomes a lookup and a scan and cannot distinguish "too old" from "invalid". And gap detection is a client capability: receiving 4711 then 4713 tells the client it lost something without asking anyone. Use a per-stream sequence, a Redis stream id, a Kafka offset, or a UUIDv7 if you need global uniqueness with ordering.
"Dedupe on the client, or idempotent application?"
Idempotent application where the events are state updates: apply by key if the version is newer, and re-delivery is a no-op by construction with no id set and no capacity parameter. Dedupe needs a bounded set, and the bound must exceed the largest possible replay, which is the buffer window times the peak rate. In one review the proposed capacity was 2,000 against a maximum replay of about 50,000, which would have silently re-applied events. For genuinely cumulative events, send the total rather than the delta, which removes the problem.
"What breaks a long-lived stream in production but not locally?"
Three things. A load balancer idle timeout killing a connection with no events, fixed with a heartbeat
comment every 15 to 30 seconds. Proxy buffering, which turns a stream into a single delayed response and
is fixed with X-Accel-Buffering: no, proxy_buffering off and Cache-Control: no-transform. And the
HTTP/1.1 six-connections-per-host browser limit, which an indefinitely-open SSE stream consumes, so the
seventh tab hangs; HTTP/2 removes it. All three work perfectly in local development, which is why they
reach production.
"What happens when everyone reconnects at once?"
The replay path, which is never load-tested because it only runs during the event that triggers it, becomes the bottleneck. In one load test 1,400 simultaneous reconnects with an unjittered retry hint produced 1,400 replay queries in two seconds, took Redis from 22 to 96 percent CPU, pushed p99 replay latency from 4 milliseconds to 3.1 seconds, and caused 340 clients to time out and retry. The fixes: full jitter on the client, an admission limit on concurrent replays, and a cap where a replay larger than the snapshot becomes a resync instead.
Common misconceptions
"The client reconnects, so we are fine." Reconnection without resumption produces a silent gap, and in one system that was 82 percent of reconnects with no error and no metric movement.
"Exactly-once delivery." Not available. At-least-once plus client-side dedupe or idempotent application is the achievable thing.
"Resend the last five minutes on reconnect." That is at-least-once without dedupe, which double-counts and re-fires notifications, and on a flapping connection it repeats.
"A UUID is fine as an event id." It cannot support a range query for resume and it cannot support client-side gap detection.
"A dedupe set is simple." Its capacity must exceed the largest possible replay, which is the buffer window times the peak rate, and getting that wrong re-applies events silently.
"It streams fine in development." Proxy buffering, load balancer idle timeouts and the browser connection limit are all absent locally and all present in production.
Interview delivery note
Say this verbatim: "Reconnecting is not resuming. Resuming needs a monotonic per-stream id and a bounded replay buffer, and above all it needs an explicit resync-required signal when the client's cursor is older than the buffer, because without that branch a gap the client cannot detect is indistinguishable from no gap at all." It names the distinction and the one branch that turns silent data loss into a handled case.
The senior-versus-staff separator is load-testing the replay path. A senior engineer implements resume correctly. A staff engineer notices that the replay path only runs during a mass disconnect, simulates one, finds that 1,400 simultaneous reconnects take the buffer from 22 to 96 percent CPU and p99 replay from 4 milliseconds to 3.1 seconds, and then bounds it: jittered reconnects, an admission limit, and a rule that a replay larger than the snapshot becomes a resync.
The second signal is preferring idempotent application to deduplication. Saying "we replaced the dedupe set with apply-by-key-if-version-is-newer, because the set's capacity has to exceed the largest possible replay and ours was twenty-five times too small" shows you know that a correctness mechanism with a tuning parameter is a correctness mechanism with a bug waiting.
Further reading
- The HTML Living Standard's server-sent events section, for
id,Last-Event-IDandretry. - Redis Streams documentation on
XADDwithMAXLENandXRANGE, for the bounded replay buffer. - nginx documentation on
proxy_bufferingandX-Accel-Buffering, for why streams do not stream behind a proxy. - AWS's "Exponential Backoff and Jitter," for why the unjittered
retryhint produces a reconnect storm. - The SSE vs WebSockets page for the transport choice, and WebSocket scaling for the connection-count side of the same system.