SSE vs WebSockets for token streaming

What it is

Server-Sent Events is a one-way streaming protocol: the client makes an ordinary HTTP GET, the server responds with Content-Type: text/event-stream and keeps the response body open, writing newline-delimited events as they occur. It is plain HTTP. Reconnection, event IDs and resumption are part of the specification.

WebSocket is a separate protocol that begins as an HTTP request with Upgrade: websocket, receives a 101 Switching Protocols, and then abandons HTTP semantics entirely in favour of a bidirectional binary frame protocol over the same TCP connection.

For streaming LLM tokens from a server to a browser, the answer is SSE, and the reason is not that SSE is better in the abstract. It is that token streaming is unidirectional, and SSE stays inside HTTP while WebSocket leaves it. Staying inside HTTP means your load balancers, authentication, compression, tracing, rate limiting, CDN, WAF and observability all continue to work without special cases.

The confusion to clear: SSE is not "long polling". Long polling closes the response after each message and re-requests, paying a round trip per message. SSE holds one response open and streams many events down it.

The problem it solves

An LLM generates tokens at roughly 20 to 80 per second. Waiting for a complete 600-token answer means a blank screen for 8 to 30 seconds. Streaming turns that into a first token in a few hundred milliseconds and a continuously updating answer, which is the difference between a product that feels broken and one that feels fast. The perceived latency metric is time-to-first-token, and it is only meaningful if there is a transport that can deliver a partial response.

The engineering question is which transport, and the reason it is an interview question is that the naive answer (WebSocket, because real-time) is wrong for a reason that tests whether you understand HTTP infrastructure.

Mechanics

The wire format

GET /v1/chat/stream?id=abc HTTP/1.1
Accept: text/event-stream

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
X-Accel-Buffering: no

event: token
id: 1
data: {"text":"The"}

event: token
id: 2
data: {"text":" capital"}

: heartbeat comment, keeps intermediaries from timing the connection out

event: done
id: 3
data: {"finish_reason":"stop","usage":{"input":412,"output":86}}

Three format rules matter and are easy to get wrong. Every event ends with a blank line; without it nothing is dispatched. Multi-line data: fields are concatenated with newlines, so any payload containing a newline (which markdown does, constantly) must be JSON-encoded or split across data: lines. A line beginning with : is a comment, which is the standard heartbeat: send one every 15 to 30 seconds so proxies with idle timeouts do not drop the connection.

The id: field is what makes resumption work. On reconnect, the browser's EventSource automatically sends Last-Event-ID with the last id it saw, and the server can resume from there. That is built in; with WebSocket you build it yourself.

Server side

# FastAPI. The key details: media_type, disabling proxy buffering, and
# yielding a heartbeat so intermediaries do not close an idle connection.
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import json, asyncio

async def token_stream(prompt: str, resume_from: int = 0):
    seq = 0
    async for chunk in model.stream(prompt):
        seq += 1
        if seq <= resume_from:      # honour Last-Event-ID on reconnect
            continue
        yield f"event: token\nid: {seq}\ndata: {json.dumps({'text': chunk})}\n\n"
    yield f"event: done\nid: {seq + 1}\ndata: {json.dumps({'finish_reason': 'stop'})}\n\n"

@app.get("/v1/chat/stream")
async def stream(prompt: str, request: Request):
    resume = int(request.headers.get("last-event-id", 0))
    return StreamingResponse(
        token_stream(prompt, resume),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache, no-transform",  # no-transform stops
                                                        # proxies rewriting body
            "X-Accel-Buffering": "no",                  # nginx: do not buffer
            "Connection": "keep-alive",
        },
    )

Client side, and the authorization problem

The browser's built-in EventSource cannot set request headers, which means it cannot send Authorization: Bearer .... That is the single most common reason teams abandon SSE, and it has two clean fixes: use a cookie (HttpOnly, Secure, SameSite=Lax) so the browser attaches credentials automatically, or drop EventSource and read the stream with fetch:

// fetch + ReadableStream: full header control, and you keep SSE's wire format.
// The cost: you implement reconnection and Last-Event-ID yourself.
const res = await fetch("/v1/chat/stream?prompt=" + encodeURIComponent(q), {
  headers: { Authorization: `Bearer ${token}`, Accept: "text/event-stream" },
  signal: abortController.signal,      // this is also how you cancel generation
});

const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
let buffer = "";
for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  buffer += value;
  // Events are separated by a blank line. Parse only complete events;
  // a chunk boundary can land in the middle of one.
  let idx;
  while ((idx = buffer.indexOf("\n\n")) !== -1) {
    handleEvent(buffer.slice(0, idx));
    buffer = buffer.slice(idx + 2);
  }
}

Note the AbortController. "How do you let the user stop generation?" is the usual objection to a unidirectional transport, and the answer is that aborting the fetch closes the connection, the server observes the disconnect, and it stops generating. You do not need a bidirectional channel to cancel; you need a way to hang up, and HTTP has one.

The infrastructure gotchas

These are the reasons SSE deployments fail, and they are all configuration:

Proxy buffering. nginx buffers proxied responses by default, so the client receives the whole answer at once and streaming silently does nothing. Fix with proxy_buffering off or the X-Accel-Buffering: no response header. Similar settings exist for every reverse proxy.
Compression. gzip in a proxy will buffer to fill its window. Either disable compression for text/event-stream or ensure the compressor flushes per event.
Idle timeouts. Load balancers close idle connections (60 seconds on an AWS ALB by default). Heartbeat comments more frequently than that.
HTTP/1.1 six-connection-per-origin limit. Browsers allow six connections per origin on HTTP/1.1, and an open SSE stream consumes one. Six tabs and the application deadlocks. HTTP/2 multiplexing removes this entirely, which is the single strongest argument for terminating HTTP/2 at your edge.
Buffering in the model client. If your server-side SDK call is not itself streaming, none of the above matters. Verify the first token leaves your process before you debug the network.

A worked example

A chat product streams answers averaging 500 output tokens at 40 tokens per second, so 12.5 seconds of generation. 10,000 concurrent users.

With SSE: 10,000 open HTTP responses. Each is a socket plus a small per-request buffer; on a Go or Node server, on the order of 10 to 50 KB apiece, so roughly 100 to 500 MB of memory spread across the fleet. They terminate at the ALB, which balances per request because SSE is a normal HTTP request, so scaling out immediately receives traffic. Auth is the same bearer token as every other endpoint. Tracing works because the request has a trace header. A user who refreshes gets automatic reconnection with Last-Event-ID and resumes mid-answer.

With WebSocket: 10,000 upgraded connections. The ALB must be configured for WebSocket, connections are pinned to a backend for their lifetime so a scale-up receives nothing until connections churn (the same problem as L4 balancing of gRPC), and you now need either sticky routing or a Redis or NATS backplane to fan messages to the right node. Auth happens once at the handshake, so a token expiring mid-connection needs its own re-authentication protocol. You write reconnection, sequencing and dedupe yourself. In exchange you gain the ability to send messages up the same connection, which for a chat product means... a POST you could have made anyway.

The comparison is not close for this workload. It becomes close the moment the client needs to send high-frequency messages up: a collaborative editor, a game, a live cursor, an audio stream. Then WebSocket is correct, and using SSE plus a POST per keystroke would be the wrong answer.

Production evidence

The major LLM APIs stream over SSE. OpenAI's and Anthropic's streaming endpoints both return text/event-stream with data: framed events, and OpenAI's uses the data: [DONE] sentinel to terminate. That is the clearest possible evidence for which transport won this particular argument: the companies with the largest token-streaming workloads in existence chose plain HTTP streaming, not WebSocket.

Vercel's AI SDK, LangChain's streaming interfaces and FastAPI's StreamingResponse all target SSE as the default browser transport for this reason, and the X-Accel-Buffering: no header appears in nginx's own documentation as the mechanism for opting a response out of buffering.

WebSocket's production home is the other side of the line: Slack, Discord and multiplayer editors like Figma use persistent bidirectional connections because their traffic genuinely is bidirectional and high frequency. Figma has written publicly about their multiplayer sync running over a persistent connection with a server-authoritative model, which is exactly the workload SSE cannot serve.

The debate

The case for WebSocket in an AI product is real and worth stating: if the product is voice, if the client sends continuous input (audio frames, cursor positions, live document edits), if you want one connection multiplexing many concurrent streams, or if you are already running a WebSocket infrastructure for other features, then adding a second transport is the more expensive choice. Multiplexing is the strongest of these: with SSE, ten simultaneous agent runs means ten connections, and on HTTP/1.1 that is over the browser limit.

The case for SSE is that every piece of your HTTP infrastructure keeps working, reconnection and resumption come free, and the protocol is small enough that nobody has to learn it. That is a large operational advantage for a feature that is, at bottom, "send text down a pipe".

My position: SSE by default for LLM token streaming, over HTTP/2 to remove the connection limit, with fetch rather than EventSource so you keep header-based auth and get AbortController cancellation. Move to WebSocket when the client becomes a real sender, not before. The migration cost from SSE to WebSocket is low and the reverse migration is high, which is another reason to start simple.

SSE is the wrong choice for a voice agent (bidirectional audio), for a mobile-first product where you want one connection carrying everything and control the client anyway, or for server-to-server streaming, where gRPC server streaming is a better fit than either: you get a schema, deadlines and cancellation semantics that SSE lacks.

Follow-up Q&A

"SSE or WebSocket for streaming LLM tokens, and why?" SSE. The traffic is unidirectional, so WebSocket's only real advantage is unused, and SSE stays inside HTTP, which means load balancers, auth, tracing, compression and rate limiting keep working unchanged. It also gives automatic reconnection and resumption via Last-Event-ID, which you would otherwise implement yourself. The cost is the HTTP/1.1 six-connection limit, which HTTP/2 removes, and EventSource's inability to set headers, which fetch plus ReadableStream solves.

"How does the user cancel a generation over a unidirectional transport?" Abort the request. The client calls AbortController.abort(), the connection closes, the server sees the disconnect and stops generating. If you need cancellation to be durable across a network partition, send an explicit POST /cancel with the request ID, which is a normal API call and does not require a bidirectional stream.

"Your SSE endpoint works locally and delivers the whole response at once in production. Diagnose." Something in the path is buffering. Check, in order: nginx or your ingress proxy_buffering, gzip compression on text/event-stream, a CDN in front that is not configured to pass through streaming responses, and your own framework (some serialise the response before writing). Confirm with curl -N against each hop, working inward, which isolates the buffering layer in about two minutes.

"How do you resume a stream after a network drop mid-answer?" Emit a monotonic id: on every event. On reconnect the browser sends Last-Event-ID automatically (or you send it yourself with fetch), and the server replays from the next sequence number. This requires that the server can reproduce or has buffered the tokens it already sent, which for a live generation means keeping the partial completion in a short-lived store keyed by request ID. Without that store, resumption restarts the generation, which the user experiences as the answer changing.

"At what scale do open connections become a problem?" The constraint is sockets and memory, not CPU. Budget roughly 10 to 50 KB per connection depending on runtime and buffer sizes, check ulimit -n and the ephemeral port range on anything acting as a client, and watch the conntrack table on stateful firewalls, which is the limit people forget. A single well-tuned node handles tens of thousands of idle streaming connections; the failure is usually a middlebox's table, not the server.

Common misconceptions

The most common is that WebSocket is required because streaming is "real-time". SSE is real-time; it just is not bidirectional. The relevant question is the direction of the data, not its latency.

The second is that SSE is deprecated or legacy. It is a living part of the HTML specification, and it is what the largest LLM APIs use today.

The third is that EventSource is the only way to consume SSE. It is the convenient way; fetch with a ReadableStream gives you headers, cancellation and full control while keeping the same wire format, and it is what most production frontends actually do.

Interview delivery note

Say this: "SSE, because token streaming is unidirectional and SSE stays inside HTTP, so every load balancer, auth layer, proxy and tracing header keeps working. It also gives me automatic reconnection with Last-Event-ID for free. The two things I have to handle are proxy buffering, which I disable explicitly, and the HTTP/1.1 six-connection-per-origin limit, which HTTP/2 removes. I would use fetch with a ReadableStream rather than EventSource so I keep bearer-token auth and get AbortController cancellation."

The depth signal is naming the infrastructure failure modes: proxy buffering, the connection limit, and idle timeouts needing heartbeats. Anyone can compare the two protocols from a table. Only someone who has shipped it mentions X-Accel-Buffering.

The Staff & Team Lead Interview Reference