Streaming AI UIs

What it is

A streaming AI UI renders a model's output while it is still being generated, along with the states around it: waiting, thinking, calling tools, citing sources, failing, and being stopped by the user.

The defining constraint is that the response arrives over seconds, in fragments, and may be abandoned. Every UI assumption that holds for a request/response API breaks:

Normal request           Streaming generation
---------------------------------------------------------------
one render               hundreds of incremental renders
complete, parseable      partially valid markdown, unclosed fences
response
succeeds or fails        can fail after producing 80% of an answer
cheap to retry           expensive, and the user has already read
                         part of it
deterministic            two identical requests give different text
final answer             may be revised, cited, or reversed by a
                         later tool call

What this is confused with: a progress bar for a slow request. A progress bar hides latency. A streaming UI uses the latency: the user starts reading at 300ms instead of waiting 20 seconds, which changes perceived quality far more than any reduction in total generation time.

Also confused: streaming as a transport choice. The transport (SSE, chunked fetch, WebSocket) is the easy part. The hard parts are incremental parsing, render batching, interruption, citation resolution and screen-reader behaviour, none of which the transport helps with.

The problem it solves

Time to first token dominates perceived latency, and total time does not.

Non-streaming:  [------------ 18s of nothing ------------] full answer
                User's experience: 18 seconds of a spinner. Many
                abandon. The answer's quality is judged against
                an 18-second wait.

Streaming:      [0.4s] first token, then text flows at ~50 tok/s
                User starts reading at 400ms. Total time is the
                same 18 seconds, and reading occupies most of it.

And the failures that a naive implementation produces are all visible:

1. Per-token re-render of a growing document.
   2,000 tokens, re-parsing and re-rendering the whole markdown tree
   each time, is O(n^2) work. On a mid-tier device this shows up as
   a stuttering stream and an unresponsive page.

2. Markdown parsed mid-stream.
   The model has emitted "```py" and not yet the closing fence, so a
   strict parser renders the rest of the document as code, then
   un-renders it three tokens later. The page flickers between two
   layouts.

3. Scroll jail.
   Auto-scrolling to the bottom on every chunk means a user who
   scrolls up to re-read something is yanked back down 20 times a
   second.

4. A live region announcing every token.
   aria-live="polite" on the streaming text floods a screen reader
   with fragments and makes the page unusable.

5. A stop button that stops the UI, not the request.
   Generation continues server-side, costing tokens, and the partial
   output is discarded so the user has lost what they had read.

All five are correctness bugs that look like polish issues, which is why they survive to production.

Mechanics

Transport, and the event shape that matters

Server-sent events over a normal HTTP response is the default, because it is one direction, it reconnects, and it passes through proxies that WebSockets do not.

The important design point is not SSE versus WebSocket, it is that the stream carries typed events rather than raw text. Anthropic's Messages API streaming, for example, emits discrete event types (message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop) rather than an undifferentiated token feed, which is what lets a UI distinguish text from tool use from a stop reason.

// Server route. The model call streams; the route re-emits typed events
// so the client never has to infer structure from raw text.
import Anthropic from '@anthropic-ai/sdk'

const client = new Anthropic()

export async function POST(req: Request) {
  const { messages } = await req.json()

  const stream = new ReadableStream({
    async start(controller) {
      const enc = new TextEncoder()
      const send = (type: string, data: unknown) =>
        controller.enqueue(enc.encode(`event: ${type}\ndata: ${JSON.stringify(data)}\n\n`))

      try {
        const s = client.messages.stream({
          model: 'claude-opus-4-8',
          max_tokens: 4096,
          messages,
        })

        // Forward the model's own event types rather than flattening
        // them: the client needs to know WHICH block a delta belongs to.
        s.on('contentBlock', (block) => send('block_start', { type: block.type }))
        s.on('text', (delta) => send('text', { delta }))
        s.on('end', () => { send('done', {}); controller.close() })
        s.on('error', (e) => { send('error', { message: String(e) }); controller.close() })

        // Client disconnect MUST abort the upstream call, or you keep
        // paying for tokens nobody will read.
        req.signal.addEventListener('abort', () => s.abort())
      } catch (e) {
        send('error', { message: String(e) })
        controller.close()
      }
    },
  })

  return new Response(stream, {
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache, no-transform',
      'Connection': 'keep-alive',
      // Some reverse proxies buffer responses and destroy streaming.
      'X-Accel-Buffering': 'no',
    },
  })
}

X-Accel-Buffering: no and no-transform are the two headers that make streaming actually stream in production, because an nginx or CDN layer that buffers the response turns your carefully streamed output back into a single 18-second wait, and it will work perfectly in local development.

Batching renders

Do not set React state per token. Accumulate in a ref and flush on a timer.

function useStreamedText(streamUrl: string) {
  const [text, setText] = useState('')
  const buffer = useRef('')
  const frame = useRef<number | null>(null)

  const flush = useCallback(() => {
    frame.current = null
    setText(buffer.current)
  }, [])

  const append = useCallback((delta: string) => {
    buffer.current += delta
    // One render per animation frame at most. For text, even that is
    // more than necessary: a 60ms interval is imperceptible and cuts
    // renders by a further ~4x.
    if (frame.current === null) frame.current = requestAnimationFrame(flush)
  }, [flush])

  return { text, append }
}

The arithmetic:

2,000-token answer at 50 tok/s = 40 seconds of streaming.

Per-token setState:   2,000 renders
Per animation frame:  40s * 60fps = 2,400 potential renders, but
                      capped by token arrival: ~2,000. NO BETTER.
Per 60ms interval:    40s / 0.06 = ~667 renders   (3x fewer)
Per 100ms interval:   400 renders                 (5x fewer)

And each render costs more as the document grows, so total work is
proportional to (renders x average document size). Cutting renders
5x cuts total work 5x.

Requesting an animation frame is not enough when tokens arrive slower than 60fps, which is the common case. Use a time-based interval of 50 to 100ms for text, which is below the threshold at which a reader perceives the stream as chunky.

The bigger win is not re-parsing the whole document. Split the accumulated text into blocks at completed block boundaries, memoise the finished blocks, and only re-render the last one:

const blocks = useMemo(() => splitIntoBlocks(text), [text])
return (
  <>
    {blocks.slice(0, -1).map((b, i) => <FrozenBlock key={i} md={b} />)}
    <LiveBlock md={blocks.at(-1) ?? ''} />
  </>
)
// FrozenBlock is React.memo'd on its string, so completed paragraphs
// and code blocks are parsed exactly once.

Parsing partial markdown

The stream will hand you syntactically invalid markdown constantly. Three approaches, in increasing order of effort:

1. TOLERANT PARSER. Use a parser that closes unterminated constructs
   at the end of input (marked and markdown-it both largely do this).
   Cheap, and it flickers: "```py" renders as a code block whose
   content grows, which is usually what you want anyway.

2. SAFE-BOUNDARY BUFFERING. Only render up to the last position where
   the document is unambiguous: the last blank line outside a fence,
   or the last closed fence. Buffer the tail as plain text.
   No flicker; adds up to a paragraph of latency.

3. STRUCTURED OUTPUT. Have the model emit blocks as discrete events
   rather than one text stream, so the UI never parses partial
   syntax. Strongest option, and it constrains the model.

The default should be 1 plus a narrow special case: track fence state yourself.

// The one construct worth handling explicitly, because it is the one
// that produces a whole-document layout flip.
function withClosedFences(md: string): string {
  const fences = (md.match(/^```/gm) ?? []).length
  return fences % 2 === 1 ? md + '\n```' : md
}

Unclosed code fences cause the worst flicker because they change the rendering of everything after them, so closing them optimistically is worth the twelve lines. Inline constructs (a lone **) cause a one-character flicker nobody notices.

The state machine, and what each state must show

idle -> submitted -> waiting -> [thinking] -> [tool_call] -> streaming
                                     ^              |
                                     +--------------+
     -> done | stopped | error(partial retained)

waiting     within 100-200ms show SOMETHING. Dead air reads as a
            broken button. The submitted message appearing plus a
            typing indicator is enough.

thinking    if the model is reasoning before answering, say so.
            An unexplained 8-second gap is the single largest source
            of "it's broken" reports. "Thinking..." with an elapsed
            timer is honest and sufficient.

tool_call   name the tool in user terms: "Searching your documents",
            not "invoking retrieve_v2". For side-effecting tools,
            this is also where a CONFIRMATION GATE belongs.

streaming   text plus a visible stop control.

stopped     KEEP THE PARTIAL OUTPUT. Mark it as stopped. Offer
            continue and retry.

error       KEEP THE PARTIAL OUTPUT. An error after 80% of an answer
            that discards the answer is the worst possible outcome.

"Keep the partial output" on both stopped and error is the rule that most implementations get wrong, and it is the cheapest quality win available.

Stopping, for real

const controller = useRef<AbortController | null>(null)

async function send(prompt: string) {
  controller.current?.abort()             // supersede any in-flight run
  const ac = new AbortController()
  controller.current = ac
  const res = await fetch('/api/chat', {
    method: 'POST', body: JSON.stringify({ prompt }), signal: ac.signal,
  })
  // ... read res.body
}

function stop() {
  controller.current?.abort()             // client stops reading
  // The server route above listens for req.signal and aborts the
  // upstream model call. WITHOUT THAT, generation continues and you
  // are billed for output nobody sees.
}

A stop button that only stops rendering is a bug with a cost line item. The client abort must propagate to the model call, which is why the server route wires req.signal to stream.abort().

Citations

A citation is only useful if it is verifiable, which means it must resolve to a specific source and ideally a specific span.

Rendering:
  inline superscript markers [1] that scroll to / expand a source
  list, with the retrieved chunk's text available on demand.

The streaming problem: markers arrive in the text before the source
metadata is resolved.
  -> Emit citations as their own typed events, keyed by marker id,
     and render an unresolved marker as a neutral placeholder that
     upgrades in place. Do NOT render a marker as a link until you
     have the target, or the user clicks a dead link.

The trust problem, which is the real one:
  a marker that points at a document which does not support the claim
  is WORSE than no citation, because it converts an unverified claim
  into an apparently verified one. Show the retrieved text, not just
  the document title, so the check is one click rather than a
  download.

Undo, edit and interruption semantics

STOP           halts generation, keeps partial output.
RETRY          regenerates the last assistant turn from the same
               input. The previous attempt should remain reachable,
               because the user may prefer it.
EDIT & RESEND  edits a user turn and regenerates from there. This
               FORKS the conversation: everything after that turn is
               invalidated. Show that explicitly rather than silently
               deleting.
UNDO           for agentic UIs, the hard case: a tool call with a
               side effect cannot be undone by the UI. The control
               that works is a confirmation gate BEFORE the call,
               not an undo after it.

For anything that writes, sends or spends, the UI's job is the gate, not the undo. This is the same argument as the confirmation step on a destructive operation anywhere else, and it is more important here because the user did not author the action.

Accessibility of a streaming region

DO NOT put aria-live="polite" on the streaming text. A screen reader
will announce every flush, producing continuous fragmentary speech.

Do:
  - aria-busy="true" on the message container while streaming
  - a separate polite live region for STATE changes only:
      "Thinking", "Searching documents", "Response complete"
  - on completion, move the announcement to the finished message, or
    let the user navigate to it normally
  - the stop button must be reachable by keyboard without traversing
    the growing text: put it before the message in DOM order, or in
    a fixed toolbar

Announce state transitions, not content. The content is available to read once it is complete, which is how a screen reader user would prefer to consume it anyway.

A worked example: a support assistant that felt slow at 50 tokens per second

An internal support assistant: a chat panel, retrieval over a documentation corpus, and two tools (search docs, look up a ticket). Users reported it was "slow" and "kept jumping around."

Measured, before:

Time to first token (server)          380ms
Generation rate                       ~52 tok/s
Median answer length                  ~600 tokens  -> ~11.5s of stream
Time to first PAINT in the UI         2,600ms      (!!)
Long tasks during streaming (p75)     18 tasks > 50ms per response
Dropped frames while streaming        heavy on mid-tier laptops
Users scrolling up mid-stream         yanked back to bottom
Stop button                           stopped rendering only

The 2,600ms first paint against a 380ms first token was the whole complaint. The model was fast; the UI was not showing anything.

Cause 1: proxy buffering. The response passed through an nginx layer that buffered until a 4KB threshold, so nothing reached the browser until roughly 4KB of SSE frames had accumulated.

Fix: proxy_buffering off for the route, plus X-Accel-Buffering: no
     and Cache-Control: no-transform on the response.
Time to first paint: 2,600ms -> 420ms.

That single header was worth more than every other change combined, and it was invisible in local development, where there is no proxy. "It streams locally and not in production" is almost always buffering.

Cause 2: per-token re-render of the full markdown document.

600 tokens = 600 setState calls = 600 full re-parses of a document
averaging 300 tokens. React re-rendered a growing tree each time.

Fix: 60ms flush interval + block splitting with memoised completed
     blocks.
  renders per response:   600 -> ~190
  markdown parses:        600 full-document -> 190 last-block-only
  long tasks > 50ms (p75): 18 -> 2
  INP during streaming:   340ms -> 90ms

Cause 3: the "jumping around" was two separate bugs.

a) Unclosed code fences. Answers frequently contained shell snippets.
   Mid-stream, "```bash" with no closing fence made the parser treat
   the remainder as code, so the layout flipped between prose and a
   code block repeatedly.
   Fix: optimistically append a closing fence when the count is odd.
   Twelve lines; the flicker disappeared.

b) Scroll jail. The panel called scrollIntoView on every flush.
   Fix: track whether the user is pinned to the bottom, and only
   auto-scroll when they are:

   const pinned = el.scrollHeight - el.scrollTop - el.clientHeight < 40
   if (pinned) el.scrollTop = el.scrollHeight

   Plus a "jump to latest" button when not pinned. Threshold 40px,
   because exactly-zero fails on fractional device pixel ratios.

Cause 4: the stop button. It set a flag that stopped reading the stream.

Measured over one week before the fix:
  stop presses:                        410
  average tokens already generated
    at press time:                     ~180
  average tokens the model went on
    to generate after the press:       ~420

So roughly 172,000 output tokens per week were generated, billed,
and thrown away.

Fix: AbortController on the client, req.signal wired to the SDK
stream's abort() on the server. Post-stop generation: ~0.

A stop button is a cost control as well as a UX control, and framing it that way is what got it prioritised.

Cause 5, found while fixing the others: the thinking gap. Before answering, the assistant often called the doc-search tool, producing a 3 to 7 second gap after the user's message with only a generic typing indicator.

Fix: emit tool events and render them in user terms.
  "Searching documentation..." (with the query shown)
  "Reading 4 documents"
  then the answer streams.

Effect on the support-ticket rate for "the assistant is stuck":
that category dropped to near zero. The system was not faster; it
stopped being silent.

Cause 6: accessibility. The streaming text had aria-live="polite".

A screen reader announced every 60ms flush: continuous fragments,
unusable.
Fix: aria-busy on the container during streaming, a separate polite
region announcing only state transitions ("Searching documentation",
"Response complete"), and the finished message navigable normally.

Result:

                              before      after
time to first paint          2,600ms      420ms
INP during streaming           340ms       90ms
long tasks > 50ms (p75)           18          2
layout flicker              frequent       none
post-stop tokens/week       ~172,000        ~0
"assistant is stuck" tickets    high     near zero

Generation speed was unchanged throughout. Every improvement came from the UI's handling of a stream it was already receiving, which is the general lesson: for an AI feature, the perceived quality is usually set by the client, and the model is not the thing to optimise first.

Production evidence

Anthropic's Messages API streaming emits typed events (message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop) rather than a flat token feed, which is what allows a client to distinguish text, tool use and stop reasons without parsing prose. The official SDKs expose this as both an event emitter and an async iterator.

The Vercel AI SDK implements a documented data-stream protocol over SSE with typed parts for text, tool calls, tool results and errors, plus client hooks that handle abort and partial-state retention. It is the most widely deployed reference implementation of the patterns above.

X-Accel-Buffering: no is nginx's documented mechanism for disabling response buffering per-response, and buffering proxies destroying SSE is a well-known operational failure, which is why the header exists.

ChatGPT, Claude and comparable assistants all render a distinct tool-use or reasoning state rather than dead air, retain partial output when a generation is stopped, and gate certain side-effecting actions behind explicit confirmation. Those are observable product behaviours, and they are the conventions users now expect.

Web Content Accessibility Guidelines 4.1.3 (Status Messages) requires that status changes be programmatically determinable without receiving focus, which is the standard behind announcing state transitions in a live region. The corresponding failure, flooding a live region with continuously changing content, is documented in ARIA authoring guidance as a reason to avoid live regions on rapidly updating text.

The debate

SSE or WebSocket? SSE for one-directional generation, which is the common case: it is plain HTTP, it reconnects natively, it passes proxies, and it needs no separate server. WebSocket when you have genuinely bidirectional, low-latency needs (live voice, collaborative editing, an agent the user steers mid-run). Choosing WebSocket by default adds connection lifecycle management for no benefit.

Should you stream at all? For anything over about a second of generation, yes. For short, structured outputs (a classification, a JSON object the UI will render as a form), no: streaming a partial JSON object gives you a parsing problem and shows the user nothing meaningful. Stream prose; do not stream structure the UI cannot render incrementally.

Should the UI render the model's reasoning? Show that it is reasoning, and show tool calls in user terms. Rendering raw reasoning text is usually a mistake for a product surface: it is long, it is sometimes wrong in ways the final answer is not, and users read it as the answer. The exception is developer-facing and debugging surfaces, where it is the point.

How aggressively should you batch? 50 to 100ms flushes. Below that you pay renders for a difference nobody perceives; above about 150ms the stream starts to look chunky. Per-animation-frame sounds right and does nothing when tokens arrive slower than 60fps, which is the normal case.

Is a confirmation gate on tool calls too much friction? For read-only tools, yes, gate nothing. For anything that writes, sends, spends or deletes, the gate is the control, because the user did not author the action and cannot undo a sent email. The right calibration is per-tool risk, not a global setting, and "remember this choice for this session" resolves most of the friction complaint.

Should partial output be kept on error? Always. The counter-argument, that a partial answer may be misleading, is real and is solved by labelling it ("Response incomplete"), not by deleting text the user has already read.

Follow-up Q&A

"Why does streaming work fine locally and not in production?"

Almost always a buffering proxy. An nginx layer, a CDN, or a serverless platform's response handling accumulates the response body before forwarding, so the browser receives nothing until a threshold is hit. The fixes are proxy_buffering off for the route, X-Accel-Buffering: no on the response, and Cache-Control: no-transform so intermediaries do not re-encode. In one case this alone moved time to first paint from 2,600ms to 420ms against an unchanged 380ms time to first token.

"How do you render markdown that is still arriving?"

Use a tolerant parser that closes unterminated constructs, and handle exactly one construct explicitly: code fences. An odd number of fences means the document is mid-block, and everything after it renders as code, so the layout flips back and forth as the stream continues. Optimistically appending a closing fence when the count is odd is about twelve lines and removes the only flicker users notice. Inline constructs like a lone ** cause a one-character flicker not worth engineering for. The stricter alternative is buffering to the last unambiguous boundary, which costs up to a paragraph of latency.

"How do you keep streaming from making the page unresponsive?"

Two things. Batch state updates on a 50 to 100ms interval rather than per token, since a per-animation-frame request does nothing when tokens arrive slower than 60fps. And split the accumulated text into blocks so completed paragraphs and code blocks are parsed once and memoised, leaving only the last block live. Without that, every token re-parses and re-renders a growing document, which is quadratic. In one case those two changes took renders per response from 600 to 190, long tasks over 50ms from 18 to 2, and INP during streaming from 340ms to 90ms.

"What does a correct stop button do?"

Aborts the client fetch, propagates that abort to the upstream model call on the server, and keeps the partial output on screen labelled as stopped. Stopping only the rendering leaves generation running and billed: in one measurement, 410 stop presses in a week left roughly 172,000 output tokens generated after the press and discarded. Keeping the partial output matters just as much, because discarding text the user has already read is worse than the wait they were trying to end.

"How do you handle citations in a streamed answer?"

Emit citations as their own typed events keyed by marker id, rather than expecting the UI to parse them out of prose. Render an unresolved marker as a neutral placeholder that upgrades in place once the source arrives, so the user never clicks a dead link. And show the retrieved text, not just a document title, because a marker pointing at a document that does not support the claim is worse than no citation: it converts an unverified statement into an apparently verified one.

"What is different about accessibility here?"

A live region on the streaming text announces every flush and produces continuous fragmentary speech, which makes the page unusable. The correct pattern is aria-busy on the message container while streaming, a separate polite live region that announces only state transitions ("Searching documentation", "Response complete"), and letting the user navigate the finished message normally. The stop control also needs to be reachable without tabbing through a growing block of text, so it belongs before the message in DOM order or in a fixed toolbar.

"How do you handle undo in an agentic UI?"

You mostly do not. A tool call with a real side effect, an email sent, a record deleted, a payment made, cannot be undone by the client. The control that works is a confirmation gate before the call, calibrated per tool: no gate for read-only tools, an explicit gate for anything that writes, sends, spends or deletes. This matters more than for ordinary destructive actions because the user did not author the action, the model proposed it. For conversational edits, "edit and resend" forks the conversation and invalidates everything after that turn, which should be shown rather than silently performed.

Common misconceptions

"Streaming is a transport choice." The transport is the easy part. Incremental parsing, render batching, real interruption, citation resolution and screen-reader behaviour are where the work is.

"Flush on every animation frame." Tokens usually arrive slower than 60fps, so that is the same as flushing per token. A 50 to 100ms interval is what actually reduces renders.

"The stop button stops generation." Only if the client abort propagates to the model call. A UI-only stop keeps generating and billing.

"An error means discard the partial answer." Keep it and label it. The user has already read part of it, and deleting it is the worse outcome.

"Put aria-live on the streaming text." That announces every flush as fragments. Announce state transitions instead.

"Slow AI features need a faster model." In one case, generation speed never changed and every user-visible improvement came from the client: a buffering header, render batching, fence handling, scroll pinning, and honest tool-state labels.

Interview delivery note

Say this verbatim: "For an AI feature the perceived speed is usually set by the client, not the model. Time to first token was 380ms and time to first paint was 2,600ms, because a proxy was buffering the response. One header closed most of that gap, and the model was never the problem." It is a concrete, memorable diagnosis and it demonstrates that you have shipped one of these.

The senior-versus-staff separator is treating the stop button as a cost control. A senior engineer implements stop as a UI state. A staff engineer notes that unless the client abort propagates to the upstream model call, generation continues and is billed, quantifies it (410 presses a week, roughly 172,000 output tokens generated after the press and discarded), and uses that number to get the work prioritised. Attaching a spend figure to a UX bug is the move.

The second signal is knowing that dead air is a correctness problem. Saying "an unexplained seven-second gap while a tool runs is the single largest source of it's-broken reports, so we render tool state in user terms rather than a generic spinner, and the ticket category disappeared without the system getting any faster" shows you understand that these UIs are judged on legibility rather than throughput.

The Staff & Team Lead Interview Reference