Design an LLM gateway

"Forty teams are calling model providers directly. Design the gateway that sits in front of them."

Step 1: clarify (4 minutes)

What problem is the gateway solving? Ask, because the answer determines what you build. Teams propose gateways for five different reasons and the designs diverge.

Cost control        Nobody knows who is spending what. -> chargeback
                    and budgets are the primary feature.
Reliability         One provider outage takes down 12 products.
                    -> failover is the primary feature.
Governance          Prompt injection, PII leakage, audit.
                    -> policy enforcement is the primary feature.
Velocity            Every team reimplements retries, streaming
                    and token counting. -> a good SDK might be enough.
Model portability   Switching providers means changing 40 codebases.
                    -> a normalised API is the primary feature.

Assume all five, which is the realistic case at 40 teams, but establish the ordering, because it determines what ships first. Assume cost and reliability are the immediate pain, since those are what get gateways funded.

Is it a proxy or a library? A proxy is a network hop and a single point of failure; a library is per-language work and cannot enforce anything centrally. Assume a proxy, and treat the added latency and the SPOF as things to design against explicitly rather than to hand-wave.

What is the traffic profile? Assume 40 teams, 2,000 requests per second aggregate, heavily skewed (three teams are 70 percent of volume), a mix of streaming and non-streaming, prompts from 500 to 100,000 tokens.

Is it in the user-facing path? Assume yes for some teams, which means added latency has a budget: under 5 ms of gateway overhead at p99, excluding the model call.

Step 2: capacity math (3 minutes)

Traffic
  2,000 RPS aggregate. Small in HTTP terms.
  BUT: each request holds a connection for 2 to 60 seconds
  (streaming, long generations).
  Concurrent connections = 2,000 x ~8 s average = 16,000 concurrent.
  -> The gateway is CONNECTION-bound, not CPU-bound. This changes
     the technology choice: async I/O, not a thread-per-request model.

Payload
  Average request 4,000 tokens in, 500 out = ~18 KB in, ~2 KB out.
  2,000 RPS x 20 KB = 40 MB/sec. Trivial bandwidth.
  Long-context requests at 100k tokens = ~400 KB each. If 1% of
  traffic, that is 20 RPS x 400 KB = 8 MB/sec extra. Fine, but the
  MEMORY matters: buffering 16,000 concurrent requests at even
  50 KB average is 800 MB of buffers. Stream, do not buffer.

Token accounting
  2,000 RPS x 4,500 tokens = 9M tokens/sec of accounting.
  Counting tokens by re-tokenising every request is expensive
  (~1 ms per 4k tokens). At 2,000 RPS that is 2 cores of pure
  tokenisation. Use provider-reported usage where available and
  estimate only for pre-flight budget checks.

Cache
  Semantic/exact cache over responses. If 15% of requests are
  repeats, that is 300 RPS served from cache at ~2 ms instead of
  ~2,000 ms, and at typical prices it is the single largest cost
  saving available.

Fleet
  Connection-bound at 16,000 concurrent: ~8 instances at 2,000
  concurrent connections each, in an async runtime. Small.

The insight that shapes everything: this service is connection-bound, not CPU-bound. Sixteen thousand concurrent long-lived streaming connections at 2,000 RPS is a completely different engineering problem from 2,000 RPS of 50 ms requests, and it dictates async I/O, streaming pass-through rather than buffering, and careful timeout handling.

Step 3: architecture

   40 client applications
        │  OpenAI-compatible or native SDK
        ▼
   ┌──────────────────────────────────────────────┐
   │  GATEWAY (stateless, async, ~8 replicas)     │
   │                                              │
   │  1. authn/authz    team identity, API key    │
   │  2. budget check   is this team over cap?    │
   │  3. policy         PII scan, injection scan  │
   │  4. cache lookup   exact -> semantic         │
   │  5. route          model tier, provider      │
   │  6. rate limit     per team, per provider    │
   │  7. call provider  stream through            │
   │  8. record         tokens, cost, latency     │
   └──────┬───────────────────────────┬───────────┘
          │                           │
    ┌─────▼──────┐            ┌───────▼────────┐
    │ providers   │            │  usage stream   │  Kafka
    │ Anthropic   │            └───────┬────────┘
    │ Bedrock     │                    ▼
    │ Vertex      │            ┌────────────────┐
    │ self-hosted │            │ cost attribution│
    └─────────────┘            │ dashboards      │
                               │ budget state    │
                               └────────────────┘

The gateway is stateless. Budget state, rate-limit counters and cache live in Redis; usage records go to Kafka. That is what allows it to scale horizontally and to be restarted freely, and it is worth saying, because a stateful gateway becomes the thing you cannot deploy during business hours.

Step 4: routing and failover

The two features that justify the project. They are related and they are not the same thing.

ROUTES = {
    # Route by declared task class, not by model name. Clients say
    # what they need; the gateway decides what serves it. This is the
    # decoupling that makes model migration a config change.
    "classification":  ["haiku",  "sonnet"],
    "generation":      ["sonnet", "opus"],
    "reasoning":       ["opus"],
    "embedding":       ["embed-v3"],
}

async def route(req: Request) -> Response:
    chain = ROUTES[req.task_class]
    if req.model_override and req.team.may_override:
        chain = [req.model_override] + chain

    last_error = None
    for model in chain:
        for provider in providers_for(model):      # e.g. direct, Bedrock
            if breaker.is_open(provider, model):
                continue
            try:
                return await call(provider, model, req,
                                  timeout=req.deadline_remaining())
            except (RateLimited, ProviderError, Timeout) as e:
                breaker.record_failure(provider, model)
                last_error = e
                continue                            # next provider
    raise AllProvidersFailed(last_error)

Three distinct failover dimensions, and conflating them is the common error:

Same model, different provider   Anthropic direct -> Bedrock -> Vertex.
                                 IDENTICAL output quality. Always safe.
                                 This is the failover you want.

Same provider, smaller model     Opus -> Sonnet. Cheaper, faster,
                                 DIFFERENT quality. A product decision,
                                 not an availability one.

Degraded response                Cached similar answer, or a canned
                                 fallback. Explicit product behaviour.

Only the first is unambiguously safe, and a gateway that silently downgrades from Opus to Haiku during an incident produces worse output that nobody attributes to the gateway. The rule: cross-provider failover is automatic; cross-model downgrade requires the team to opt in per route. That distinction is the thing worth saying, because it is where gateways cause harm.

Circuit breakers per (provider, model), not per provider: one model can be rate limited while others on the same provider are healthy.

Deadline propagation rather than a fixed per-attempt timeout. If the client has a 10-second budget and the first attempt consumed 7, the second attempt gets 3, not another 10. Without this, a three-provider failover chain turns a 10-second budget into 30 seconds and the client has given up long before.

Step 5: caching, which is where the money is

Tier 1  EXACT match          hash(model, messages, params) -> response
        Hit rate: 5-20% depending on workload. Zero risk.
        This is a plain key-value lookup, ~1 ms.

Tier 2  PROVIDER PROMPT CACHE  Not the gateway's cache; the provider's.
        The gateway's job is to STRUCTURE prompts so it is usable:
        stable system prompt and few-shot examples FIRST, variable
        content LAST. Getting the ordering wrong makes the provider
        cache useless, and most teams get it wrong.

Tier 3  SEMANTIC match       embed the query, ANN search, return if
        similarity > threshold. Hit rate: another 10-30%.
        RISK: "what is our refund policy for EU customers" and
        "...for US customers" are semantically close and have
        different correct answers. Never enable by default.

The honest ordering: exact caching first, provider prompt caching second, semantic caching last and opt-in per route. Semantic caching is the one that gets demoed and the one that causes incidents, because a near-miss returns a confidently wrong answer with no signal that anything happened.

# Structuring for the provider's prompt cache. The gateway can
# enforce this, which is a real service to 40 teams who each would
# otherwise discover it independently.
messages = [
    {"role": "system", "content": STABLE_SYSTEM_PROMPT,      # cached
     "cache_control": {"type": "ephemeral"}},
    {"role": "user",   "content": STABLE_FEW_SHOT_EXAMPLES,  # cached
     "cache_control": {"type": "ephemeral"}},
    {"role": "user",   "content": variable_user_input},      # not cached
]

At typical prompt-cache economics (cache reads costing a fraction of fresh input tokens), a workload with a large stable prefix and a small variable suffix sees the majority of its input cost disappear. For a RAG application with a 20,000-token system prompt and a 200-token question, this is the largest single saving in the system, and it costs nothing but message ordering.

Step 6: cost attribution and budgets

The feature that funds the gateway.

@dataclass(frozen=True)
class UsageRecord:
    request_id: str
    team_id: str
    # Nested attribution, because "team X spent $40k" is not
    # actionable and "feature Y in team X spent $38k of it" is.
    project: str
    feature: str
    environment: str          # prod / staging / dev
    model: str
    provider: str
    input_tokens: int
    cached_input_tokens: int  # priced differently, must be separate
    output_tokens: int
    cost_usd: Decimal         # computed at request time, from a
                              # versioned price table, so historical
                              # records stay correct after price changes
    latency_ms: int
    cache_tier: str | None    # exact / semantic / provider / none
    task_class: str

Two decisions worth defending:

Compute cost at request time from a versioned price table, not at query time from current prices. If a provider changes prices, historical records must keep their original cost, or every past report changes retroactively and reconciliation with the provider's invoice becomes impossible.

Separate cached from uncached input tokens. They are priced very differently, and merging them makes the single biggest optimisation in the system invisible in reporting.

Budget enforcement

async def check_budget(team: Team, estimated_cost: Decimal) -> None:
    spent = await redis.get(f"spend:{team.id}:{current_month()}")
    ratio = (Decimal(spent or 0) + estimated_cost) / team.monthly_budget

    if ratio > 1.0 and team.hard_cap:
        raise BudgetExceeded(team.id)        # 429, with a clear message
    if ratio > 0.9:
        alert_once(team, "90% of monthly budget")
    if ratio > 0.8:
        # Soft degradation: route to a cheaper tier where the team
        # has allowed it. Better than a hard stop at month end.
        team.prefer_cheaper_tier = True

A hard cap in production is dangerous and worth saying so: a team hitting its cap on the 28th takes down a customer-facing feature. The defensible policy is hard caps in dev and staging, soft degradation plus loud alerting in production, with an explicit override path. Getting this wrong turns a cost-control feature into an availability incident.

Step 7: policy enforcement

INBOUND
  PII detection      Scan prompts for card numbers, national IDs,
                     health data. Redact, block, or log by policy.
                     This is why a proxy beats a library: it cannot
                     be forgotten.
  Injection scan     Detect instruction-override patterns in
                     retrieved content. Detection is imperfect and
                     should be treated as defence in depth, not as
                     the control. See: prompt injection.
  Size limits        Reject a 900k-token prompt before paying for it.

OUTBOUND
  PII leakage        Scan responses for training-data leakage or
                     for PII echoed back.
  Schema validation  If the route declares a JSON schema, validate
                     and retry once on a parse failure rather than
                     returning malformed output to the client.

AUDIT
  Every request and response, hashed and stored, with retention
  matched to the compliance requirement. This is often the actual
  reason the gateway gets approved.

The strongest argument for a proxy over a library is right here: a policy in a library is enforced only by teams who upgraded the library. A policy in a proxy is enforced by construction, and "we cannot be sure all 40 teams are scanning for PII" is the sentence that gets the project funded.

Step 8: failure modes and degradation

The gateway itself goes down
  -> This is the SPOF the design created, so treat it seriously.
     Stateless and horizontally scaled; multi-AZ; and a documented
     break-glass path where teams can call providers directly with
     an emergency credential. That escape hatch is worth having and
     worth auditing, because without it a gateway outage is a
     company-wide outage.

One provider degrades (slow, not failing)
  -> The dangerous case, since a circuit breaker on errors does not
     trip. Break on LATENCY percentile, not just error rate.

Rate limited by a provider
  -> Failover to the same model on another provider first, which
     preserves quality. Queue with backpressure if no alternative.
     Return 429 with Retry-After rather than holding the connection.

Redis (budget/cache/rate-limit state) down
  -> Fail OPEN for budget checks and rate limits, because failing
     closed turns a cache outage into a total outage. Log the gap
     and reconcile spend afterwards from the usage stream, which is
     in Kafka and is the real record anyway.

Cache poisoning
  -> A cached response for a prompt that has since become wrong.
     Version cache keys with the prompt template version and the
     model version, so a template change invalidates by construction.

Streaming client disconnects mid-response
  -> Cancel the upstream call. Otherwise you pay for tokens nobody
     receives, and at scale that is a measurable share of the bill.

The degradation ladder: disable semantic caching first (correctness risk under load is not worth it), then downgrade opted-in routes to cheaper models, then shed non-production traffic, then queue with backpressure. Production traffic from teams within budget is the last thing to touch.

Step 9: what changes at ten times the scale

At 20,000 RPS and 160,000 concurrent connections:

Connection handling dominates. 160,000 concurrent streaming connections needs careful event-loop tuning and probably a dedicated L7 proxy layer (Envoy) in front of the application logic, with the gateway becoming a filter chain rather than an application.

The usage stream becomes a real pipeline. 20,000 usage records per second is 1.7 billion a day, so it stops being "write to Kafka and aggregate" and becomes a proper streaming aggregation with rollups, which is the ad click aggregation problem with different nouns.

Provider rate limits become the binding constraint, not the gateway. The design shifts toward capacity management: reserved throughput commitments with providers, a fair-share scheduler across teams so one team's batch job cannot consume the shared quota, and priority classes so interactive traffic preempts batch.

Self-hosted models enter the routing table for the highest-volume, lowest-complexity task classes, because at that volume the crossover where running your own inference is cheaper has been passed for classification-shaped work.

Production evidence

Anthropic's prompt caching documentation specifies the cache-read pricing and the requirement that cached content be a stable prefix, which is what makes the message ordering in step 5 a first-order cost decision rather than a micro-optimisation.

Amazon Bedrock, Google Vertex AI and Microsoft Foundry all offer Claude models, which is what makes same-model cross-provider failover possible at identical quality. That is the practical basis for the routing design, and it is worth naming because it is what distinguishes safe failover from a silent quality downgrade.

LiteLLM, Portkey, Kong AI Gateway and Cloudflare AI Gateway are the existing implementations of this pattern, and their common feature set (normalised API, routing, fallbacks, caching, per-key budgets, observability) is convergent evidence for the component list here.

OpenTelemetry's GenAI semantic conventions define standard span attributes for model calls, which is the right basis for the observability layer rather than inventing a schema.

Simon Willison's "lethal trifecta" framing (private data, untrusted content, and external communication) is the reference for why inbound scanning is defence in depth rather than a control, and why the gateway should not be presented as solving prompt injection.

The debate

The case for a gateway: central policy enforcement that cannot be forgotten, real cost attribution, cross-provider failover, and one place to change when a model is deprecated. At 40 teams, the alternative is 40 implementations of retries, token counting and budget tracking, most of them wrong.

The case against: you have created a single point of failure in front of every AI-powered product, added a network hop to a latency-sensitive path, and built a platform team's worth of ongoing work. Teams will route around it the moment it is slower or less capable than calling the provider directly, and then you have a gateway that enforces nothing and a false belief that it does.

The case for a shared library instead: no SPOF, no added latency, no platform team. It fails on enforcement (a policy in a library is enforced by whoever upgraded) and on polyglot estates.

My position: build the gateway, but ship it in the order the pain is felt, and design against the SPOF from day one. Concretely: usage recording and cost attribution first, because that is what is actually funded and it is read-only, so it can ship behind a sidecar or a wrapper before anything is in the request path. Then routing and cross-provider failover, which is the highest-value reliability feature and is safe because the model is identical. Then caching. Then policy enforcement, which is the hardest to get right and the most damaging to get wrong.

The design decision I would defend hardest is automatic failover across providers for the same model, and opt-in only for cross-model downgrade. A gateway that silently serves Haiku when Opus is unavailable produces worse output during an incident, nobody attributes it to the gateway, and the team's evaluation results become non-reproducible. Same model on Bedrock instead of direct is genuinely identical and can be automatic; anything that changes the model is a product decision.

The second is failing open when Redis is down. Budget checks and rate limits are important and they are not worth converting a cache outage into a company-wide AI outage. The usage stream in Kafka is the real record, so spend can be reconciled afterwards, and the gap is a logged known-unknown rather than a lost one.

And the thing I would insist on regardless: a documented break-glass path for teams to bypass the gateway during a gateway outage. It feels like admitting defeat and it is the difference between a gateway incident and a company incident. It should require an emergency credential, be loudly audited, and exist.

Follow-up Q&A

"What's the first thing you'd ship?" Usage recording and cost attribution, because that is what actually gets these funded and it is read-only, so it can go out as a wrapper or sidecar before anything sits in the request path. It also builds the case for the rest with data: once you can say which three teams are seventy percent of spend and which feature inside one of them is most of that, the routing and caching work prioritises itself. Putting a proxy in the critical path before you have earned trust is how gateways get routed around.

"How do you do failover without degrading quality?" Separate the dimensions. Same model on a different provider, Anthropic direct to Bedrock to Vertex, is identical output and can be fully automatic. A smaller model on the same provider is cheaper, faster and different, so it is a product decision and must be opt-in per route. A gateway that silently serves Haiku when Opus is unavailable produces worse output during an incident that nobody attributes to the gateway, and it makes the team's evaluation results non-reproducible.

"Where does the money actually go?" Almost always into prompt structure rather than model choice. The gateway can enforce putting the stable system prompt and few-shot examples first and the variable content last, which makes the provider's prompt cache usable. For a RAG workload with a 20,000-token stable prefix and a 200-token question, that removes most of the input cost, and it costs nothing but message ordering. Forty teams would otherwise each discover this independently, and most would not.

"What about semantic caching?" Last, and opt-in per route. It is the feature that demos well and causes incidents, because "what is our refund policy for EU customers" and the same question for US customers are semantically close and have different correct answers. A near-miss returns a confidently wrong answer with no signal that anything happened. Exact caching first, provider prompt caching second, semantic third with an explicit similarity threshold the team chose.

"You've created a single point of failure. What do you do about it?" Take it seriously rather than arguing it away. Stateless and horizontally scaled so it restarts freely, multi-AZ, and a documented break-glass path with an emergency credential so teams can call providers directly during a gateway outage. That escape hatch feels like admitting defeat and it is the difference between a gateway incident and a company-wide one. It should be loudly audited so it is not used casually.

"Redis goes down. Budgets and rate limits are in Redis." Fail open. Failing closed converts a cache outage into a total AI outage across forty teams, which is a much worse outcome than a few hours of unenforced budgets. The usage stream is in Kafka and is the real record, so spend gets reconciled afterwards and the gap is a logged known-unknown. The general rule is that a control-plane dependency should not be able to take down the data plane.

"Should budgets be hard caps?" In dev and staging, yes. In production, no, because a team hitting its cap on the 28th takes down a customer-facing feature and that is a cost-control feature causing an availability incident. The defensible policy is soft degradation at 80 percent, routing to a cheaper tier where the team has allowed it, loud alerting at 90, and an explicit override path. A hard production cap should require someone to have chosen it deliberately.

"What's the argument for a proxy over a shared library?" Enforcement. A policy in a library is enforced only by teams that upgraded, and in a polyglot estate you are maintaining it several times. "We cannot be sure all forty teams are scanning prompts for PII" is usually the sentence that funds the project. The cost is that you own a network hop and a single point of failure, which is why the latency budget (under five milliseconds of gateway overhead at p99) and the break-glass path are design requirements rather than nice-to-haves.

"Why is this connection-bound rather than CPU-bound?" Because each request holds a connection for seconds, not milliseconds. Two thousand requests per second at an eight second average is sixteen thousand concurrent connections, mostly idle, waiting on a provider. That dictates async I/O rather than thread-per-request, streaming pass-through rather than buffering (buffering sixteen thousand requests is hundreds of megabytes of buffers), and cancelling the upstream call when a streaming client disconnects, because otherwise you pay for tokens nobody receives.

Common misconceptions

"The gateway solves prompt injection." Detection is imperfect and belongs in defence in depth. Presenting the gateway as the control gives teams false confidence.

"Failover is failover." Cross-provider same-model is safe. Cross-model is a product decision. Conflating them degrades quality silently during incidents.

"Semantic caching is a cost feature." It is a correctness risk with a cost benefit. Opt-in, per route, with a threshold someone chose.

"Cost is about picking cheaper models." It is mostly about prompt structure and cache hit rates. Model choice is the last lever, not the first.

"A stateless service can't have a SPOF problem." Stateless helps you scale and restart; it does not help when every AI product in the company routes through one service. The break-glass path is the actual mitigation.

Interview delivery note

Ask what the gateway is for before designing it, because the ordering is the answer: "Teams propose gateways for five different reasons: cost attribution, reliability, governance, velocity, and model portability. They produce different first releases. I'd ship usage recording and cost attribution first, because that's what actually gets funded and it's read-only, so it can go out as a wrapper before anything sits in the request path."

Then the capacity insight, because it changes the technology: "The thing that shapes this is that it's connection-bound, not CPU-bound. Two thousand requests a second at eight seconds each is sixteen thousand concurrent connections, mostly idle waiting on a provider. So async I/O, streaming pass-through rather than buffering, and cancelling upstream when a streaming client disconnects, because otherwise you pay for tokens nobody receives."

Make the failover distinction explicitly, because it is where gateways cause harm: "I'd separate two things people conflate. Same model on a different provider, direct to Bedrock to Vertex, is identical output and can be fully automatic. A smaller model is cheaper, faster and different, so it's opt-in per route. A gateway that silently serves Haiku when Opus is down produces worse output during an incident that nobody attributes to the gateway."

The line that shows where the money actually is: "and most of the cost saving isn't model choice, it's prompt structure. Putting the stable system prompt and examples first and the variable content last makes the provider's prompt cache usable, and for a RAG workload with a twenty-thousand-token prefix that removes most of the input cost. Forty teams would each discover that independently, and most wouldn't."

Close by owning the downside: "and I'd say plainly that I've just built a single point of failure in front of every AI product in the company. So: stateless, multi-AZ, a p99 overhead budget of five milliseconds, and a documented break-glass credential so teams can bypass it during a gateway outage. That escape hatch feels like admitting defeat and it's the difference between a gateway incident and a company incident."

The Staff & Team Lead Interview Reference