Why an L4 load balancer breaks gRPC

What it is

gRPC runs over HTTP/2, which multiplexes many concurrent requests as streams inside a single, long-lived TCP connection. A layer-4 load balancer makes its balancing decision once, when the TCP connection is established, and then blindly forwards bytes for the life of that connection. Put the two together and every RPC a client makes goes to whichever backend won the connection lottery, forever.

The failure is not that gRPC stops working. It is that load stops being balanced: with ten clients and ten backends you get an arbitrary assignment, and with one client (a gateway, a batch job, a sidecar-less service with a shared channel) you get all traffic on one backend while the other nine idle. Newly scaled-up backends receive nothing at all, because no new connections are being made.

This is commonly confused with "gRPC needs sticky sessions" or "gRPC is stateful". gRPC is not stateful at the application level. The stickiness is an artifact of connection reuse, and connection reuse is the entire performance argument for HTTP/2.

The problem it solves, and the problem it creates

HTTP/1.1 needed one connection per in-flight request, so a connection-level load balancer was also, in effect, a request-level load balancer: a client making 100 requests opened and reused connections in a pool, and each new connection got a fresh balancing decision. HTTP/2 fixed head-of-line blocking and connection churn by multiplexing, which is a large win: no repeated TLS handshakes, no slow start, far fewer sockets.

The cost is that the unit the load balancer sees (a connection) and the unit you want balanced (a request) have decoupled. Every mitigation below is a way of re-coupling them.

Mechanics

Consider a Kubernetes ClusterIP service in front of three replicas. kube-proxy in iptables or IPVS mode is a layer-4 balancer: it DNATs the first packet of a new connection to one of the endpoints and installs a conntrack entry, and every subsequent packet of that connection follows the same entry.

client                     kube-proxy (L4)              backends
  |                              |
  |--- TCP SYN to 10.96.0.5 ---->|  pick endpoint: pod-B
  |                              |  conntrack: (client:51234 -> pod-B)
  |<---------- SYN/ACK ----------|
  |=== HTTP/2 connection established, stays open for hours ===
  |
  |--- stream 1: /Search --------|--------------------> pod-B
  |--- stream 3: /Search --------|--------------------> pod-B
  |--- stream 5: /Search --------|--------------------> pod-B
  |    ... 100,000 more RPCs ... |--------------------> pod-B
                                       pod-A: idle
                                       pod-C: idle (scaled up 10 min ago)

There are four ways out, and the right answer depends on where you can put intelligence.

1. Client-side load balancing (the gRPC-native answer)

The gRPC client resolves the service to the full set of backend addresses, opens a subchannel to each, and applies a load balancing policy per RPC.

// Java: resolve via DNS to ALL A records, then round-robin across subchannels.
// In Kubernetes this requires a HEADLESS service (clusterIP: None) so DNS
// returns pod IPs rather than the single virtual IP.
ManagedChannel channel = Grpc.newChannelBuilder(
        "dns:///search-service.default.svc.cluster.local:9090",
        InsecureChannelCredentials.create())
    .defaultLoadBalancingPolicy("round_robin")   // default is pick_first
    .keepAliveTime(30, TimeUnit.SECONDS)
    .build();

// Go: same idea. The dns:/// scheme plus a round_robin service config.
conn, err := grpc.NewClient(
    "dns:///search-service.default.svc.cluster.local:9090",
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithDefaultServiceConfig(`{"loadBalancingConfig":[{"round_robin":{}}]}`),
)

Two details decide whether this works. First, the default policy is pick_first, which connects to the first resolved address and stays there, so you must set round_robin explicitly. Second, the DNS resolver re-resolves on a schedule (30 seconds by default in the Go and Java implementations) and on connection failure, so a scale-up is picked up within that window rather than immediately.

2. Force periodic reconnection (the cheap mitigation)

If you cannot change the clients, change the server. Set a maximum connection age so the server politely closes connections and clients rebalance on reconnect.

// Server side. GOAWAY after ~10 minutes (plus jitter that gRPC adds
// automatically), with a 30s grace period so in-flight RPCs finish.
srv := grpc.NewServer(
    grpc.KeepaliveParams(keepalive.ServerParameters{
        MaxConnectionAge:      10 * time.Minute,
        MaxConnectionAgeGrace: 30 * time.Second,
    }),
)

This turns a permanently skewed assignment into one that reshuffles every ten minutes. It does not balance a single client's RPCs, and it costs a handshake per connection per interval, but it is one config line and it rescues the scaled-up-backend-gets-no-traffic case. I reach for this first when the client is a third party.

3. An L7 proxy that speaks HTTP/2

Put something in the path that terminates HTTP/2 and balances per stream: Envoy, Linkerd's proxy, nginx with grpc_pass, HAProxy in HTTP/2 mode, or an AWS Application Load Balancer with a gRPC-protocol target group. The proxy holds its own connections to the backends and dispatches each stream independently.

An AWS Network Load Balancer does not do this. It is layer 4 by design, so it exhibits exactly the behaviour described above. An Application Load Balancer with ProtocolVersion=GRPC does, and it also handles gRPC status codes in health checks. Choosing NLB "because it is faster" is the single most common way teams walk into this problem on AWS.

4. Lookaside load balancing and xDS

A control plane tells clients where to send traffic. gRPC has first-class xDS support, the same discovery protocol Envoy uses, so a gRPC client can consume endpoint and policy configuration from Istio, Google Cloud Traffic Director or any xDS control plane and do weighted, locality-aware, per-RPC balancing without a proxy in the data path. This is the answer at large scale, and it is also the answer when you want circuit breaking and outlier detection alongside balancing.

A worked example

A recommendation gateway calls a ranking service. The gateway runs 4 pods, the ranking service runs 20 pods behind a ClusterIP, and traffic is 2,000 RPCs per second.

Each gateway pod opens one HTTP/2 connection through kube-proxy and holds it. So at most 4 of the 20 ranking pods receive traffic, and if two gateway pods happen to land on the same ranking pod, only 3 do. Measured effect: those pods run at roughly 500 to 660 RPS each while 16 or 17 pods sit at zero. The HPA, scaling on average CPU across the deployment, sees a low average and scales down, which concentrates load further. The graph looks like a capacity problem and is a routing problem.

The fix, in the order I would apply it: switch the ranking service to a headless service and the gateway's channel to dns:/// with round_robin, which distributes across all 20 immediately; then set MaxConnectionAge on the ranking server as a belt-and-braces measure so future clients that forget the policy still rebalance; then change the HPA to scale on RPS per pod rather than average CPU, because average CPU across an unbalanced fleet is a meaningless number even after the routing is fixed.

Production evidence

The canonical write-up is Linkerd's "gRPC Load Balancing on Kubernetes without Tears" (William Morgan, 2018), which describes precisely this failure in a Kubernetes ClusterIP setup and positions a per-request proxy as the fix. The official gRPC blog post "gRPC Load Balancing" (2017) sets out the same taxonomy used above: proxy versus client-side versus lookaside, and it is where the MaxConnectionAge mitigation is described as the practical answer for unmodifiable clients.

Envoy exists in large part because of this class of problem; Lyft built it to put an L7-aware data plane between services that were otherwise being balanced at layer 4. Google's Traffic Director and the gRPC xDS integration are the productised form of the lookaside pattern, and AWS documents gRPC support as an ALB feature rather than an NLB one, which is the clearest vendor statement that L4 is not sufficient.

The debate

The alternative to fixing the balancing is not to use long-lived connections: open a new connection per request, or per small batch. Some teams do this accidentally by creating a channel per call. It restores L4 balancing and it throws away everything HTTP/2 bought: a TLS handshake per request, connection setup latency in the p99, and socket exhaustion under load. It is the wrong trade at any meaningful RPS.

Between the real options, the choice is about where you can put intelligence. Client-side balancing is the cheapest in the data path (no extra hop, no extra latency, no proxy to operate) and the most expensive organisationally, because every client language and every client team has to configure it correctly, and a single misconfigured client silently reverts to pick_first. A service mesh moves that burden to the platform team and gives you mTLS, retries and outlier detection at the same time, at the cost of a sidecar's memory, CPU and roughly a millisecond per hop.

My position: for a small number of internal services with a homogeneous client stack, use client-side round_robin with a headless service and set MaxConnectionAge on the servers as insurance. Past roughly a dozen services or two languages, adopt a mesh, because the failure mode of client-side balancing is silent and per-client, and silent per-client failures do not scale with headcount.

Client-side balancing is the wrong choice when clients are outside your control, when you need weighted or locality-aware routing that DNS cannot express, or when the backend set is large enough that every client holding a subchannel to every backend is itself a scaling problem. That last case is real: 500 clients times 500 backends is 250,000 connections, and it is why lookaside and xDS exist.

Follow-up Q&A

"Why does this not happen with REST over HTTP/1.1?" Because HTTP/1.1 cannot multiplex. A client that wants 20 concurrent requests must open 20 connections, and each one gets its own L4 balancing decision. The balancing was accidental, a side effect of the protocol's limitation, and HTTP/2 removed the limitation. Note that REST over HTTP/2 has exactly the same problem, so this is a protocol issue, not a gRPC issue. Saying that explicitly is a good signal.

"Does a Kubernetes Service of type LoadBalancer fix it?" No. That provisions a cloud load balancer in front of the nodes, and unless it is an L7 load balancer configured for HTTP/2 or gRPC, it makes the same one-decision-per-connection choice. On AWS the distinction is NLB versus ALB with a gRPC target group. On GCP it is the network load balancer versus the global HTTP(S) load balancer.

"How would you detect this in production before someone reports it?" Plot request rate per backend pod, not aggregate. A healthy fleet has a tight distribution; this failure produces a bimodal one, with a set of pods at zero. The second detector is a scale-up event that does not change latency: if adding pods has no effect, traffic is not reaching them. The third is connection count per pod, which should be roughly clients x subchannels and will instead be one or zero.

"You add round_robin and traffic is still uneven. What now?" Check that DNS actually returns all endpoints: a ClusterIP service returns one virtual IP, so you need clusterIP: None. Check the resolver is re-resolving (the default interval is 30 seconds and can be tuned with GRPC_DNS_RESOLVER settings or a custom resolver). Check for a pick_first fallback caused by a service config that failed to parse. And check whether the imbalance is actually in work rather than requests: round_robin balances RPC count, so a service where 1 percent of queries cost 100 times more will still be unbalanced in CPU. That case needs least_request or a weighted policy fed by backend load reports.

"What breaks if you set MaxConnectionAge too low?" Handshake cost dominates and you get periodic latency bumps. gRPC sends a GOAWAY and lets in-flight RPCs drain during the grace period, so correctness is fine, but at a 30 second age with TLS you are paying a handshake per connection every 30 seconds, and with many clients that is measurable. Ten minutes is a reasonable default; the value should be well above your p99 RPC duration and well below your scaling reaction time.

Common misconceptions

The most persistent one is that this is a gRPC problem. It is an HTTP/2 connection-reuse problem, and any protocol that multiplexes over long-lived connections has it, including REST over HTTP/2, GraphQL over HTTP/2 and database drivers that hold pooled connections through an L4 balancer. Database connection pools behind an NLB exhibit the same skew for the same reason.

The second is that a service mesh is required. It is a good answer, not the only one, and offering MaxConnectionAge plus client-side round_robin as a zero-infrastructure fix is a stronger answer than reaching straight for Istio, because it shows you can solve the problem at the cost the problem deserves.

Interview delivery note

Say this: "gRPC multiplexes RPCs over one long-lived HTTP/2 connection, and an L4 load balancer picks a backend once per connection, so all of a client's RPCs pin to one backend and freshly scaled pods get nothing. The fixes are client-side round-robin over a headless service, an L7 proxy or mesh that balances per stream, xDS lookaside balancing, or as a cheap mitigation, MaxConnectionAge on the server so connections recycle."

The depth signal is naming the HPA feedback loop: unbalanced traffic makes average CPU low, which scales the deployment down, which concentrates load further. Candidates who have only read about this describe the skew; candidates who have lived through it describe the autoscaler making it worse.

The Staff & Team Lead Interview Reference