WebSocket scaling: sticky routing, backplanes, and connection math

What it is

A WebSocket connection is stateful and long-lived, which breaks the assumption every stateless-HTTP scaling technique rests on. Three consequences follow, and they are the whole topic:

A connection is pinned to one process. Once established, every message for that client must reach that server, so any other server holding a message for that client has a routing problem.

Capacity is measured in concurrent connections, not requests per second. A server handling 50,000 requests per second might hold 200,000 idle WebSocket connections, and the constraints are file descriptors, memory per connection and ephemeral ports rather than CPU.

A deploy disconnects everyone. Rolling a stateless service moves traffic; rolling a WebSocket service severs every connection on the replaced instance, and they all reconnect at once.

Two architectures for the routing problem:

	Sticky routing	Pub/sub backplane
How	The load balancer pins a client to a server	Any server can publish; all servers subscribe
Fan-out	Requires knowing which server holds the client	Broadcast, every server filters
Scaling	Connections scale, cross-server messaging does not	Both scale, at the cost of a message bus
Failure	The server dies, the client reconnects elsewhere	Same, plus the bus is a dependency
Right for	Client-to-server request/response over a socket	Server-initiated fan-out, chat, presence

What this is confused with: needing one or the other. Almost every real system needs both: sticky routing so a client's messages reach a consistent process, and a backplane so a message generated anywhere reaches the process holding the recipient.

The problem it solves

HTTP load balancing assumes any server can handle any request. WebSockets break that, and the naive deployment fails in a specific order:

1. Two servers, round-robin. Client A on server 1, client B on server 2.
   A sends a message for B. Server 1 has no connection to B. The message
   is silently dropped.

2. Add a backplane. Now it works, and every server receives every message
   and discards 99.9% of them.

3. Scale to 40 servers and 400,000 connections. The backplane is now
   delivering every message to 40 servers. Bus throughput is the limit,
   not connections.

4. Deploy. All 400,000 clients reconnect within seconds. The reconnect
   storm exceeds the accept rate, clients retry, and the storm feeds itself.

Step 4 is the one that takes systems down, and it is the failure that has nothing to do with steady-state capacity.

The resource arithmetic that decides the design:

Per idle WebSocket connection, typical Node.js/Go server:
  kernel socket buffers:     ~4-16 KB (tunable, and the default is generous)
  application state:         ~1-10 KB (session, subscriptions, buffers)
  file descriptor:           1
  conntrack entry (if NAT):  1

100,000 connections:
  memory:                    ~0.5-2.5 GB
  file descriptors:          100,000  (default ulimit is 1024)
  ephemeral ports at the LB: 100,000 toward one backend IP:port

The ephemeral port limit is the one people meet first and understand last. A load balancer opening connections to a backend has about 28,000 ephemeral ports per destination ip:port tuple, so 28,000 connections per backend per LB instance unless you widen the range or add backend addresses.

Mechanics

Sticky routing

upstream ws_backend {
    ip_hash;                        # or `hash $cookie_sid consistent;`
    server ws1.internal:8080;
    server ws2.internal:8080;
    server ws3.internal:8080;
}

ip_hash is the weakest form and it is what most examples show. It breaks with carrier-grade NAT (thousands of mobile users behind one IP land on one server), it rebalances everything when a server is added or removed, and it gives no control.

Consistent hashing on a client-supplied key is the right version:

upstream ws_backend {
    hash $arg_client_id consistent;    # `consistent` = ketama: only 1/N moves
    server ws1.internal:8080;
    server ws2.internal:8080;
}

Adding a 4th server to 3:
  plain hash:       ~75% of clients move to a different server
  consistent hash:  ~25% move

On a WebSocket service every move is a disconnect, so the difference between 75 percent and 25 percent of clients reconnecting is the difference between an incident and a blip.

Layer 7 versus layer 4 matters here, and it connects to the gRPC load balancing page: an L4 balancer pins the TCP connection, which is what you want, and it cannot read a cookie or a path, so the key must be derivable from the connection. An L7 balancer can route on anything and must handle the upgrade correctly.

The backplane

                     ┌──────────┐
   client A ────────▶│ server 1 │──publish──┐
                     └──────────┘           │
                                       ┌────▼─────┐
                     ┌──────────┐      │  Redis   │
   client B ◀────────│ server 2 │◀─sub─│  Pub/Sub │
                     └──────────┘      │  / NATS  │
                                       │  / Kafka │
                     ┌──────────┐      └──────────┘
   client C ◀────────│ server 3 │◀─sub──────┘
                     └──────────┘

// The naive version: every server subscribes to everything.
redis.subscribe('messages');
redis.on('message', (channel, payload) => {
  const msg = JSON.parse(payload);
  const socket = localConnections.get(msg.recipientId);
  if (socket) socket.send(payload);       // 99.9% of the time: not here, discard
});

Every server receives every message. At 40 servers that is 40x the message volume on the bus, and the bus becomes the scaling limit long before connections do.

Channel-per-topic reduces the fan-out to what is needed:

// Subscribe only to the channels this server actually holds subscribers for.
function onClientSubscribe(clientId, topic) {
  localTopics.get(topic).add(clientId);
  if (localTopics.get(topic).size === 1) {
    redis.subscribe(`topic:${topic}`);       // first local subscriber: subscribe
  }
}
function onClientLeave(clientId, topic) {
  localTopics.get(topic).delete(clientId);
  if (localTopics.get(topic).size === 0) {
    redis.unsubscribe(`topic:${topic}`);     // last one left: unsubscribe
  }
}

Broadcast to 1 recipient, 40 servers:
  subscribe-to-everything:   40 deliveries, 39 discarded
  channel-per-topic:          1 delivery

The cost is subscription churn, which for a chat application with users joining and leaving rooms constantly is its own load, and Redis Pub/Sub handles subscription changes less efficiently than message delivery.

Choosing the bus:

Redis Pub/Sub:   fire-and-forget, no persistence, no delivery guarantee.
                 A subscriber that is down misses messages permanently.
                 Simplest, and correct for presence and ephemeral state.
Redis Streams:   persistent, consumer groups, replay by ID.
                 Right when a missed message matters.
NATS:            purpose-built, very low latency, JetStream for persistence.
Kafka:           persistent, ordered per partition, replayable.
                 Heavy for this, and right when the messages are also
                 a business event stream.

Redis Pub/Sub losing messages during a subscriber restart is the property that surprises people, because it is invisible in testing and appears as "occasionally a message is not delivered."

Connection resource math

# 1. File descriptors. The default of 1024 is the first wall.
$ ulimit -n
1024
# systemd unit:
LimitNOFILE=1048576
# and the system-wide ceiling:
fs.file-max = 2097152

# 2. Ephemeral ports, at the LOAD BALANCER, toward each backend.
net.ipv4.ip_local_port_range = 1024 65535       # ~64,500 per backend ip:port
# Each additional backend PORT multiplies this:
#   backend listening on 8080-8083 = 4x the port budget

# 3. conntrack, if there is NAT anywhere in the path.
net.netfilter.nf_conntrack_max = 2097152
net.netfilter.nf_conntrack_tcp_timeout_established = 86400   # long-lived!

Long-lived connections and conntrack interact badly: the established timeout must exceed your connection lifetime or entries are evicted while the connection is alive, which drops traffic silently. See TCP tuning.

# 4. Socket memory. The defaults are generous for many small connections.
net.ipv4.tcp_rmem = 4096 87380 6291456
net.ipv4.tcp_wmem = 4096 16384 4194304
# For many idle connections, lower the DEFAULT (middle value):
net.ipv4.tcp_rmem = 4096 16384 6291456
net.ipv4.tcp_wmem = 4096 16384 4194304

100,000 connections:
  default rmem+wmem:   100,000 x ~104 KB = ~10 GB
  tuned:               100,000 x ~32 KB  = ~3.2 GB

Socket buffer defaults are sized for throughput per connection, not for connection count, and that is a 7 GB difference on one machine.

# 5. Keepalives, so dead connections are reaped rather than accumulating.
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 6

Plus application-level pings, because a TCP connection through a NAT or a load balancer can be silently dropped by an intermediary while both endpoints believe it is alive. WebSocket ping/pong frames at 30 seconds is the standard, and it is also what keeps intermediaries from timing the connection out.

The reconnect storm

This is the failure that takes down WebSocket systems, and it has nothing to do with steady-state capacity.

40 servers, 400,000 connections. One server is replaced.
  -> 10,000 clients disconnect simultaneously
  -> all reconnect within ~1 second (naive clients)
  -> 10,000 TCP handshakes + 10,000 TLS handshakes + 10,000 auth calls
     in one second, spread over 39 servers
  -> auth service saturates, some reconnects fail
  -> those clients retry immediately
  -> the storm sustains itself

Exponential backoff with full jitter is the client-side requirement:

let attempt = 0;
function reconnect() {
  const base = Math.min(30000, 1000 * Math.pow(2, attempt));
  const delay = Math.random() * base;         // FULL jitter, not base + jitter
  attempt++;
  setTimeout(connect, delay);
}
socket.onopen = () => { attempt = 0; };       // reset ONLY after a successful open

Full jitter (random(0, base)) rather than base + random() is what actually spreads the herd: the second form still has every client waiting at least base, so the storm is delayed rather than dispersed.

Server-side, the deploy should drain rather than sever:

process.on('SIGTERM', async () => {
  server.close();                                  // stop accepting new connections
  // Tell clients to reconnect, spread over a window.
  const clients = [...connections];
  const windowMs = 60_000;
  clients.forEach((c, i) => {
    setTimeout(() => c.close(1001, 'server_going_away'), (i / clients.length) * windowMs);
  });
  await sleep(windowMs + 5000);
  process.exit(0);
});

Closing with code 1001 (going away) over a 60-second window converts a 10,000-client instantaneous storm into 167 reconnects per second, which is a normal rate. terminationGracePeriodSeconds must exceed the window or Kubernetes kills the process mid-drain.

A worked example: a deploy that took down a platform

A collaborative document editor. About 340,000 concurrent WebSocket connections at peak, 60 Node.js servers, Redis Pub/Sub backplane, behind an L7 load balancer.

The incident:

14:02  routine deploy begins, rolling 6 servers at a time
14:02  ~34,000 clients disconnected
14:02  all reconnect within ~2 s
14:03  auth service (JWT validation, a database lookup) saturates: p99 4s -> 40s
14:03  reconnects time out; clients retry IMMEDIATELY (no backoff)
14:04  remaining 54 servers at 100% CPU handling handshakes
14:05  healthy servers fail liveness probes -> Kubernetes restarts them
14:05  their connections drop -> more reconnects
14:12  full outage
14:41  recovered by pausing the deploy, scaling auth 5x, and rate-limiting
       connections at the LB

Twenty-nine minutes of outage from a routine deploy, and steady-state capacity was never the constraint.

Root causes, in order of contribution:

1. No client backoff. The client reconnected immediately on close and retried immediately on failure.

// What it was:
socket.onclose = () => connect();      // immediate, forever

// What it became: full jitter, capped, reset only on success.
let attempt = 0;
socket.onclose = (e) => {
  if (e.code === 1000) return;                        // clean close: do not reconnect
  const base = Math.min(30_000, 1_000 * 2 ** attempt);
  attempt++;
  setTimeout(connect, Math.random() * base);          // FULL jitter
};
socket.onopen = () => { attempt = 0; };

2. Auth on every reconnect, hitting the database. A JWT was validated by a database lookup of the session, so 34,000 reconnects were 34,000 queries in two seconds.

// Before: a DB lookup per connection.
// After: verify the JWT signature locally; check revocation against a
// bloom filter refreshed every 30s.
const claims = jwt.verify(token, publicKey);          // local, ~50 us
if (revokedFilter.mightContain(claims.jti)) {         // in-memory
  await checkRevocationInDb(claims.jti);              // rare
}

auth cost per reconnect:  ~14 ms (DB) -> ~0.06 ms (local)

3. No drain on shutdown. SIGTERM closed the server immediately, severing all connections at once.

// After: spread closes over 90 seconds, with code 1001.

terminationGracePeriodSeconds: 120     # must EXCEED the drain window

4. The backplane amplified it. Every server subscribed to a global channel and filtered locally.

steady state:      12,000 msg/s x 60 servers = 720,000 deliveries/s
during the storm:  presence updates for 34,000 reconnecting clients,
                   each broadcast to 60 servers
                   -> Redis at 100% CPU on a single thread

// After: one channel per document, subscribed only where there are
// local participants.
redis.subscribe(`doc:${docId}`);

deliveries/s at steady state:  720,000 -> 14,000     (-98%)
Redis CPU:                     78% -> 6%

Redis Pub/Sub is single-threaded, so a busy backplane is a single-core limit, and this was the second-largest contributor.

5. Connection limits at the LB were absent. Nothing bounded the accept rate, so a storm reached the application in full.

# At the load balancer: cap NEW connections per second per backend.
limit_conn_zone $server_name zone=ws:10m;
limit_req_zone  $binary_remote_addr zone=wsconn:10m rate=5r/s;

Verified by replaying the same deploy in a load test:

                              before      after
clients disconnected per
  rolled server               ~5,700      ~5,700 (unchanged: they must move)
reconnect window              ~2 s        ~90 s (drained)
peak reconnects/s             ~17,000     ~190
auth service p99              40 s        41 ms
Redis CPU during a deploy     100%        11%
servers failing liveness      54 of 60    0
deploy outcome                outage      no user-visible impact

The connection count did not change and the rate did. That is the whole lesson: a WebSocket platform's capacity has two numbers, concurrent connections and connection establishment rate, and the second is what deploys and network blips test.

The steady-state tuning done afterwards:

                              before      after
connections per server        5,700       14,000
memory per server             12 GB       9 GB       (socket buffer tuning)
servers                       60          26
ulimit -n                     65,536      1,048,576
tcp_rmem default              87,380      16,384
ephemeral ports at the LB     default     widened + 4 backend ports

Twenty-six servers instead of sixty, from socket buffer defaults and file descriptor limits, once the reconnect behaviour was safe enough to consolidate.

Production evidence

Slack, Discord and Figma have all published on WebSocket scale, and the recurring themes are the same three: consistent hashing for connection placement, a purpose-built backplane rather than broadcast-to-everything, and reconnect behaviour as the dominant operational concern. Discord's published work on Elixir and later Rust for their gateway is largely about the fan-out problem.

Redis Pub/Sub's single-threaded delivery is documented, and it is why Redis Cluster's pub/sub broadcasts to all nodes (which does not help) and why sharded pub/sub was added in Redis 7.0 to confine a channel to one shard.

Socket.IO's Redis adapter implements the broadcast-to-everything model by default, which is why Socket.IO deployments hit the backplane limit at a few dozen servers, and why the sharded and cluster adapters exist.

The full-jitter backoff formula is from AWS's "Exponential Backoff And Jitter" article, which measured that random(0, base) disperses a herd substantially better than base + random(0, jitter), and the difference is largest exactly in the reconnect-storm case.

Kubernetes' terminationGracePeriodSeconds and preStop are the mechanism for draining, and the WebSocket case is the clearest illustration of why the grace period must exceed the drain window: the same argument as the kube-proxy page, with a much longer window.

Cloudflare's and Fastly's documentation on WebSocket support both note the connection duration limits imposed by their infrastructure, which is a constraint worth knowing: an intermediary that closes idle connections at 100 seconds makes application-level pings mandatory rather than optional.

The debate

Sticky routing or a backplane? Both, and framing it as a choice is the error. Sticky routing solves "this client's messages reach a consistent process"; a backplane solves "a message produced anywhere reaches the process holding the recipient." A system with only sticky routing cannot fan out; a system with only a backplane still needs the client's own messages to land somewhere consistent for any per-connection state.

Which backplane? Redis Pub/Sub for ephemeral state (presence, typing indicators, cursor positions) where a missed message is invisible, and it is single-threaded so it becomes a one-core limit. Redis Streams or NATS JetStream when a missed message matters. Kafka when the messages are also a business event stream and you want replay. The mistake is defaulting to Redis Pub/Sub for messages that must not be lost, because its failure mode (a restarting subscriber misses everything sent while it was down) is invisible in testing.

Should every server subscribe to everything? No, and this is the change that most often unblocks scale. Broadcast-to-all means the bus carries messages x servers, so it becomes the limit at a few dozen servers. Channel-per-topic with dynamic subscribe and unsubscribe costs subscription churn and reduces delivery volume by the fan-out factor: 98 percent in the worked example. The trade is bus CPU against subscription-management complexity, and above about 20 servers the first dominates.

How do you deploy without disconnecting everyone? You cannot avoid disconnecting the clients on a replaced instance; you can control the rate. Draining over 60 to 90 seconds with close code 1001 turns an instantaneous storm into a normal reconnect rate, and it requires terminationGracePeriodSeconds to exceed the window. The alternative sometimes proposed, connection migration, is not realistic for most stacks: it requires transferring socket state between processes and the complexity dwarfs the benefit.

Are WebSockets the right choice at all? Frequently not. For server-to-client streaming only, SSE is simpler: it is plain HTTP, it reconnects and resumes automatically via Last-Event-ID, it works through every proxy, and it has no upgrade handshake. See SSE vs WebSockets. WebSockets earn their place when the client needs to send frequently on the same connection, which is collaborative editing, gaming and bidirectional protocols, and not most notification use cases.

What should you measure? Concurrent connections and connection establishment rate, and the second is the one that is usually not on a dashboard. Steady-state capacity planning misses the failure entirely, because a platform comfortable at 340,000 connections went down at 17,000 new connections per second. Add reconnect rate, close codes by category, and time to drain during a deploy.

Follow-up Q&A

"How do you scale WebSockets across many servers?"

Two mechanisms for two problems. Sticky routing, ideally consistent hashing on a client key rather than ip_hash, so a client's connection and any per-connection state stay on one process, and so adding a server moves 1/N of clients rather than most of them. And a pub/sub backplane so a message produced on any server reaches the process holding the recipient. You need both: sticky alone cannot fan out, and a backplane alone still leaves per-connection state scattered.

"Why not ip_hash?"

It breaks with carrier-grade NAT, where thousands of mobile clients share one source IP and land on one server. It rebalances almost everything when the server set changes, and on a WebSocket service every rebalance is a disconnect: plain hashing moves about 75 percent of clients when going from 3 servers to 4, consistent hashing moves about 25. And it gives you no control over the key, so you cannot pin by tenant or by document.

"What breaks first when you scale a WebSocket service?"

Usually file descriptors, because the default ulimit -n is 1024. Then ephemeral ports at the load balancer, which are about 28,000 per backend ip:port tuple, so you widen the range or add backend ports. Then socket buffer memory, because the defaults are sized for throughput per connection rather than connection count: tuning tcp_rmem/tcp_wmem defaults took one service from 12 GB to 9 GB per server at the same connection count. And then the backplane, if every server subscribes to everything.

"What is the reconnect storm and how do you prevent it?"

When a server is replaced, all its clients reconnect at once, and naive clients retry immediately on failure, so a failed reconnect feeds the storm. In one case a routine deploy produced 17,000 reconnects per second, saturated the auth service, caused healthy servers to fail liveness probes, and became a 29-minute outage. The fixes are client-side full-jitter backoff (random(0, base), not base + random), server-side draining that spreads closes over 60 to 90 seconds with code 1001, and making the reconnect path cheap: local JWT verification instead of a database lookup.

"Why is full jitter better than adding jitter to a base delay?"

Because base + random(0, jitter) still has every client waiting at least base, so the herd is delayed rather than dispersed and arrives as a slightly wider spike. random(0, base) spreads clients uniformly across the whole interval. AWS measured the difference and the effect is largest exactly in this scenario, where a large population fails and retries simultaneously.

"When would you use SSE instead?"

When the traffic is server-to-client only, which covers most notification, feed and streaming cases. SSE is plain HTTP, so it works through every proxy without an upgrade, and it reconnects and resumes automatically via Last-Event-ID, which is the reconnect handling you would otherwise write yourself. WebSockets earn their place when the client sends frequently on the same connection: collaborative editing, gaming, bidirectional protocols.

Common misconceptions

"Sticky routing or a backplane." Both, for different problems. Sticky keeps a client's connection and state on one process; the backplane gets messages to whichever process holds the recipient.

"ip_hash is sticky routing." It is the weakest form: it breaks under carrier-grade NAT, it rebalances most clients when the server set changes, and every rebalance is a disconnect.

"WebSocket capacity is concurrent connections." It is two numbers, and the second (establishment rate) is what deploys and network events test. A platform comfortable at 340,000 connections failed at 17,000 new connections per second.

"Redis Pub/Sub is a message queue." It is fire-and-forget with no persistence: a subscriber that is down misses everything sent while it was down, permanently, and the failure is invisible in testing.

"A rolling deploy is safe because it is gradual." It is gradual in servers and instantaneous in disconnections: replacing one server of 60 severs all of its connections at once. Draining spreads the reconnects; the deploy pace does not.

Interview delivery note

Say this verbatim: "A WebSocket platform has two capacity numbers, concurrent connections and connection establishment rate, and the second is the one nobody dashboards. A system comfortable at 340,000 connections went down at 17,000 new connections a second during a routine deploy, because the clients had no backoff and the server severed connections instead of draining." Two numbers, and the specific failure the second one predicts.

The senior-versus-staff separator is full jitter versus jittered backoff. A senior engineer adds exponential backoff. A staff engineer specifies random(0, base) rather than base + random(0, jitter), because the second form still has every client waiting at least base, so it delays the herd rather than dispersing it. That is a one-line difference that determines whether backoff works at all in the reconnect-storm case.

The second signal is treating the drain window as a deploy parameter. Saying "spread closes over 90 seconds with code 1001, and terminationGracePeriodSeconds must exceed that or Kubernetes kills the process mid-drain" shows you have connected the WebSocket lifecycle to the orchestrator's, which is where the practical failure is.

The Staff & Team Lead Interview Reference