Deploying stateful services and long-lived connections

What it is

Two deployment problems that share one root cause: the unit being replaced is not interchangeable.

Stateless HTTP           a replica is fungible. Kill it, the load
                         balancer routes elsewhere, the request
                         retries. Nothing is lost.

Long-lived connections   the replica holds thousands of sockets that
(WebSocket, SSE, gRPC    represent user sessions. Killing it does not
streams, MQTT)           lose a request, it loses N sessions, and all
                         N clients reconnect at once.

Stateful services        the replica holds DATA and a ROLE. It may be
(Kafka, etcd, Cassandra, the leader, or one of three quorum members.
Postgres, Elasticsearch, Killing the wrong one at the wrong time
Redis)                   loses availability or, worse, writes.

What this is confused with: PodDisruptionBudgets protecting rollouts. A PDB constrains the Eviction API, which is what kubectl drain, the cluster autoscaler and the descheduler use. A Deployment or StatefulSet rolling update deletes pods directly and is governed by maxUnavailable in the workload spec, not by the PDB. This is the single most common misconception in this area and it produces a false sense of safety.

Also confused: graceful shutdown and connection draining. Graceful shutdown means the process finishes in-flight work before exiting. Draining means the load balancer stops sending new work before the process is told to stop. You need both, and they are configured in different places.

The problem it solves

The connection case: a rolling restart is a synchronised reconnect storm.

100,000 WebSocket clients, 20 pods -> 5,000 connections per pod.
Rolling update with maxUnavailable: 25% -> 5 pods replaced at once.

25,000 clients disconnect within the same second.
If clients reconnect immediately with no jitter, the remaining 15
pods receive 25,000 connection attempts in ~1 second, which is
1,667/s per pod against a normal rate of maybe 20/s.

Each reconnect costs a TLS handshake, an auth check, and a session
restore. The surviving pods saturate, drop connections, and those
clients reconnect too. The rollout has become an outage.

The stateful case: the rollout violates a quorum invariant.

A 3-node etcd cluster tolerates 1 failure.
maxUnavailable: 1 is mandatory, not a preference.

But the danger is compounded: during a rolling restart of node 3,
node 1 is cordoned by an unrelated cluster autoscaler action.
Two of three are gone. The cluster loses quorum and every write
in the platform fails.

This is exactly the case PDBs exist for, AND the case where people
believe the PDB is protecting the rollout when it is only
protecting against the drain.

And the Kubernetes race that catches everyone:

On pod deletion, two things happen CONCURRENTLY:
  A) the kubelet sends SIGTERM to the container
  B) the endpoints controller removes the pod from the Endpoints/
     EndpointSlice, which then propagates to kube-proxy on every
     node, to the ingress controller, and to the cloud load balancer

B is eventually consistent and takes hundreds of milliseconds to
several seconds. A is immediate.

So a pod routinely receives new connections AFTER it has begun
shutting down, and those requests fail.

Mechanics

Draining correctly in Kubernetes

The standard shape, and every part of it is load-bearing:

spec:
  # Must exceed preStop sleep + the app's own drain time.
  terminationGracePeriodSeconds: 120
  containers:
    - name: app
      lifecycle:
        preStop:
          exec:
            # Do NOT start shutting down yet. Sleep long enough for
            # endpoint removal to propagate to every proxy and LB.
            # SIGTERM is not sent until this completes.
            command: ["/bin/sh", "-c", "sleep 15"]
      readinessProbe:
        httpGet: { path: /readyz, port: 8080 }
        periodSeconds: 2
        failureThreshold: 2

The sequence this produces:

t=0     pod marked Terminating; endpoints controller begins removal
t=0     preStop starts: sleep 15
t=0-5   endpoint removal propagates to kube-proxy / ingress / ALB
        (the pod still serves during this window, correctly)
t=15    preStop returns; kubelet sends SIGTERM
t=15+   app stops accepting NEW connections, finishes in-flight
        work, closes idle connections
t<=120  process exits, or SIGKILL at the grace period

Without the preStop sleep, the app receives SIGTERM at t=0 and starts refusing connections while proxies are still sending them. The sleep is not a hack; it is the documented way to wait out an eventually-consistent removal you cannot observe from inside the pod.

For long-lived connections, the application must also drain them deliberately:

// On SIGTERM: stop accepting, then close existing connections over a
// window with jitter, so 5,000 clients do not reconnect in the same
// second.
func (s *Server) Drain(ctx context.Context, window time.Duration) {
    s.acceptingNew.Store(false)          // readiness probe now fails
    conns := s.snapshotConnections()      // e.g. 5,000

    // Spread closes across the window. With window=60s and 5,000
    // connections that is ~83 closes/second, and each client's own
    // jittered backoff spreads the reconnects further.
    step := window / time.Duration(len(conns))
    for _, c := range conns {
        // Tell the client to go away politely and when to come back.
        // For WebSocket: a close frame with a reconnect hint.
        // For HTTP/2 and gRPC: GOAWAY, which lets the client finish
        // in-flight streams and open a new connection elsewhere.
        c.CloseWithHint(reconnectAfter(2*time.Second, 30*time.Second))
        select {
        case <-time.After(step):
        case <-ctx.Done():
            return
        }
    }
}

Three client-side requirements that the server cannot provide:

1. JITTERED EXPONENTIAL BACKOFF. Never a fixed retry interval.
     delay = min(cap, base * 2^attempt) * random(0.5, 1.5)
   Without jitter, the herd re-synchronises on every retry round.

2. HONOUR THE RECONNECT HINT if the server sends one.

3. RESUMABLE SESSIONS. A reconnect that replays state from scratch
   turns a connection storm into a data storm. Last-event-id for
   SSE, a session token plus a cursor for WebSocket.

A fourth control removes the problem's cause: bounded connection lifetime.

Set a max connection age of, say, 30 to 60 minutes with jitter, so
clients reconnect continuously at a low rate rather than all at once
at deploy time.

100,000 connections / 45 min mean lifetime = ~37 reconnects/second,
continuously. The system is always handling reconnects, so a deploy
is not a special event, and the capacity to absorb them is proven
every minute rather than assumed.

That inversion, making the exceptional case continuous so it is always tested, is the strongest form of the fix, and it is the same reasoning as chaos engineering applied to a specific mechanism.

Rolling stateful services

Start from the invariant, not from the config.

Quorum systems (etcd, ZooKeeper, Consul, Kafka's KRaft controllers,
Raft-based databases):
    N nodes tolerate floor((N-1)/2) failures.
    3 -> 1,   5 -> 2,   7 -> 3
    So maxUnavailable is 1 for a 3-node cluster. Always.

Replication systems (Cassandra, Elasticsearch):
    the constraint is per-shard/per-token-range replica count and
    the consistency level, not a cluster-wide number. With RF=3 and
    QUORUM reads, you can lose 1 replica per range. Restarting two
    nodes that happen to share a range breaks it even if the
    cluster-wide count looks safe.
    -> use rack/zone awareness so one "rack" can be restarted as a
       unit, which is what Cassandra's rack concept is for.

Primary/replica (Postgres, MySQL, Redis):
    the primary is special. Restarting it means a failover, which
    has a cost measured in seconds of write unavailability and, if
    replication is asynchronous, possible data loss.
    -> restart replicas first, then fail over deliberately, then
       restart the old primary.

StatefulSet mechanics:

apiVersion: apps/v1
kind: StatefulSet
spec:
  podManagementPolicy: OrderedReady    # start/stop one at a time
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      # Only pods with ordinal >= partition are updated. This is the
      # canary mechanism for stateful sets: set partition to N-1,
      # observe the single updated pod, then lower it.
      partition: 2

With replicas: 3 and partition: 2
  -> only pod-2 is updated. pod-0 and pod-1 stay on the old version.
  Observe pod-2 for as long as you want (it holds real data and
  serves real traffic), then set partition: 1, then 0.

This is a genuine canary for a stateful service, and it is the
feature most teams do not know exists.

The PodDisruptionBudget, and what it actually covers:

apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  maxUnavailable: 1
  selector:
    matchLabels: { app: etcd }

PROTECTS AGAINST (voluntary disruptions, via the Eviction API):
  kubectl drain
  cluster autoscaler scaling down a node
  descheduler rebalancing
  node upgrades performed by a managed control plane

DOES NOT PROTECT AGAINST:
  a StatefulSet or Deployment rolling update (the controller
    deletes pods directly; use maxUnavailable / OrderedReady)
  a node crashing or a kernel panic
  a kubelet losing contact
  someone running `kubectl delete pod`

So: PDB *and* an update strategy. They cover different things and
neither substitutes for the other.

Readiness must mean "caught up", not "process started".

readinessProbe:
  # For a replica rejoining a cluster, this endpoint must return
  # 200 only when replication lag is within tolerance. A probe that
  # checks the port routes reads to a replica that is minutes behind.
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
startupProbe:
  # Stateful nodes can take many minutes to load or catch up. The
  # startup probe grants that time WITHOUT making the liveness
  # probe's timeout absurdly long for steady state.
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 60
  periodSeconds: 10          # allows up to 10 minutes to start

Without a startupProbe, teams lengthen the liveness probe's failureThreshold to survive startup, which means a genuinely wedged process is not restarted for ten minutes in steady state. The two probes exist to decouple those.

Graceful role transfer beats election timeout.

Kafka: controlled.shutdown.enable=true (the default) makes the
  broker move partition leadership to other replicas BEFORE it
  stops. Without it, every partition it led becomes leaderless and
  waits for the controller to elect a new leader, which is a burst
  of unavailability proportional to the number of partitions.

etcd/Raft: `etcdctl move-leader <id>` transfers leadership
  explicitly. Otherwise followers wait out the election timeout
  (default around 1s, with heartbeats at 100ms) before starting an
  election, and writes stall for that period plus the election.

Postgres: a planned switchover (Patroni's `switchover`) is a
  coordinated handoff. A failover is the unplanned version and
  costs more.

Rule: restart followers first, transfer leadership deliberately,
restart the old leader last.

Anti-affinity and topology spread, so the topology does not undo the arithmetic:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels: { app: etcd }

A 3-node quorum with two nodes in the same availability zone tolerates zero zone failures, which makes the maxUnavailable: 1 reasoning meaningless. Spread is a precondition for the quorum arithmetic to hold.

Local persistent volumes pin a pod to a node, so a node drain cannot reschedule it. That is a deliberate trade (local NVMe performance) and it means node maintenance requires an explicit data movement step rather than an eviction.

A worked example: a chat platform's deploy that took the platform down

A messaging product. 180,000 concurrent WebSocket connections across 30 gateway pods, backed by a 5-node Redis cluster for presence and a 3-node etcd for coordination.

The incident:

14:02  routine gateway deploy begins. Deployment defaults:
       maxUnavailable 25%, maxSurge 25%, no preStop, grace period 30s.
14:02  8 pods enter Terminating. SIGTERM immediately. The app closes
       the listener and exits in ~2 seconds.
       48,000 connections dropped in under 3 seconds.
14:02  clients reconnect immediately, fixed 1-second retry, no jitter.
       ~48,000 connection attempts hit the remaining 22 pods within
       2 seconds: ~1,090/s per pod against a baseline of ~25/s.
14:03  each reconnect does a TLS handshake + auth + presence restore
       (3 Redis round trips). The 22 pods saturate CPU; the Redis
       cluster's ops/s goes from 40k to 310k.
14:03  Redis latency p99: 0.8ms -> 340ms. Presence restore times out.
       Gateways drop the new connections.
14:03  clients retry again, still synchronised. The herd re-forms.
14:11  full outage of the messaging product. 9 minutes.

Every element of the cascade came from a default.

The five fixes, and what each was worth:

1. preStop sleep 15 + terminationGracePeriodSeconds 180
   Endpoint removal now propagates before SIGTERM, so no new
   connections arrive at a shutting-down pod.
   -> removed the "connection refused during rollout" errors
      entirely (previously ~4,000 per deploy).

2. Application drain over a 90-second window with per-connection
   jitter, sending a close frame carrying a reconnect hint.
   6,000 connections per pod / 90s = ~67 closes/s per pod.
   8 pods draining concurrently = ~530 reconnects/s platform-wide,
   against a previous 48,000-in-3-seconds.
   -> peak reconnect rate fell by ~99%.

3. Client backoff with full jitter, and honouring the hint:
     delay = random(0, min(30s, 0.5s * 2^attempt))
   -> retry rounds stop re-synchronising. This was the fix that
      prevented the herd RE-FORMING after the first failure, which
      is what turned a 3-second event into a 9-minute one.

4. Resumable sessions: a session token plus a last-received cursor,
   so a reconnect restores presence with 1 Redis round trip
   instead of 3, and does not re-send history.
   -> Redis ops per reconnect: 3 -> 1.

5. Bounded connection age: 40 minutes, +/- 25% jitter.
   180,000 / 40 min = ~75 reconnects/s, continuously.
   -> a deploy is no longer a special event. The system's capacity
      to absorb reconnects is exercised every minute.

Fix 5 is the one that changed the character of the problem. After it, the deploy-time reconnect rate (~530/s) was only about seven times the continuous baseline (~75/s), rather than a 2,000x spike against a system that had never handled one.

Measured after:

                            before        after
connections dropped
  per deploy                48,000        drained over 90s
peak reconnect rate       ~24,000/s       ~530/s
Redis peak ops/s            310,000       58,000
errors per deploy            ~4,000       0
deploy duration              4 min        11 min
deploys per day                  1        8

Deploys got slower and became routine, which was the correct trade and needed to be stated explicitly, because "the deploy takes three times as long" was raised as an objection.

The stateful half, found during the same review.

The 3-node etcd cluster had:
  - a PDB with maxUnavailable: 1  (correct)
  - a StatefulSet with no partition strategy
  - no topology spread constraints
  - all three pods, as scheduled, on nodes in TWO availability
    zones: two in zone a, one in zone b.

Consequences:
  a) The PDB did NOT protect the rolling update, because the
     StatefulSet controller deletes pods directly. The team
     believed it did.
  b) Losing zone a would have taken 2 of 3 nodes and lost quorum,
     so the cluster tolerated zero zone failures despite being a
     3-node quorum.

Fixes:
  - topologySpreadConstraints with maxSkew 1 over zone, and
    DoNotSchedule, forcing one node per zone across three zones.
  - podManagementPolicy: OrderedReady, plus updateStrategy
    partition used as a canary: update ordinal 2, observe for 10
    minutes, then 1, then 0.
  - readiness reflecting raft membership and applied-index lag
    rather than the port being open.
  - `etcdctl move-leader` before restarting the pod that held
    leadership, so writes stalled for the transfer rather than for
    the election timeout.

Measured: write unavailability during an etcd rollout fell from
~3.2s per node restart (election timeout plus reconnection) to
~180ms (leadership transfer only).

The PDB misconception is the finding worth carrying, because the configuration looked correct in review and protected against the wrong thing.

Production evidence

Kubernetes documentation states that PodDisruptionBudgets apply to voluntary disruptions via the Eviction API and explicitly notes they do not apply to deletions performed by controllers during updates, which is the documented basis for the distinction this page draws.

The preStop sleep pattern is the standard mitigation for the endpoint-propagation race, and Kubernetes' own documentation on pod termination describes the concurrency between SIGTERM delivery and endpoint removal that makes it necessary.

Kafka's controlled.shutdown.enable (enabled by default) moves partition leadership off a broker before it stops, and Kafka's documentation describes the alternative as leaving partitions leaderless until the controller elects replacements.

etcd's move-leader command exists specifically so a planned restart transfers leadership rather than triggering an election, and etcd's tuning documentation gives the default heartbeat (100ms) and election timeout (1000ms) that set the cost of not using it.

Cassandra's rack awareness places replicas in distinct racks so that a whole rack can be restarted without losing a quorum for any token range, which is the replication-system analogue of topology spread.

AWS's guidance on jittered exponential backoff (the "Exponential Backoff and Jitter" article from the AWS Architecture Blog) is the canonical treatment of why unjittered retries re-synchronise a herd, and full jitter is the variant it recommends.

Slack's published writing on its WebSocket infrastructure describes the reconnect-storm problem directly, including that a mass disconnect is far more expensive than the connections themselves because each reconnect carries session restoration work.

The debate

Should long-lived connections be drained slowly or dropped fast? Drained, over a window proportional to the connection count, with a reconnect hint. The counter-argument is deploy duration, and it is legitimate: a 90-second drain per batch multiplies rollout time. The resolution is bounded connection age, which makes reconnects continuous, proves the capacity every minute, and makes the drain window less critical.

Is a preStop sleep a hack? It is the documented answer to an eventually-consistent removal you cannot observe from inside the pod. The alternative, having the application keep serving after SIGTERM until it observes no traffic for N seconds, is more precise and more complex, and it fails when traffic is naturally bursty. Prefer the sleep; length it from measured propagation time rather than folklore.

Should stateful services run on Kubernetes at all? Increasingly yes for well-supported systems with mature operators, and the honest caveat is that the operator is doing the work described here on your behalf and you should know what it does. For a small team running one Postgres, a managed service is usually the better trade, because the failure modes above are the operator's core competence and not yours.

Is maxUnavailable: 1 sufficient for a quorum system? Only in combination with topology spread and a PDB. The workload's maxUnavailable governs your rollout; the PDB governs someone else's node drain; topology spread governs the cloud provider's zone failure. All three are needed, and any one of them alone gives a false sense of safety, which is exactly what a correct-looking PDB provides.

Should you use the StatefulSet partition canary? Yes, for anything holding data. It is the only mechanism that lets a stateful node run the new version against real data and real traffic for an arbitrary period before its peers follow. The reason it is rarely used is discoverability rather than any drawback.

Is bounded connection age wasteful? It spends a continuous low rate of reconnect work to remove a large periodic spike. For any system where a mass reconnect is expensive, it is a good trade, and its real value is that it converts an untested emergency path into an exercised steady-state path.

Follow-up Q&A

"Why does a pod receive requests after it has started shutting down?"

Because SIGTERM delivery and endpoint removal happen concurrently, and endpoint removal is eventually consistent: the endpoints controller updates the EndpointSlice, which then propagates to kube-proxy on every node, to the ingress controller and to any cloud load balancer, taking hundreds of milliseconds to several seconds. The pod is told to stop immediately. The standard fix is a preStop hook that sleeps for longer than the measured propagation time, since SIGTERM is not sent until preStop returns, and a terminationGracePeriodSeconds that exceeds the sleep plus the application's own drain time.

"How do you deploy a service holding 100,000 WebSocket connections?"

Four things. Drain rather than drop: on SIGTERM stop accepting, then close existing connections spread across a window with a reconnect hint, so 5,000 per pod become roughly 67 closes per second rather than 5,000 at once. Require jittered exponential backoff on the client, because without jitter every retry round re-synchronises the herd and one spike becomes a sustained outage. Make sessions resumable so a reconnect restores state with one round trip rather than replaying from scratch. And bound connection age with jitter, so reconnects happen continuously at a low rate and a deploy stops being a special event.

"What does a PodDisruptionBudget actually protect?"

Voluntary disruptions that go through the Eviction API: kubectl drain, the cluster autoscaler, the descheduler, and managed node upgrades. It does not constrain a Deployment or StatefulSet rolling update, because those controllers delete pods directly, and it does not help with node crashes or a manual kubectl delete pod. So a quorum system needs a PDB and a workload maxUnavailable and topology spread, covering three different threats, and having only the PDB is the common correct-looking mistake.

"How do you roll a 3-node etcd or ZooKeeper cluster safely?"

One node at a time, because a 3-node quorum tolerates exactly one failure. Use OrderedReady pod management and the StatefulSet partition field to update the highest ordinal first and observe it against real traffic before lowering the partition. Make readiness reflect raft membership and applied-index lag rather than the port being open, and use a startupProbe so a slow catch-up does not force an absurd liveness timeout in steady state. Transfer leadership explicitly before restarting the leader, since otherwise followers wait out the election timeout, which took write unavailability per node restart from about 3.2 seconds to about 180 milliseconds in one case. And spread across three zones, because two of three nodes in one zone means the cluster tolerates no zone failure at all.

"Why is readiness different for stateful services?"

Because the process being up does not mean the node can serve. A replica rejoining after a restart may be minutes behind, and a readiness probe that checks the port will route reads to it and serve stale data. Readiness must encode the actual condition, replication lag within tolerance, raft membership established, shards recovered. And because catch-up can take many minutes, use a startupProbe to grant that time rather than lengthening the liveness probe, which would leave a genuinely wedged process unrestarted for the same duration in steady state.

"What is the argument for bounded connection lifetime?"

It converts a rare, expensive, untested event into a continuous, cheap, always-exercised one. With 180,000 connections and a 40-minute jittered maximum age, the system handles about 75 reconnects per second all the time, so the capacity to absorb reconnects is proven every minute rather than assumed at deploy time. After that change the deploy-time reconnect rate was only about seven times baseline rather than a thousandfold spike, which is the difference between a load the system routinely handles and one it has never seen.

Common misconceptions

"The PDB protects my rolling update." It constrains the Eviction API. Controllers delete pods directly during updates; use maxUnavailable and OrderedReady for that.

"Graceful shutdown is enough." Graceful shutdown finishes in-flight work. It does not stop the load balancer from sending new work, which is what the preStop delay and readiness failure handle.

"A 3-node quorum tolerates one node failure." Only if the three nodes are in three failure domains. Two in one zone means it tolerates zero zone failures.

"Dropping connections is fine, clients reconnect." They reconnect simultaneously. Without jitter and resumable sessions, the reconnect is more expensive than the traffic it replaces, and the herd re-forms on every retry round.

"Readiness means the process started." For a stateful node it must mean caught up, or you route reads to a replica minutes behind.

"Restarting the leader is the same as restarting a follower." It costs an election unless you transfer leadership deliberately, and for a primary/replica database it may cost data if replication is asynchronous.

Interview delivery note

Say this verbatim: "A PodDisruptionBudget constrains the Eviction API, so it protects against node drains and the autoscaler, not against your own rolling update, which deletes pods directly and is governed by maxUnavailable. A quorum service needs all three: maxUnavailable for your rollout, the PDB for someone else's drain, and topology spread for the cloud provider's zone failure." It is a precise correction of a belief most engineers hold.

The senior-versus-staff separator is bounded connection age as the structural fix. A senior engineer drains connections slowly at deploy time. A staff engineer points out that this leaves an expensive path exercised once per deploy, and instead bounds connection lifetime with jitter so reconnects run continuously at 75 per second, which proves the capacity every minute and reduces the deploy-time spike from roughly 24,000 per second to 530. Converting an untested emergency path into an exercised steady-state path is the general move, and it is the same reasoning as chaos engineering applied to one mechanism.

The second signal is naming the reconnect cascade rather than the disconnect. The nine-minute outage in the worked example was not caused by dropping 48,000 connections, which took three seconds; it was caused by unjittered client retries re-synchronising after the first failure. Knowing that the client's backoff policy is part of your deployment safety, and that you cannot fix this server-side alone, is the depth signal.

The Staff & Team Lead Interview Reference