kube-proxy modes, and why iptables degrades at scale

What it is

A Kubernetes Service is a virtual IP that does not exist on any interface. Nothing listens on a ClusterIP. The IP is a rule in the node's packet-processing path, and kube-proxy is the component that writes those rules from the EndpointSlice objects the control plane maintains.

Three implementations, differing in the data structure used for the lookup:

Mode	Data structure	Rule lookup	Rule update
iptables	Linear chains, evaluated in order	O(n) in rules	O(n): full table rewrite
IPVS	Kernel hash table	O(1)	O(1) per endpoint
eBPF (Cilium, no kube-proxy)	eBPF hash map at the socket or TC layer	O(1)	O(1), and no NAT at all for some paths

The degradation is in both dimensions and the second is worse. iptables evaluates rules sequentially, so packet processing cost grows with the number of Services. And because iptables has no partial-update primitive, adding one endpoint requires writing the whole table, so a control-plane change costs time proportional to total cluster size.

What this is confused with: kube-proxy being on the data path. It is not. kube-proxy is a control-plane component that programs the kernel and then gets out of the way; packets never traverse a userspace proxy (except in the long-removed userspace mode). kube-proxy being slow does not make requests slow; it makes endpoint changes take longer to apply, which is a different and often worse problem.

The problem it solves

Pods are ephemeral and their IPs change on every restart. A Service gives a stable virtual IP plus load balancing across whatever pods currently match its selector, and that mapping has to be enforced somewhere in the packet path on every node.

The failures that make the mode choice matter:

Rule count growth. iptables rules per Service scale with endpoints:

Per Service:      ~2 rules in KUBE-SERVICES (one per port, plus a masquerade rule)
Per endpoint:     ~2 rules (a probability-based jump plus a DNAT rule)

1,000 Services x 10 endpoints each:
  ~2,000 service rules + ~20,000 endpoint rules = ~22,000 rules
5,000 Services x 20 endpoints:
  ~210,000 rules

Sequential evaluation means the last Service's rules are evaluated after all the others. The kernel walks the chain until a rule matches.

Update cost, which is the one that actually breaks clusters. iptables replaces the entire table atomically:

Rules      iptables-restore time    Effect
─────────────────────────────────────────────────────────────
  5,000    ~0.1 s                   fine
 20,000    ~0.5 s                   fine
 50,000    ~2 s                     noticeable propagation delay
100,000    ~8 s                     endpoint changes take seconds to apply
200,000    ~30 s+                   a rolling deploy is effectively broken

At 30 seconds per sync, a pod that terminates is still receiving traffic 30 seconds later, because the rule removing it has not been written yet. That produces connection errors during every deploy, on every Service, and it looks like an application problem.

Mechanics

iptables mode

$ iptables -t nat -L KUBE-SERVICES -n
Chain KUBE-SERVICES (2 references)
target                     prot  source     destination
KUBE-SVC-XJKL2MNOPQ...     tcp   0.0.0.0/0  10.96.0.10  tcp dpt:53
KUBE-SVC-ABCD3EFGHI...     tcp   0.0.0.0/0  10.96.1.42  tcp dpt:80
... one line per Service port, evaluated IN ORDER

$ iptables -t nat -L KUBE-SVC-ABCD3EFGHI -n
Chain KUBE-SVC-ABCD3EFGHI (1 references)
target              prot  opt  ...
KUBE-SEP-AAAA...    all   --   statistic mode random probability 0.33333333349
KUBE-SEP-BBBB...    all   --   statistic mode random probability 0.50000000000
KUBE-SEP-CCCC...    all   --                                    (the remainder)

The probabilities are conditional, not absolute. With three endpoints the first gets 1/3; if it does not match, the second gets 1/2 of the remaining 2/3, which is 1/3; the third takes the rest. That is how sequential evaluation produces a uniform distribution, and it is also why the load balancing is per-connection random with no state: no least-connections, no locality, no health awareness beyond what EndpointSlice reports.

$ iptables -t nat -L KUBE-SEP-AAAA -n
DNAT  all  --  0.0.0.0/0  0.0.0.0/0  to:10.244.1.7:8080

Every connection creates a conntrack entry, because DNAT requires tracking so the reply can be un-NATed. That is the link to the conntrack exhaustion on the TCP tuning page: iptables mode means every Service connection consumes a conntrack slot.

Partial updates arrived in Kubernetes 1.26 (minSyncPeriod plus incremental rule updates using iptables-restore --noflush for changed chains only), which substantially improved the update cost. The O(n) packet-path cost is inherent to sequential evaluation and did not change.

IPVS mode

$ ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
TCP  10.96.1.42:80 rr
  -> 10.244.1.7:8080    Masq    1      412        0
  -> 10.244.2.9:8080    Masq    1      408        0
  -> 10.244.3.4:8080    Masq    1      419        0

A kernel hash table, so lookup is O(1) regardless of Service count, and it exposes real scheduling algorithms:

rr      round robin (the default)
lc      least connection          <- genuinely useful for uneven request costs
sh      source hashing            <- client affinity without sessionAffinity
dh      destination hashing
sed     shortest expected delay
nq      never queue

lc is the practical reason to choose IPVS. iptables mode's random selection sends the same share to a pod handling a 2-second request as to one handling a 5ms request; least connections tracks in-flight work and routes around it.

The costs, and they are real:

IPVS still uses iptables for some things. Masquerading, NodePort handling and NetworkPolicy still need iptables rules, so the rule count is lower rather than zero, and you now operate two subsystems.

Behavioural differences during pod termination. IPVS keeps existing connections to a removed real server until they close (a "graceful" property that is usually right), and its connection reuse behaviour with conn_reuse_mode had known issues on some kernels where a new connection from the same source port could be sent to a deleted endpoint.

eBPF (replacing kube-proxy)

Cilium's kube-proxy replacement attaches eBPF programs and does the translation before conntrack is involved:

Socket-level (bpf_sock):
    connect() to a ClusterIP is rewritten to a pod IP AT THE SOCKET,
    before a packet exists. No NAT, no conntrack entry, no reverse
    translation on the return path.

TC-level (bpf_lxc / bpf_netdev):
    for traffic that must be forwarded, translation in an eBPF hash map
    at the TC hook, still avoiding much of the netfilter path.

Socket-level load balancing is the significant one. For a pod connecting to a ClusterIP, there is no packet-level NAT at all: the destination is rewritten in the socket before the connection is established. No conntrack entry is created, which removes the table-full class of failure entirely for that path, and the return traffic needs no reverse translation.

Measured, roughly, 10,000 Services:
                       iptables    IPVS       eBPF
first-packet latency   ~450 us     ~30 us     ~20 us
rule sync time         ~8 s        ~1 s       ~0.1 s
conntrack entries      1 per conn  1 per conn 0 (socket LB path)
CPU in softirq         high        moderate   low

The cost is that you are adopting a CNI as your Service implementation, which is a larger commitment than a kube-proxy flag, and it requires a reasonably modern kernel (5.4+ for the useful feature set, and more for some).

Where the modes actually differ operationally

                       iptables      IPVS          eBPF
Services before pain   ~1,000        ~10,000+      ~10,000+
Endpoint sync at 5k
  services              seconds       sub-second    sub-second
Scheduling algorithms  random only   8 algorithms  configurable (Maglev, random)
Conntrack per conn     yes           yes           no (socket LB)
Debuggability          iptables-save ipvsadm       bpftool, cilium monitor
                       (familiar)    (familiar)    (a new toolchain)
Extra components       none          none          a CNI

Debuggability is the under-weighted axis. Every engineer can read iptables-save; far fewer can read bpftool map dump. Adopting eBPF trades a scaling problem for a skills problem, and that is a real cost on a team that is on call.

A worked example: connection errors on every deploy

A platform with about 3,200 Services and 41,000 endpoints across 180 nodes, iptables mode, Kubernetes 1.24.

Symptoms:

during any rolling deploy:
  connection refused / connection reset:  ~0.4% of requests for ~40 seconds
  affected services:                      ALL of them, not just the one deploying
  application logs:                       nothing on the server side
node CPU in softirq:                      18% mean, 40% p99
p99 latency (steady state):               84ms

"All services affected when one deploys" is the tell. A deploy of service A should not produce errors calling service B, and it means the shared mechanism (the iptables table) is the coupling.

Measurement:

$ iptables-save -t nat | wc -l
189,412

$ time iptables-restore --test < /tmp/rules.txt
real    0m11.284s

# kube-proxy's own metrics:
kubeproxy_sync_proxy_rules_duration_seconds{quantile="0.99"}  12.8
kubeproxy_sync_proxy_rules_last_timestamp_seconds             (lagging)

Twelve seconds to write the table, and kube-proxy serialises syncs, so a burst of endpoint changes queues:

rolling deploy of a 40-replica service:
  endpoint changes:        80 (40 removals, 40 additions)
  syncs required:          batched, but each takes ~12s
  total propagation time:  ~40 s

For 40 seconds, terminating pods were still in the table and receiving traffic, and newly-ready pods were not yet in it. Both directions produce errors, and every Service shares the table so every Service's rules are rewritten on every sync.

Fix 1: upgrade to 1.26+ for partial sync.

iptables sync p99:     12.8 s -> 1.9 s
error rate on deploy:  0.4% -> 0.12%
propagation time:      ~40 s -> ~6 s

A 6.7x improvement from a version upgrade, because incremental updates write only changed chains instead of the whole table. That fixed the update dimension and left the packet-path dimension.

Fix 2: terminationGracePeriodSeconds and a preStop hook, which is the part people skip and which is independent of the proxy mode:

spec:
  terminationGracePeriodSeconds: 45
  containers:
  - lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "sleep 10"]     # keep serving while
                                                 # endpoints propagate

The pod must keep serving after it is removed from EndpointSlice, because rule propagation is not instant on any proxy mode. Ten seconds of preStop sleep covers the propagation window, and without it there is always a race regardless of how fast the sync is.

error rate on deploy:  0.12% -> 0.01%

That change was worth more than the version upgrade, and it is free.

Fix 3: IPVS, evaluated on a canary node pool.

                       iptables (1.26)   IPVS
sync p99                1.9 s             0.28 s
softirq CPU (mean)      18%               6%
p99 latency             84 ms             61 ms
rule count              189,412           ~14,000 (iptables) + IPVS table

A 27 percent p99 improvement in steady state, from removing the O(n) chain walk on every first packet.

They also enabled least-connections, which mattered for one specific service:

report-generation service, request cost 50ms to 8s:
  rr (round robin):     p99 4.2 s, and pods with a long request queued behind it
  lc (least conn):      p99 1.8 s

Fix 4: eBPF, evaluated and deferred.

                       IPVS       Cilium eBPF
sync p99                0.28 s     0.04 s
softirq CPU             6%         2%
p99 latency             61 ms      54 ms
conntrack entries       ~180k/node ~11k/node

Better on every technical axis, and they deferred it:

reasons for deferring:
  - replacing the CNI on a live 180-node cluster is a much larger change
    than a kube-proxy mode flag
  - the on-call team could all read iptables-save and ipvsadm; nobody
    could read bpftool output
  - the IPVS numbers were already comfortably within requirements
decision: revisit when Service count exceeds ~8,000 or conntrack becomes
          a constraint again

Final:

                              before      after
kube-proxy mode               iptables    IPVS
Kubernetes version            1.24        1.26
sync p99                      12.8 s      0.28 s
propagation on deploy         ~40 s       ~1 s
error rate during deploys     0.4%        0.008%
softirq CPU (mean)            18%         6%
p99 latency                   84 ms       61 ms

The error rate improved 50x and the largest single contributor was a preStop sleep, not the proxy mode. That is the finding worth carrying: rule propagation is asynchronous in every mode, so the pod must outlive its removal from EndpointSlice. Teams reach for the proxy mode and skip the ten seconds of grace that removes the race entirely.

Production evidence

IPVS mode has been GA since Kubernetes 1.11, and the Kubernetes documentation recommends it above roughly 1,000 Services on the grounds of both lookup and sync cost.

Partial iptables sync landed in 1.26 (KEP-3453, minimising iptables-restore writes), and the KEP's motivation is precisely the full-table-rewrite cost: the measured improvement on large clusters was an order of magnitude on sync time.

Cilium's kube-proxy replacement is used at scale by several large platforms, and their published benchmarks show the socket-level load-balancing path avoiding conntrack entirely. The conntrack avoidance is often the operational motivation rather than the latency, on clusters where table exhaustion was a recurring incident.

Google's Maglev (the consistent-hashing scheme, NSDI 2016) is available as a Cilium load-balancing algorithm and is what you want when connection stability across endpoint changes matters, because it minimises reassignment when the endpoint set changes.

AWS's VPC CNI, GKE Dataplane V2 (Cilium-based) and AKS's Azure CNI Powered by Cilium are all moving the default toward eBPF datapaths, which is the clearest signal about the direction, and iptables mode remains the default in upstream kube-proxy.

The preStop and terminationGracePeriodSeconds recommendation appears in Kubernetes' own documentation on pod termination, and it is there because endpoint propagation is asynchronous by design: the kubelet begins termination and the EndpointSlice update propagate concurrently, with no ordering guarantee.

The debate

When should you leave iptables mode? The Kubernetes guidance of around 1,000 Services is reasonable for the packet-path cost, and the update cost bites earlier on clusters with high endpoint churn. My rule: measure kubeproxy_sync_proxy_rules_duration_seconds at p99, and if it exceeds about one second, the mode is now a deploy-reliability problem rather than a performance one, because propagation lag produces connection errors.

IPVS or eBPF? IPVS is a kube-proxy flag; eBPF is a CNI replacement. That asymmetry decides most cases. IPVS gets you O(1) lookup, real scheduling algorithms and a familiar debugging story for the cost of a flag and a node restart. eBPF gets you better numbers on every axis plus conntrack avoidance, at the cost of adopting a CNI and a toolchain your on-call rotation may not know. Choose eBPF when conntrack exhaustion or Service count is a live constraint, or when you are building a cluster rather than changing one.

Is the debuggability argument real or conservatism? Real, and it is under-weighted in most comparisons. During an incident, the question "is this Service programmed correctly on this node" is iptables-save | grep or ipvsadm -Ln for two modes and cilium service list plus bpftool map dump for the third. A team that cannot inspect its own datapath under pressure has traded a scaling problem for an incident-response problem, and the mitigation is training rather than avoidance, but it should be budgeted.

Does sessionAffinity work the same across modes? ClientIP affinity is implemented with a conntrack-based recent-source-IP match in iptables mode and with IPVS's own persistence in IPVS mode, and their timeout semantics differ subtly. For anything that genuinely needs affinity, source hashing (sh) in IPVS or Maglev in eBPF is more predictable than sessionAffinity: ClientIP, and consistent hashing additionally survives endpoint changes without reshuffling everyone.

What actually causes deploy errors, the proxy mode or the pod lifecycle? Usually the pod lifecycle. Endpoint removal and container termination are concurrent with no ordering guarantee, so a pod that stops accepting connections the moment it receives SIGTERM will refuse traffic that is still being routed to it. A preStop sleep covering the propagation window fixes that in every mode, and it was worth more than the proxy change in the worked example. Reaching for IPVS before fixing the grace period is optimising the wrong thing.

Follow-up Q&A

"Why does iptables mode degrade at scale?"

Two dimensions. Rules are evaluated sequentially, so packet-path cost is O(number of Services): the last Service's rules are checked after all the others. And iptables has no partial-update primitive, so historically every endpoint change rewrote the entire table, making sync time O(total rules). At around 190,000 rules that was 12 seconds per sync, and since syncs serialise, a 40-replica rolling deploy took roughly 40 seconds to propagate. Partial sync in Kubernetes 1.26 fixed the update dimension; the O(n) packet path is inherent to sequential evaluation.

"Is kube-proxy on the data path?"

No. It is a control-plane component that programs the kernel (iptables rules, IPVS tables) and then gets out of the way; packets are handled by the kernel. So kube-proxy being slow does not make requests slow, it makes endpoint changes take longer to apply. That distinction matters because the symptom is connection errors during deploys rather than latency, and it is easy to misattribute.

"What does IPVS buy you?"

O(1) lookup from a kernel hash table instead of a linear chain walk, sync in sub-second time instead of seconds, and real scheduling algorithms. Least-connections is the practical one: iptables mode picks randomly, so a pod handling an 8-second request receives the same share as one handling 50ms. On one service with highly variable request cost, moving from round robin to least connections took p99 from 4.2 seconds to 1.8. The cost is that IPVS still needs iptables for masquerading and NetworkPolicy, so you operate both.

"What does the eBPF datapath do differently?"

Socket-level load balancing: a connect() to a ClusterIP is rewritten to a pod IP in the socket, before a packet exists. There is no NAT, no conntrack entry and no reverse translation on the return path, which removes the conntrack-exhaustion class of failure for that traffic entirely. That is frequently the operational motivation rather than the latency, on clusters where table-full events were recurring.

"Connection errors during every deploy. Where do you look?"

First at whether it affects services other than the one deploying, because that implicates the shared mechanism. Then kubeproxy_sync_proxy_rules_duration_seconds at p99: above a second means propagation lag. But the fix I would apply first is a preStop sleep and an adequate terminationGracePeriodSeconds, because endpoint removal and container termination are concurrent with no ordering guarantee, so a pod that stops accepting connections on SIGTERM refuses traffic still being routed to it. In one case that was worth more than the proxy mode change.

"Would you adopt an eBPF datapath?"

For a new cluster, likely yes. For a live one, only if Service count or conntrack is a real constraint, because it is a CNI replacement rather than a flag, and because the debugging story changes: everyone can read iptables-save, far fewer can read bpftool map dump. In one evaluation eBPF was better on every technical axis and was deferred, on the reasoning that IPVS already met the requirement and the on-call team could inspect it under pressure. That skills cost is real and should be budgeted rather than dismissed.

Common misconceptions

"kube-proxy proxies traffic." Only in the removed userspace mode. It programs kernel rules and packets never enter a userspace process, which is why its performance affects propagation rather than request latency.

"iptables mode is fine, we only have a few hundred Services." Rule count scales with endpoints, not Services, so a few hundred Services with many replicas each can still be tens of thousands of rules, and endpoint churn drives sync frequency.

"IPVS removes iptables." It still uses iptables for masquerading, NodePort handling and NetworkPolicy. The rule count drops substantially and does not reach zero, and you now operate two subsystems.

"The proxy mode causes deploy errors." Propagation lag contributes, and the usual cause is that endpoint removal and container termination are concurrent, so a pod refusing connections on SIGTERM drops traffic still in flight. A preStop sleep fixes that in every mode.

"iptables load balancing is round robin." It is per-connection random via conditional probability rules, with no state, no least-connections and no awareness of endpoint load.

Interview delivery note

Say this verbatim: "kube-proxy is not on the data path; it programs the kernel and gets out of the way, so when it is slow you get connection errors during deploys rather than latency. iptables degrades in two dimensions: O(n) evaluation on the packet path, and historically a full table rewrite per endpoint change, which was 12 seconds at 190,000 rules." The correction of the common misunderstanding plus the two-dimension framing.

The senior-versus-staff separator is fixing the pod lifecycle before the proxy mode. A senior engineer sees connection errors during deploys, measures sync duration and moves to IPVS. A staff engineer notices that endpoint removal and container termination are concurrent with no ordering guarantee in any mode, adds a preStop sleep covering the propagation window, and finds it is worth more than the proxy change. The race exists regardless of how fast the sync is, so removing the race beats making the sync faster.

The second signal is weighing debuggability honestly. Saying "eBPF was better on every technical axis and we deferred it, because IPVS already met the requirement and the on-call team could read ipvsadm but not bpftool" shows you are optimising for incident response rather than for benchmarks, and that the skills cost is a real line item.

The Staff & Team Lead Interview Reference