TCP: TIME_WAIT, accept queues, Nagle, BBR, conntrack exhaustion

What it is

Five TCP-adjacent failures that present as application problems and are not. Each has a distinctive signature, and recognising the signature is most of the value:

Symptom	Actual cause	The counter that proves it
"Cannot assign requested address"	Ephemeral port exhaustion from `TIME_WAIT`	`ss -s`, `net.ipv4.ip_local_port_range`
Connections dropped under load, no error	Accept queue overflow	`ListenOverflows` in `netstat -s`
Exactly 40ms latency on small writes	Nagle interacting with delayed ACK	Consistent 40ms, never 39 or 41
Poor throughput on a fast, lossy link	Loss-based congestion control	`ss -ti` showing cwnd collapse
Random connection failures at scale	`nf_conntrack` table full	`nf_conntrack_count` vs `_max`, dmesg

What these are confused with: application bugs. Every one of them produces a symptom that looks like the service is broken, and every one is diagnosable from a counter in netstat -s or /proc in under a minute. The single most useful habit here is reading netstat -s before reading application logs, because it either implicates the network stack immediately or rules it out.

The problem it solves

Understanding these prevents a specific and expensive category of mistake: tuning the wrong layer. A service dropping connections under load gets more replicas, and the accept queue on each replica is still 128, so the problem scales with it. A service with 40ms latency gets its code profiled for weeks, and the 40ms is a kernel timer.

The economic version: these are all configuration, and the fixes are minutes of work. A team that does not recognise them spends engineer-weeks on the application instead.

Mechanics

TIME_WAIT and ephemeral port exhaustion

When a connection closes, the side that closes first holds the socket in TIME_WAIT for 2 x MSL, which on Linux is a fixed 60 seconds:

Why it exists:
  1. Absorb delayed duplicate segments so they cannot be delivered to a
     NEW connection with the same 4-tuple.
  2. Ensure the final ACK can be retransmitted if lost.

The failure arithmetic:

Ephemeral port range (default):   32768-60999  = 28,232 ports
TIME_WAIT duration:               60 seconds
Max NEW connection rate to ONE destination (ip:port):
                                  28,232 / 60 = 470 connections/sec

Four hundred and seventy connections per second to a single destination, and past that you get EADDRNOTAVAIL: "Cannot assign requested address." A service making short-lived connections to one upstream hits this at a rate most people would consider modest.

$ ss -s
TCP:   48291 (estab 412, closed 47102, orphaned 0, timewait 47098)
                                                   ^^^^^^^^^^^^^^

$ sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768 60999

The fixes, in order of preference:

# 1. BEST: reuse connections. This is an application fix and it is the real answer.
#    Go: MaxIdleConnsPerHost (defaults to 2)
#    JVM: a pooled HTTP client
#    curl in scripts: --keepalive, or stop making a connection per request

# 2. Widen the range.
sysctl -w net.ipv4.ip_local_port_range="1024 65535"     # ~64,500 ports -> ~1,075/sec

# 3. Allow reuse of TIME_WAIT sockets for OUTBOUND connections.
sysctl -w net.ipv4.tcp_tw_reuse=1      # safe: requires timestamps, outbound only

net.ipv4.tcp_tw_recycle does not exist any more and you should not look for it. It was removed in kernel 4.12 because it broke connections from clients behind NAT: it dropped SYNs whose timestamps appeared to go backwards, and behind a NAT gateway different clients have unrelated timestamp clocks. A great deal of blog advice still recommends it, which is worth knowing so you can reject it.

SO_REUSEADDR is not the fix either. It allows binding to a port in TIME_WAIT, which helps a server restart quickly; it does nothing for outbound ephemeral port exhaustion.

Accept queues

Two queues, and conflating them is the usual confusion:

client SYN ──▶ ┌──────────────┐  SYN-ACK  ──▶
               │  SYN QUEUE   │             (half-open)
               │  (incomplete)│  ◀── ACK
               └──────┬───────┘
                      ▼  handshake complete
               ┌──────────────┐
               │ ACCEPT QUEUE │   <- waits here for accept()
               │  (complete)  │
               └──────┬───────┘
                      ▼
                 accept() by the application

SYN queue size:     net.ipv4.tcp_max_syn_backlog        (default 128-1024)
Accept queue size:  min(listen(fd, BACKLOG), net.core.somaxconn)
                                              ^^^^^^^^^^^^^^^^^
                                              default 4096 since kernel 5.4,
                                              128 before that

The accept queue is min of the application's listen() backlog and somaxconn, so raising the sysctl alone does nothing if the application passes 128, and raising the application's value alone does nothing if somaxconn is lower. Both must change, and that is the detail that makes this a recurring problem.

$ ss -lnt
State   Recv-Q  Send-Q  Local Address:Port
LISTEN  129     128     0.0.0.0:8080
        ^^^     ^^^
        current backlog size
                accept queue MAXIMUM

$ netstat -s | grep -i listen
    4821 times the listen queue of a socket overflowed
    4821 SYNs to LISTEN sockets dropped

ListenOverflows climbing is definitive. The connection was accepted at the TCP level, the handshake completed, and the application never called accept() fast enough, so the kernel dropped it. The client sees a timeout or a reset with no error anywhere on the server, which is why this looks like a network problem.

# What the kernel does on overflow:
net.ipv4.tcp_abort_on_overflow = 0    # DEFAULT: silently drop, client retries
net.ipv4.tcp_abort_on_overflow = 1    # send RST: client fails fast and clearly

Setting it to 1 during an investigation converts a mysterious timeout into an immediate connection reset, which is far easier to attribute. It is a diagnostic setting rather than a production one.

Common defaults that cause this:

Node.js:  server.listen(port)          -> backlog 511
Python:   socket.listen()              -> backlog 128 (or 0 in some versions!)
Java:     new ServerSocket(port)       -> backlog 50
nginx:    listen 80;                   -> backlog 511
          listen 80 backlog=4096;      -> explicit

Java's default of 50 is the one that surprises people.

Nagle and delayed ACK: the 40ms signature

Nagle's algorithm (RFC 896) buffers small writes: do not send a small segment while a previously-sent small segment is unacknowledged. It prevents a telnet session sending a 41-byte packet per keystroke.

Delayed ACK (RFC 1122) waits up to 40ms (Linux) before acknowledging, hoping to piggyback the ACK on a response.

Together they deadlock:

t=0     app writes 100 bytes.  Nagle sends it (nothing outstanding).
t=0     app writes 50 bytes.   Nagle HOLDS it (100 bytes unacked).
t=0     receiver gets 100 bytes. Delayed ACK: waits for a response to piggyback on.
        But the response needs the 50 bytes Nagle is holding.
t=40ms  delayed ACK timer fires. ACK sent.
t=40ms  Nagle releases the 50 bytes.

Exactly 40 milliseconds, reproducibly, on a request that should take microseconds. The tell is the consistency: a p50 of 40ms with almost no variance is a timer, not work.

int one = 1;
setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &one, sizeof one);   /* disable Nagle */

Go:      TCPConn.SetNoDelay(true)      // Go sets TCP_NODELAY by DEFAULT
Java:    socket.setTcpNoDelay(true)    // Netty sets it by default
Node.js: socket.setNoDelay(true)
Python:  sock.setsockopt(IPPROTO_TCP, TCP_NODELAY, 1)

Almost every modern framework sets TCP_NODELAY by default, which is why this is rarer than it was. It still appears in hand-rolled protocol clients, in database drivers, and in anything that does a small write followed by a read.

The better fix, where you control the protocol, is to write once. Nagle exists because applications write in pieces; a single writev of the header and the body never triggers it.

Congestion control: CUBIC and BBR

CUBIC (the Linux default) is loss-based: grow the window until a packet is lost, then back off. That works when loss means congestion, and on modern networks loss frequently means a lossy wireless link or a policer, not a full buffer.

BBR models the path instead: estimate bottleneck bandwidth and round-trip propagation time, and pace sending to that estimate. It does not interpret loss as congestion.

Long-haul link, 100ms RTT, 1% random loss:
  CUBIC:  ~1.2 Mb/s      (the loss keeps collapsing the window)
  BBR:    ~180 Mb/s

Congested link with deep buffers (bufferbloat):
  CUBIC:  fills the buffer, adding hundreds of ms of queueing delay
  BBR:    keeps the queue short, much lower latency at similar throughput

sysctl -w net.ipv4.tcp_congestion_control=bbr
sysctl -w net.core.default_qdisc=fq          # BBR requires fair queueing for pacing

The fq qdisc is not optional: BBR paces packets and needs a qdisc that can pace, and running BBR without it gives most of the complexity and little of the benefit.

The honest criticism of BBR: BBRv1 was documented to be unfair to CUBIC flows sharing a bottleneck, taking a disproportionate share. BBRv2 and v3 address this and are what modern kernels ship. For a link you control end to end (a CDN to its origin, a data-centre interconnect), BBR is close to unambiguously better. On a shared path it is a fairness question.

conntrack exhaustion

nf_conntrack tracks every connection through a NAT or stateful-firewall path, and in Kubernetes that is every connection through kube-proxy in iptables mode.

$ sysctl net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_max = 262144

$ cat /proc/sys/net/netfilter/nf_conntrack_count
261847                                      # <- at 99.8% of the table

$ dmesg | tail
nf_conntrack: table full, dropping packet

"nf_conntrack: table full, dropping packet" is unambiguous, and the resulting failures look entirely random: some connections work, some are silently dropped, retries usually succeed. It presents as flakiness affecting no particular service.

# The table is sized from RAM by default; raise it and the hash bucket count together.
sysctl -w net.netfilter.nf_conntrack_max=1048576
echo 262144 > /sys/module/nf_conntrack/parameters/hashsize     # max/4 is the convention

# The bigger lever is usually the timeouts, not the size.
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=3600   # default 432000 (5 DAYS)
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30       # default 120

The established timeout defaults to five days. A connection that closed uncleanly holds a conntrack entry for five days, and on a node making millions of short-lived connections that is how the table fills. Lowering it to an hour is the highest-value change and is safe for anything that is not deliberately holding idle connections for longer.

Each entry is roughly 300 bytes, so a million-entry table is about 300 MB of kernel memory, which is the constraint on simply making it enormous.

A worked example: 0.3 percent of requests failing, four causes

An API gateway fronting about 200 internal services. Roughly 40,000 requests per second across 24 nodes. A persistent 0.3 percent failure rate that had resisted investigation for months.

Symptoms:

error rate:               0.31% (mixture of timeouts and connection resets)
pattern:                  no correlation with service, endpoint, time of day,
                          or node
application logs:         nothing; the failures happen before the handler
p99 latency:              340ms (p50 12ms)

Nothing in the application logs is the signal. Failures occurring before any handler runs are connection-level.

Step 1: netstat -s, which took a minute.

$ netstat -s | grep -Ei 'overflow|listen|prune|timewait|retrans'
    18492 times the listen queue of a socket overflowed
    18492 SYNs to LISTEN sockets dropped
    2841 packets pruned from receive queue because of socket buffer overrun
    412984 segments retransmitted

$ dmesg -T | grep conntrack | tail -3
[Tue Aug  4 09:12:44] nf_conntrack: table full, dropping packet
[Tue Aug  4 11:41:02] nf_conntrack: table full, dropping packet

$ ss -s
TCP: 184291 (estab 8412, closed 172104, timewait 171998)

Three problems visible in two commands: accept queue overflow, conntrack table full, and 172,000 sockets in TIME_WAIT.

Cause 1: accept queue.

$ ss -lnt | grep 8443
LISTEN  129   128   0.0.0.0:8443

The gateway was Java, using the default ServerSocket backlog of 50, raised to 128 by someone at some point, against somaxconn of 4096.

// Before: the framework's default.
// After:
serverBootstrap.option(ChannelOption.SO_BACKLOG, 4096);

ListenOverflows:      18,492 -> 0
error rate:           0.31% -> 0.19%

Cause 2: conntrack.

$ sysctl net.netfilter.nf_conntrack_max
262144
$ cat /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established
432000                                       # five days

# Applied via a privileged DaemonSet, per node.
sysctl -w net.netfilter.nf_conntrack_max=1048576
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=3600
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30

nf_conntrack_count:   261,847 -> 84,102 (steady)
table full events:    ~6/day -> 0
error rate:           0.19% -> 0.08%

The timeout change did more than the size increase. Five days of retention on a node handling millions of short connections is what filled a 262,144-entry table.

Cause 3: TIME_WAIT and connection reuse.

172,000 sockets in TIME_WAIT on a node with 28,232 ephemeral ports
-> connections to 200 different upstream services, so no single
   destination had exhausted its range, but the node was close to
   the global socket limit

// The gateway used a new connection per upstream request.
// Apache HttpClient, connection pool per route:
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
cm.setMaxTotal(2000);
cm.setDefaultMaxPerRoute(50);           // was the default of 2

TIME_WAIT sockets:    172,000 -> 9,400
outbound connections
  established:        412 -> 3,800 (pooled and reused)
p99 latency:          340ms -> 91ms
error rate:           0.08% -> 0.02%

Connection pooling was the largest single improvement, and it fixed latency as well as errors, because every request had been paying a handshake. Note the default of 2 connections per route, which is the same class of default as Go's MaxIdleConnsPerHost.

Cause 4: the residual 0.02 percent, which was BBR-shaped.

$ ss -ti dst 10.42.0.0/16 | grep -A1 cubic | head -4
cubic wscale:7,7 rto:204 rtt:1.2/0.4 cwnd:10 ssthresh:7 bytes_sent:...
                                     ^^^^^^^ collapsed to the initial window

Cross-AZ traffic showed periodic congestion-window collapse. The links had low but non-zero loss from a policer, and CUBIC was interpreting it as congestion.

sysctl -w net.core.default_qdisc=fq
sysctl -w net.ipv4.tcp_congestion_control=bbr

cross-AZ p99:         91ms -> 64ms
retransmits:          412,984 -> 88,201
error rate:           0.02% -> 0.004%

Final:

                        before      after
error rate              0.31%       0.004%     (78x reduction)
p99 latency             340ms       58ms
ListenOverflows         18,492      0
conntrack table full    ~6/day      0
TIME_WAIT sockets       172,000     9,400
segments retransmitted  412,984     88,201
time to diagnose        4 months    ~1 hour once netstat -s was read

Four independent causes, all in the network stack, all visible in netstat -s, ss -s and dmesg in about a minute. Four months of application-level investigation had found nothing, because there was nothing in the application to find.

The transferable practice: read netstat -s before the application logs. It either implicates the network stack immediately (overflows, pruning, retransmits) or rules it out, and it costs one command. In this case it named three of the four causes on the first run.

Production evidence

net.core.somaxconn was raised from 128 to 4096 in kernel 5.4, an acknowledgement that the old default was too low for modern servers. Systems on older kernels or with older distributions still ship 128, and it is a common finding.

tcp_tw_recycle was removed in kernel 4.12 because it broke clients behind NAT. Published tuning advice recommending it predates the removal and is actively harmful, which makes it a useful thing to be able to reject in an interview.

BBR was developed at Google and deployed on google.com and YouTube, with published results showing large throughput improvements on lossy long-haul paths and reduced queueing delay. It is available in Linux from kernel 4.9 and is used by Cloudflare, Dropbox and Spotify among others. BBRv1's fairness against CUBIC was a documented and legitimate criticism, addressed in v2 and v3.

Conntrack exhaustion is a well-known Kubernetes failure, and the default nf_conntrack_tcp_timeout_established of 432000 seconds is documented as a source of table growth on high-churn nodes. Cilium's eBPF datapath avoids conntrack entirely for many paths, which is one of its stated advantages over iptables-based kube-proxy.

The Nagle plus delayed-ACK interaction is described in RFC 896 and RFC 1122 and was analysed by John Nagle himself, who has publicly said the interaction with delayed ACK was the mistake and that delayed ACK is the part he would remove.

The debate

Should you tune these, or fix the application? Fix the application where you can, and recognise the kernel-level fixes as either mitigations or genuine configuration errors. Connection pooling is the real answer to TIME_WAIT exhaustion; widening the port range buys time. A listen() backlog of 4096 is not a mitigation, it is the correct value, and the default of 50 or 128 is simply wrong for a server.

Is BBR safe to enable everywhere? For paths you control end to end, yes, and it is a clear win on lossy or long-haul links. On a shared bottleneck with CUBIC flows, BBRv1's fairness problem was real; v2 and v3 largely resolve it. My position: enable it for cross-region and internet-facing egress where loss is not congestion, verify the kernel has v2 or later, and always set fq as the qdisc, because BBR without pacing is not BBR.

Should you raise conntrack limits or avoid conntrack? Both, in that order. Raising the table and, more importantly, lowering the established timeout from five days to an hour is minutes of work and fixes the immediate problem. Avoiding conntrack is the structural answer: Cilium's eBPF datapath bypasses it for most traffic, and IPVS mode in kube-proxy uses it less than iptables mode. If you are hitting conntrack limits regularly, the datapath is the conversation.

Is TCP_NODELAY always right? For request-response protocols, yes, and every modern framework sets it. The case for Nagle is a protocol that genuinely writes in many small pieces and does not need low latency, which describes almost nothing built today. The better fix where you own the protocol is to write once with writev rather than to disable Nagle, because a single write never triggers it and you keep the protection for whatever else shares the socket.

What should you check first? netstat -s, always, before application logs, for any symptom involving connections, timeouts or unexplained latency. It is one command, it names overflows, pruning, retransmits and the TIME_WAIT population, and it either implicates the stack or eliminates it. Four months of investigation in the worked example was resolved in an hour once someone ran it.

Follow-up Q&A

"You see 'Cannot assign requested address' under load. Explain."

Ephemeral port exhaustion. The client-side ephemeral range is about 28,000 ports by default and TIME_WAIT holds each for 60 seconds, so the ceiling is roughly 470 new connections per second to a single destination address and port. Beyond that there is no free 4-tuple. The real fix is connection reuse in the application; widening ip_local_port_range and enabling tcp_tw_reuse buy headroom. Not tcp_tw_recycle, which was removed in kernel 4.12 because it broke clients behind NAT.

"What are the two accept queues and how do you know one overflowed?"

The SYN queue holds half-open connections awaiting the final ACK; the accept queue holds completed handshakes waiting for the application to call accept(). Accept queue depth is min(listen() backlog, net.core.somaxconn), so both must be raised. Overflow shows as "times the listen queue of a socket overflowed" in netstat -s, and the connection is dropped silently so the client sees a timeout with nothing on the server. Setting tcp_abort_on_overflow=1 during an investigation turns that into an immediate RST, which is much easier to attribute.

"What causes exactly 40 milliseconds of latency?"

Nagle interacting with delayed ACK. Nagle holds a small write while a previous small segment is unacknowledged; delayed ACK waits up to 40ms hoping to piggyback the ACK on a response that cannot be sent because Nagle is holding it. The signature is the consistency: a p50 of exactly 40ms with almost no variance is a timer rather than work. Fix with TCP_NODELAY, or better, where you own the protocol, write the header and body in a single writev so Nagle never engages.

"When is BBR better than CUBIC?"

When loss does not mean congestion: lossy wireless links, long-haul paths with policers, and anywhere bufferbloat means CUBIC fills a deep buffer and adds hundreds of milliseconds of queueing. CUBIC grows until it loses a packet and backs off; BBR models bottleneck bandwidth and round-trip time and paces to that. On a 100ms path with 1 percent random loss the difference is roughly two orders of magnitude of throughput. BBR requires the fq qdisc for pacing, and BBRv1's unfairness to CUBIC on a shared bottleneck was a legitimate criticism addressed in v2 and v3.

"Random connection failures across a whole node. Where do you look?"

dmesg for "nf_conntrack: table full, dropping packet", and nf_conntrack_count against nf_conntrack_max. Conntrack tracks every connection through NAT or a stateful firewall, which in Kubernetes with iptables kube-proxy is every connection. The default established timeout is 432000 seconds, five days, so uncleanly-closed connections accumulate. Lowering it to an hour usually matters more than raising the table size, and each entry is about 300 bytes so a million-entry table is 300 MB of kernel memory.

"What is the first command you run for a connection-level problem?"

netstat -s, before the application logs. It reports listen queue overflows, receive-queue pruning, retransmits and socket states in one output, so it either implicates the network stack immediately or rules it out. In one case it named three of four independent causes on the first run, after four months of application-level investigation had found nothing, because there was nothing in the application to find.

Common misconceptions

"TIME_WAIT is a leak." It is required for correctness: it absorbs delayed duplicate segments and allows the final ACK to be retransmitted. The problem is not its existence, it is making enough connections that 60 seconds times your rate exceeds the port range.

"Use tcp_tw_recycle." Removed in kernel 4.12 because it dropped SYNs from clients behind NAT whose timestamps appeared to move backwards. Advice recommending it predates the removal.

"Raise somaxconn and the accept queue grows." The queue is the minimum of somaxconn and the application's listen() backlog, so both must change. Java's ServerSocket default is 50.

"Nagle is obsolete and always harmful." It prevents a class of small-packet flooding that still exists. Modern frameworks disable it because request-response protocols suffer from the delayed-ACK interaction, and where you control the protocol, writing once is a better fix than disabling it.

"BBR is strictly better than CUBIC." It is better where loss does not indicate congestion. BBRv1 was documented as unfair to CUBIC flows sharing a bottleneck, and it requires the fq qdisc to pace properly.

Interview delivery note

Say this verbatim: "For anything that looks like a connection problem I read netstat -s before the application logs, because it names listen queue overflows, pruning and retransmits in one command. In one case that found three of four independent causes in a minute, after four months of application investigation had found nothing." A concrete practice with a measured payoff.

The senior-versus-staff separator is recognising timer signatures. A senior engineer knows about Nagle and TCP_NODELAY. A staff engineer sees a p50 of exactly 40 milliseconds with almost no variance and identifies it as a timer rather than work before opening a profiler, the same way an exact 5-second DNS latency is a resolver timeout. Round-number latencies with low variance are always a timer, and knowing whose is most of the diagnosis.

The second signal is knowing that the conntrack established timeout defaults to five days. Raising the table size is the obvious response and lowering the timeout is usually the larger effect, and the difference between those two is whether you understand why the table is filling.

The Staff & Team Lead Interview Reference