The ndots:5 DNS latency classic

What it is

Every pod gets a /etc/resolv.conf written by the kubelet, and by default it looks like this:

search default.svc.cluster.local svc.cluster.local cluster.local ec2.internal
nameserver 10.96.0.10
options ndots:5

ndots:5 means: if a name has fewer than 5 dots, try appending each search domain before trying the name as-is.

Query: api.stripe.com          (2 dots, fewer than 5)

Attempted, in order:
  1. api.stripe.com.default.svc.cluster.local    NXDOMAIN
  2. api.stripe.com.svc.cluster.local            NXDOMAIN
  3. api.stripe.com.cluster.local                NXDOMAIN
  4. api.stripe.com.ec2.internal                 NXDOMAIN
  5. api.stripe.com                              -> 34.98.x.x  finally

Five lookups instead of one, and because glibc issues A and AAAA queries in parallel for each, ten DNS packets to resolve one external hostname.

What it is confused with: a CoreDNS performance problem. CoreDNS is usually fine; it is answering ten times the queries it should. The fix is almost never "scale CoreDNS," and teams that scale it first get a smaller version of the same latency at higher cost.

The reason ndots:5 exists is legitimate: it makes short in-cluster names work. payments resolves via the search path to payments.default.svc.cluster.local, and payments.billing resolves to payments.billing.svc.cluster.local. Without a high ndots, those would fail. The default optimises for in-cluster convenience and taxes every external lookup, and most workloads make more external lookups than the default assumes.

The problem it solves

Understanding this is what stops three wrong diagnoses.

"Our p99 is spiky and we cannot find it in the application." DNS resolution is often not instrumented, so the time appears as unexplained latency before the first byte of a request. An external call that takes 5ms of network time and 42ms of DNS resolution looks like a slow API.

"CoreDNS is the bottleneck, scale it up." At 10x the necessary query volume, CoreDNS saturates at a tenth of the traffic it should handle. Scaling it works, in the sense that paying ten times over works, and the queries were never needed.

"It is intermittent, so it must be the network." The classic symptom is a bimodal latency distribution: most requests fast, some very slow. That is not the search path alone; it is the search path interacting with UDP packet loss and the 5-second resolver timeout, which is covered below and is where the multi-second outliers come from.

The measurable shape:

External hostname resolution, ndots:5, no cache:
  DNS queries per resolution:     10 (5 search attempts x A + AAAA)
  p50 resolution time:            4ms
  p99 resolution time:            38ms
  p99.9:                          5,020ms      <- a retransmit after a lost packet

That p99.9 of just over 5 seconds is the fingerprint. It is the glibc resolver's default timeout of 5 seconds before retrying, and seeing 5-second or 10-second latencies in a service's tail is nearly always DNS.

Mechanics

The search path, and why AAAA doubles it

/* glibc getaddrinfo, simplified */
if (count_dots(name) < ndots)
    for (domain in search)
        if (query(name + "." + domain)) return;
return query(name);          /* the absolute name, tried LAST */

glibc sends A and AAAA queries in parallel over the same socket and waits for both. That is the second multiplier, and it is also the source of a historically nasty failure:

The kernel conntrack race (fixed in kernel 5.1, and present in many long-lived clusters). Two UDP packets sent from the same source port at the same time to the same destination could race in nf_conntrack, and one entry was dropped. The A response returned, the AAAA response was lost, and the resolver waited its full 5-second timeout before retrying. That is where the 5-second tail comes from, and it only manifests under load, which made it look like a capacity problem.

symptom:            occasional exactly-5-second (or 5.000s, 10.000s) DNS resolutions
cause:              conntrack race on parallel A/AAAA from the same socket
kernel fix:         5.1+ (and Alpine/musl was affected differently)
workarounds:        single-request-reopen, or use TCP, or NodeLocal DNSCache

The fixes, ranked

1. A trailing dot: the free one.

requests.get("https://api.stripe.com./v1/charges")
#                                   ^ FQDN: skips the search path entirely

queries: 10 -> 2      (A + AAAA, once)
p99:     38ms -> 4ms

One character, and it works everywhere, because a name ending in a dot is absolute by definition. The problems are that it looks like a typo so reviewers delete it, some HTTP clients and TLS libraries mishandle it in SNI or certificate validation, and it must be applied at every call site.

2. dnsConfig per pod: the right fix for a workload that mostly calls out.

spec:
  dnsConfig:
    options:
    - name: ndots
      value: "2"          # or "1" for a service making only external calls

"api.stripe.com" has 2 dots. With ndots:2, 2 is not < 2, so it is tried
absolute FIRST. One query pair.

"payments" has 0 dots, so the search path still applies. In-cluster short
names keep working.

ndots:2 is the setting that gets almost all of the benefit with almost none of the risk, because in-cluster short names (payments, payments.billing) have 0 or 1 dots and still traverse the search path. The case it breaks is a two-label in-cluster name that you expect the search path to complete, which is unusual.

3. NodeLocal DNSCache: the cluster-wide fix.

A DaemonSet running a caching resolver on every node, with pods pointed at a link-local address:

pod -> 169.254.20.10 (node-local cache, TCP upstream) -> CoreDNS -> upstream

- Cache hits never leave the node: no network, no conntrack entry.
- Cache misses go upstream over TCP, which avoids the UDP conntrack race entirely.
- CoreDNS query volume drops by the cache hit rate, typically 60-85%.

This is the fix that addresses the 5-second tail rather than the average, because TCP upstream removes the conntrack race and local hits remove the network entirely. It does not reduce the search path, so it composes with ndots tuning rather than replacing it.

4. Application-level caching. Most runtimes cache DNS badly by default:

JVM:     networkaddress.cache.ttl defaults to 30s (or FOREVER with a SecurityManager)
Go:      NO caching in the standard resolver. Every dial is a lookup.
Node.js: NO caching. Every request is a lookup.
Python:  NO caching in requests/urllib3.

Go and Node.js resolving on every connection is the reason DNS volume is often far higher than expected. A service making 2,000 outbound requests per second with no connection reuse is making 2,000 DNS resolutions per second, times ten for the search path.

Connection pooling is therefore a DNS fix as well as a TCP one, and it is often the largest single reduction available.

Diagnosis

# 1. What is the pod actually configured with?
kubectl exec -it POD -- cat /etc/resolv.conf

# 2. How many queries does one resolution take?
kubectl exec -it POD -- dig +search +trace api.stripe.com | grep -c "^;; QUESTION"

# 3. Time it, with and without the trailing dot.
kubectl exec -it POD -- sh -c \
  'time nslookup api.stripe.com; time nslookup api.stripe.com.'

# 4. CoreDNS query volume and the NXDOMAIN ratio: the smoking gun.
#    A high NXDOMAIN fraction means the search path is being walked.

sum(rate(coredns_dns_responses_total{rcode="NXDOMAIN"}[5m]))
  / sum(rate(coredns_dns_responses_total[5m]))

An NXDOMAIN ratio above about 0.5 means most of your DNS traffic is search-path failures. That single metric identifies this problem definitively and it is rarely on anyone's dashboard.

A worked example: 47ms of p99 that was not in the code

A payments service. Go, calling three external providers plus two internal services. p99 had been climbing for months.

Symptoms:

p99 latency:                    340ms
p50 latency:                    28ms
application-instrumented time:  p99 190ms
unexplained (p99 total minus instrumented): ~150ms
occasional outliers:            5.02s, 10.03s   <- suspiciously exact

The exact 5.02-second outliers were the tell. Nothing in the application had a 5-second timeout, and 5.000 seconds is the glibc resolver default.

Measurement:

$ kubectl exec -it payments-7d4 -- cat /etc/resolv.conf
search payments.svc.cluster.local svc.cluster.local cluster.local eu-west-1.compute.internal
nameserver 172.20.0.10
options ndots:5

$ kubectl exec -it payments-7d4 -- sh -c 'time nslookup api.provider.com'
real  0m0.047s

$ kubectl exec -it payments-7d4 -- sh -c 'time nslookup api.provider.com.'
real  0m0.004s

47ms versus 4ms, for one character.

# CoreDNS NXDOMAIN ratio, cluster-wide
0.83

Eighty-three percent of all cluster DNS queries were NXDOMAIN, which is the search path being walked and failing four times before succeeding.

And the Go-specific multiplier:

// The service used the default http.Client with no connection reuse tuning.
client := &http.Client{Timeout: 5 * time.Second}

outbound requests/sec:           1,900
connection reuse rate:           11%     (measured: most dials were new)
DNS resolutions/sec:             ~1,690
DNS queries/sec (x10):           ~16,900   from ONE service

Fix 1: connection pooling.

transport := &http.Transport{
    MaxIdleConns:        200,
    MaxIdleConnsPerHost: 50,        // default is 2: the actual problem
    IdleConnTimeout:     90 * time.Second,
}
client := &http.Client{Transport: transport, Timeout: 5 * time.Second}

MaxIdleConnsPerHost defaults to 2 in Go, so a service making 1,900 requests per second to three hosts was constantly opening new connections and resolving DNS for each.

connection reuse rate:    11% -> 94%
DNS resolutions/sec:      1,690 -> 108
p99 latency:              340ms -> 218ms

Fix 2: ndots:2.

spec:
  dnsConfig:
    options:
    - {name: ndots, value: "2"}
    - {name: single-request-reopen}     # A/AAAA on separate sockets

DNS queries per external resolution:  10 -> 2
p99 latency:                          218ms -> 174ms
CoreDNS QPS (cluster):                41,000 -> 9,200

In-cluster lookups were verified unaffected: ledger (0 dots) and ledger.core (1 dot) both still resolved through the search path.

Fix 3: NodeLocal DNSCache, deployed cluster-wide.

CoreDNS QPS:                    9,200 -> 2,100
DNS p99 (measured in-pod):      11ms -> 0.4ms (cache hit)
5-second outliers:              ~40/day -> 0
CoreDNS pods:                   12 -> 4

The 5-second outliers went to zero, because NodeLocal talks to CoreDNS over TCP, which removes the UDP conntrack race that caused them. That was the fix for the tail specifically, and the other two fixed the average.

Final:

                              before      after
p99 latency                   340ms       161ms     (-53%)
p50 latency                   28ms        24ms
5s/10s outliers               ~40/day     0
DNS queries/sec (this svc)    16,900      216
CoreDNS QPS (cluster)         41,000      2,100     (-95%)
CoreDNS pods                  12          4
NXDOMAIN ratio                0.83        0.04
connection reuse              11%         94%

A 53 percent p99 reduction with no change to the service's business logic. The three fixes were a Transport config, a dnsConfig block and a DaemonSet.

The finding worth carrying is the diagnostic sequence. The instrumented time was 190ms of a 340ms p99, and the 150ms gap was invisible because DNS resolution happens before the application's first span. The NXDOMAIN ratio of 0.83 identified the cause in one query, and it is a metric almost nobody collects. The exact-5.000-second outliers identified the conntrack race independently.

Production evidence

Kubernetes' ndots:5 default is documented and the DNS specification for services explains why: it makes single-label and two-label in-cluster names resolve through the search path. The tradeoff is acknowledged in the documentation and the default has not changed, because lowering it would break clusters relying on short names.

NodeLocal DNSCache is a Kubernetes SIG-Network addon, and its documented motivations are exactly the three above: reduced CoreDNS load, lower latency via local caching, and avoiding the UDP conntrack race by using TCP upstream.

The conntrack race was analysed publicly by Weave and others in 2017 to 2019, traced to nf_conntrack handling of parallel UDP inserts, and fixed in kernel 5.1. The single-request and single-request-reopen resolver options exist as userspace workarounds and remain useful on older kernels.

Go's MaxIdleConnsPerHost default of 2 is documented and is a recurring source of this problem, because a high-throughput service with the default http.Client opens connections constantly. Node.js had the same shape until keep-alive became the default agent behaviour in Node 19.

Alpine and musl behave differently. musl's resolver queries all nameservers in parallel rather than sequentially and historically handled search paths and TCP fallback differently, which is why "it works on Debian and not on Alpine" is a recognised class of DNS bug in containers.

The debate

Should the Kubernetes default be lower? The argument for ndots:5 is that short in-cluster names are the ergonomic reason to have a search path at all, and lowering the default would break clusters using two-label names. The argument against is that most workloads make more external calls than the default optimises for, and it taxes every one of them tenfold. My position: the default is defensible for the cluster and wrong for most individual workloads, and dnsConfig per pod is the right place to fix it rather than changing a cluster-wide default that other teams depend on.

Trailing dot or ndots? ndots, because a trailing dot must be applied at every call site, looks like a typo so reviewers remove it, and can break SNI or certificate validation in some clients. ndots:2 is one block in a pod spec, applies to everything the pod does, and keeps short in-cluster names working. Use the trailing dot as a diagnostic (timing with and without it identifies the problem in ten seconds) rather than as a fix.

Is NodeLocal DNSCache worth deploying? For any cluster of meaningful size, yes. It reduces CoreDNS load by the cache hit rate, removes network round trips for hits, and its TCP upstream removes the conntrack race that causes multi-second tails. The costs are a DaemonSet on every node and one more component to operate. The tail-latency fix alone justifies it, because 5-second DNS outliers are otherwise very hard to diagnose and appear as application flakiness.

Should applications cache DNS? Carefully. Go and Node cache nothing by default, which is why volume is high; the JVM caches for 30 seconds by default and historically forever under a SecurityManager, which is why JVM services sometimes hold a dead IP after a failover. Connection pooling is the better lever, because it reduces resolutions without introducing staleness: a reused connection needs no lookup and a new connection gets a fresh one.

Is this still relevant? The conntrack race is fixed in modern kernels, and the search path multiplication is unchanged and unaffected by kernel version. Service meshes change the picture (Istio's sidecar can intercept DNS), and most clusters still run pods with ndots:5 and default HTTP clients. The NXDOMAIN ratio is the check, and it takes one PromQL query to find out whether it applies to you.

Follow-up Q&A

"What does ndots:5 do?"

If a hostname has fewer than 5 dots, the resolver appends each search domain and tries those before trying the name as written. So api.stripe.com becomes five lookups, four of which are NXDOMAIN, and because glibc sends A and AAAA in parallel it is ten DNS packets to resolve one external hostname. The default exists so that short in-cluster names like payments resolve through the search path, and it taxes every external lookup tenfold.

"You see exactly 5.00-second latencies. What is that?"

Almost certainly the glibc resolver's default timeout before retrying a lost DNS query. The classic cause is the nf_conntrack race on parallel A and AAAA queries from the same socket, which drops one response so the resolver waits its full timeout. Fixed in kernel 5.1, and the userspace workarounds are single-request-reopen or moving to TCP, which is what NodeLocal DNSCache does. Exact round-number multi-second outliers are nearly always a timeout rather than real work.

"How do you diagnose this in one query?"

The CoreDNS NXDOMAIN ratio: NXDOMAIN responses over total responses. Above about 0.5 means most cluster DNS traffic is search-path failures. In one case it was 0.83. That single metric identifies the problem definitively and is rarely on anyone's dashboard. The ten-second confirmation is timing nslookup name against nslookup name. inside the pod.

"What is the right fix?"

dnsConfig with ndots:2 on pods that mostly call outward, because in-cluster short names have 0 or 1 dots and still traverse the search path, so nothing breaks. NodeLocal DNSCache cluster-wide, which cuts CoreDNS load by the hit rate and removes the multi-second tail by using TCP upstream. And connection pooling in the application, because Go's MaxIdleConnsPerHost defaults to 2 and Go caches no DNS, so a high-throughput service resolves on nearly every request.

"Why is connection pooling a DNS fix?"

Because Go and Node.js do not cache DNS at all, so every new connection is a fresh resolution. A service making 1,900 requests a second with 11 percent connection reuse was doing about 1,690 resolutions a second, times ten for the search path. Raising MaxIdleConnsPerHost from its default of 2 took reuse to 94 percent and resolutions to 108 a second, which was the largest single reduction available and it required no DNS change at all.

"Why not just scale CoreDNS?"

Because it is answering ten times the queries it should. Scaling works in the sense that paying ten times over works, and the queries were never needed. In one case fixing the search path and adding NodeLocal took cluster DNS from 41,000 to 2,100 QPS and CoreDNS from 12 pods to 4. Scaling first is treating the symptom and it leaves the latency, because the extra nine round trips remain.

Common misconceptions

"DNS is fast, it is not the problem." With ndots:5 and no caching, one external resolution is ten queries and can be 40ms at p99 and 5 seconds at p99.9. It also usually sits outside application instrumentation, so it appears as unexplained latency.

"CoreDNS needs scaling." It is usually answering ten times the necessary volume. Fix the query count first; the NXDOMAIN ratio tells you whether that is the case.

"Set ndots:1 for everything." That breaks short in-cluster names, which is what the search path is for. ndots:2 keeps payments and payments.billing working while making any two-dot external name resolve directly.

"A trailing dot is the fix." It is the diagnostic. As a fix it must be applied at every call site, looks like a typo so it gets removed, and breaks SNI in some clients.

"Applications cache DNS." Go and Node.js do not, at all. The JVM caches for 30 seconds by default and historically forever under a SecurityManager, which causes the opposite problem of holding a dead IP after failover.

Interview delivery note

Say this verbatim: "With ndots:5, an external hostname with two dots is five lookups, and glibc sends A and AAAA in parallel, so it is ten DNS packets to resolve one name. The one-query diagnostic is the CoreDNS NXDOMAIN ratio: above 0.5 means most of your DNS traffic is search-path failures, and in one case it was 0.83." The mechanism, the multiplier, and a diagnostic that is faster than anything else available.

The senior-versus-staff separator is recognising exact multi-second outliers as a timeout signature. A senior engineer knows about ndots and fixes the average. A staff engineer sees 5.02-second and 10.03-second outliers, recognises 5.000 seconds as the glibc resolver default rather than real work, traces it to the UDP conntrack race on parallel A/AAAA, and knows that NodeLocal DNSCache fixes it by going upstream over TCP. Round-number multi-second latencies are a timeout, and identifying whose timeout it is narrows the search enormously.

The second signal is connecting connection pooling to DNS. Saying "Go's MaxIdleConnsPerHost defaults to 2 and Go caches no DNS, so low connection reuse means a resolution per request" shows you understand where the volume comes from, and it was the largest single reduction in the worked example without touching DNS configuration at all.

The Staff & Team Lead Interview Reference