Memory: page cache, faults, huge pages, NUMA, the OOM killer

What it is

Five mechanisms that together decide how much memory your process appears to use, how fast it can reach that memory, and who gets killed when there is not enough.

Page cache is the kernel's cache of file contents in RAM. Reads are served from it and writes land in it and are flushed later. It is counted as "used" memory and is reclaimable on demand, which is why "free memory" on a healthy Linux box is close to zero and that is correct.

Page faults are what happen when a process touches an address that is not currently mapped to physical memory. Minor faults are resolved from memory already present (page cache, a shared library another process loaded, copy-on-write); major faults require disk I/O. The ratio between them is a diagnostic worth more than most memory metrics.

Huge pages replace 4 KB page-table entries with 2 MB or 1 GB ones, reducing TLB misses for large working sets.

NUMA means memory is attached to a specific socket, and a core reading memory attached to the other socket pays roughly 1.5 to 2x the latency.

The OOM killer picks a process to terminate when the kernel cannot reclaim enough. There are two of them, and confusing them is the most common diagnostic error here: the cgroup OOM killer fires when a container exceeds memory.max and kills within that cgroup, while the global OOM killer fires under system-wide pressure and picks by oom_score across the whole machine.

What this is confused with: RSS as "memory used." RSS includes shared pages counted once per process, so summing RSS across processes over-counts, and it excludes swapped-out pages. The number the cgroup OOM killer actually acts on is the working set, and using the wrong metric is why memory limits are routinely set two or three times higher than needed (see requests, limits and QoS).

The problem it solves

Three failures that are all misdiagnosed in the same direction, as "we need more memory."

"The container is at 95 percent of its limit." Usually it is page cache, which is reclaimable, and the container is fine. container_memory_usage_bytes includes page cache; container_memory_working_set_bytes is usage minus inactive file pages, which is what the OOM killer considers. A container reading large files will approach its limit and never OOM, and a dashboard using the wrong metric produces a permanent false alarm.

"The process was OOM-killed but the node had free memory." That is the cgroup OOM killer, and it means the container exceeded its own limit. Node-level free memory is irrelevant. Exit code 137, OOMKilled in the pod status, and the kernel log names the cgroup.

"Adding memory did not help." If the workload is fault-bound rather than capacity-bound, more memory does nothing. A process with a large randomly-accessed working set and high TLB pressure is limited by address translation, and the fix is huge pages, not gigabytes.

The measurable shapes:

Symptom                         Metric that identifies it
──────────────────────────────────────────────────────────────────
Approaching limit, healthy      working_set flat while usage climbs
                                (page cache growing)
Genuine memory growth           working_set climbing
Thrashing                       major faults per second, high
                                pgscan/pgsteal ratio
TLB-bound                       high dTLB-load-misses in perf,
                                low IPC
NUMA-remote access              numastat: high numa_foreign / numa_miss

Mechanics

Page cache, and why "free" is the wrong number

$ free -h
               total   used   free   shared  buff/cache   available
Mem:            62Gi   18Gi  1.2Gi    340Mi        43Gi        43Gi

free of 1.2 GiB and available of 43 GiB. The 43 GiB in buff/cache is page cache that will be evicted the instant anything needs it. available is the number that matters and free is close to meaningless on a warm system.

Inside a cgroup:

$ cat /sys/fs/cgroup/.../memory.stat
anon 2147483648          # anonymous: heap, stack. NOT reclaimable.
file 8589934592          # page cache. Reclaimable.
inactive_file 7516192768 # the part that is cheaply reclaimable
active_file 1073741824
slab 268435456           # kernel structures for this cgroup

working_set = memory.current - inactive_file
            = 10.7 GB - 7.5 GB = 3.2 GB     <- what the OOM killer considers

That subtraction is the whole distinction. A container with a 4 GB limit showing 10.7 GB of memory.current is not about to be killed if 7.5 GB is inactive file cache; a container showing 3.9 GB of working set is.

Page faults, and what the ratio tells you

$ ps -o min_flt,maj_flt,cmd -p 8841
 MINFL  MAJFL CMD
1284102     47 /app/server            # healthy: faults resolved from memory

$ vmstat 1
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  8      0 210344  12044 894112    0    0  8420   112 4102 8841 12  8  4 76
                                                  ^^^^                    ^^
                                        heavy read I/O            76% iowait

Major faults are disk reads. A steady rate of major faults means the working set does not fit and the process is faulting pages back in continuously: thrashing. It presents as high iowait, low CPU utilisation and terrible latency, and it looks like a slow disk.

# Per-cgroup, the same signal:
$ cat /sys/fs/cgroup/.../memory.stat | grep -E 'pgfault|pgmajfault|pgscan|pgsteal'
pgfault 84102934
pgmajfault 291043        # <- climbing steadily is the tell
pgscan 41028394          # pages examined for reclaim
pgsteal 8841029          # pages actually reclaimed

pgscan divided by pgsteal is the reclaim efficiency. A ratio near 1 means reclaim is easy; a ratio of 10 or more means the kernel is scanning ten pages to free one, which is a system spending its time looking for memory rather than doing work.

Huge pages

4 KB pages, 32 GB working set:  8,388,608 page table entries
                                TLB holds ~1,500 entries
                                -> ~0.02% coverage, constant TLB misses

2 MB pages, same working set:   16,384 entries
                                -> ~9% coverage, far fewer misses

Each TLB miss is a page-table walk: up to four memory accesses to translate one address. For a large randomly-accessed working set, translation can be a significant fraction of total time, and no amount of extra RAM helps.

# Transparent Huge Pages: automatic, and the mode matters enormously.
$ cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

always is a well-known problem for databases. THP in always mode causes the kernel to compact memory synchronously to produce huge pages, which introduces multi-hundred-millisecond stalls in the allocation path. MongoDB, Redis, Couchbase and Oracle all document turning it off, and the recommended setting is madvise, so a process opts in with madvise(MADV_HUGEPAGE) rather than having it imposed.

echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
echo defer+madvise > /sys/kernel/mm/transparent_hugepage/defrag   # never compact synchronously

Explicit huge pages, for a JVM or a database that wants them deliberately:

echo 8192 > /proc/sys/vm/nr_hugepages          # 8192 x 2MB = 16 GB reserved
# JVM: -XX:+UseLargePages -XX:LargePageSizeInBytes=2m

In Kubernetes they are a schedulable resource:

resources:
  limits:
    hugepages-2Mi: "8Gi"
    memory: "16Gi"          # huge pages are counted SEPARATELY from memory

NUMA

$ numactl --hardware
node 0 cpus: 0-23,48-71     node 0 size: 128000 MB
node 1 cpus: 24-47,72-95    node 1 size: 128000 MB
node distances:
        0   1
  0:   10  21               # 2.1x the latency to reach node 1's memory
  1:   21  10

$ numastat -p 8841
                    Node 0     Node 1     Total
Private            41204.3     8102.1   49306.4
# A process pinned to node 0 with 8 GB on node 1 is paying 2x on that portion.

The default policy allocates on the node the faulting thread is running on, which is usually right, and it goes wrong when a process allocates a large arena at startup on one node and then runs threads on both.

numactl --cpunodebind=0 --membind=0 ./server      # pin both

In Kubernetes, the Topology Manager aligns CPU and memory allocation:

# kubelet config
topologyManagerPolicy: single-numa-node
cpuManagerPolicy: static

single-numa-node will refuse to admit a pod that cannot be satisfied within one node, which is a scheduling failure rather than a silent performance loss, and that is the right trade for latency-sensitive workloads.

The two OOM killers

Cgroup OOM killer fires when a cgroup's usage exceeds memory.max and reclaim cannot recover enough:

[12345.678] Memory cgroup out of memory: Killed process 8841 (server)
            total-vm:4194304kB, anon-rss:2097152kB, file-rss:32768kB
            oom_score_adj:968
[12345.679] Memory cgroup stats for /kubepods.slice/.../cri-containerd-abc.scope:
            anon:2097152kB file:8192kB

"Memory cgroup out of memory" names the cgroup, which tells you it was a limit breach rather than node pressure. The container is killed, restarted by the kubelet, and the node is otherwise unaffected.

Global OOM killer fires under system-wide pressure and chooses across the machine:

oom_score = (RSS + swap + page tables) / total_memory * 1000
            adjusted by oom_score_adj (-1000..1000)

Kubernetes sets oom_score_adj:
  Guaranteed:  -997        (protected)
  BestEffort:  1000        (killed first)
  Burstable:   computed from the memory request relative to node capacity

A global OOM kill on a Kubernetes node means the node's own reservations were wrong, because the kubelet should have evicted pods before the kernel had to act. Seeing "Out of memory: Killed process" without "Memory cgroup" in the log is a signal to check systemReserved, kubeReserved and the eviction thresholds.

# Protect a specific process from the global killer:
echo -1000 > /proc/8841/oom_score_adj      # never chosen

Swap, and why Kubernetes disabled it for years

Kubernetes required swap off until 1.22 and now supports it as beta (NodeSwap), because swap makes memory accounting and eviction decisions much harder: a container over its limit can be swapped rather than killed, which turns a fast failure into a slow one.

vm.swappiness is not a percentage of memory to swap. It is the relative cost the kernel assigns to reclaiming anonymous pages versus file pages:

vm.swappiness = 60   (default)   balanced
vm.swappiness = 1                strongly prefer dropping page cache
vm.swappiness = 0                only swap to avoid OOM

For a database with its own cache, swappiness=1 is standard, because swapping the database's buffer pool to disk to keep file cache in memory is exactly backwards.

A worked example: 4 GB of "leak" that was page cache

A log-processing service. Reads compressed files from object storage, parses, writes to a data warehouse. Java, running in Kubernetes.

Symptoms:

memory limit:                   8 GB
container_memory_usage_bytes:   climbs to 7.9 GB over ~6 hours, stays there
JVM heap (from JMX):            steady at 2.1 GB
alert:                          "memory > 95% of limit" fires every 6 hours
OOMKills:                       0 in three months
engineer response:              raise the limit to 12 GB, alert fires again
                                at 11.9 GB after ~9 hours

Zero OOM kills in three months is the tell. A container genuinely near its limit gets killed; one that sits at 99 percent indefinitely is holding reclaimable memory.

$ kubectl exec -it logproc-7d4 -- cat /sys/fs/cgroup/memory.stat
anon 2415919104           # 2.25 GB: the JVM heap plus native
file 5368709120           # 5.0 GB: PAGE CACHE from reading files
inactive_file 4831838208  # 4.5 GB of it is inactive and cheaply reclaimable
active_file 536870912

memory.current    = 7.9 GB
working_set       = 7.9 - 4.5 = 3.4 GB      <- the real number

The service was using 3.4 GB and the dashboard said 7.9. It read hundreds of gigabytes of files a day, and every read populated page cache that was charged to the cgroup and never needed to be evicted because nothing was pressing.

Fix 1: the metric.

# Wrong:
container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.95

# Right:
container_memory_working_set_bytes / container_spec_memory_limit_bytes > 0.90

false alerts:   ~120/month -> 0
memory limit:   12 GB -> 5 GB (working set p99 was 3.6 GB, plus headroom)
nodes required: 34 -> 26        (-24%, because limits drove scheduling)

Twenty-four percent fewer nodes from a metric change, because the inflated limits were consuming schedulable capacity for memory nobody was using.

Fix 2: the real memory problem, found once the noise was gone. With accurate metrics, a genuine slow growth was visible in anon:

anon over 7 days:  2.25 GB -> 2.31 GB -> 2.38 GB -> ...
JVM heap:          flat at 2.1 GB
-> the growth was NATIVE, not heap

$ kubectl exec -it logproc-7d4 -- jcmd 1 VM.native_memory summary
Total: reserved=4108MB, committed=2401MB
-  Thread (reserved=812MB, committed=812MB)      # <- 812 MB of thread stacks
                (thread #812)

Eight hundred and twelve threads, each with a 1 MB stack. A connection pool to the warehouse had maxPoolSize unset and was creating a thread per concurrent write, and the threads were never reaped.

// The fix, and it is the same shape as the Go MaxIdleConnsPerHost issue.
HikariConfig cfg = new HikariConfig();
cfg.setMaximumPoolSize(32);
cfg.setMinimumIdle(8);

threads:      812 -> 41
anon:         2.38 GB -> 1.62 GB
memory limit: 5 GB -> 3 GB

Fix 3: the fault profile, checked while they were in there.

$ kubectl exec -it logproc-7d4 -- cat /proc/1/stat | awk '{print "minflt="$10, "majflt="$12}'
minflt=48210394 majflt=1204          # major faults negligible: not thrashing

$ perf stat -p 1 -e dTLB-load-misses,instructions,cycles sleep 30
     412,884,102      dTLB-load-misses
  84,102,394,201      instructions
 102,884,392,104      cycles
                      -> IPC 0.82, dTLB miss rate ~0.5% of loads

A 0.5 percent dTLB miss rate on a parsing workload is high. They enabled THP in madvise mode and had the JVM opt in:

-XX:+UseTransparentHugePages

dTLB-load-misses:   412M -> 89M over 30s
IPC:                0.82 -> 1.04
throughput:         +19%

They did not use always mode, because the same node ran a Redis instance and THP always is documented as harmful there.

Final:

                              before      after
memory limit                  12 GB       3 GB
actual working set            3.4 GB      1.7 GB
false memory alerts           ~120/mo     0
threads                       812         41
nodes                         34          22        (-35%)
throughput                    baseline    +19%      (THP)
OOMKills                      0           0

Thirty-five percent fewer nodes and 19 percent more throughput, and the starting point was a dashboard using the wrong metric. No memory was ever leaking in the sense the team believed; the page cache accounting hid a genuine but much smaller native growth.

The transferable finding: usage minus inactive_file is the number, and a container that sits at 99 percent of its limit for months without being killed is holding reclaimable memory by definition. Zero OOM kills alongside a persistent high-memory alert is a contradiction, and resolving it is faster than any investigation of the application.

Production evidence

cAdvisor exposes both container_memory_usage_bytes and container_memory_working_set_bytes, and Kubernetes' own eviction logic uses working set. That the platform uses one metric and most dashboards use the other is the source of a large fraction of memory confusion.

THP in always mode is documented as harmful by MongoDB, Redis, Couchbase, Oracle and SAP HANA, all of which ship instructions to disable it. The mechanism is synchronous compaction stalls in the allocation path, and the consistency of the recommendation across independent database vendors is strong evidence.

Facebook's oomd and systemd-oomd act on PSI before the kernel's OOM killer fires, because by the time the kernel acts the system has usually been degraded for a while. Pressure-based intervention ahead of the kernel is the current direction.

Kubernetes' NodeSwap went beta in 1.28 after years of requiring swap disabled, and the KEP is explicit about why it was hard: swap makes limit enforcement and eviction ordering ambiguous, since a container over its limit can be slowed rather than killed.

The Topology Manager and CPU Manager exist because NUMA misalignment is a measurable latency cost on multi-socket nodes, and single-numa-node policy failing admission rather than silently degrading is a deliberate design choice.

The debate

Should you set memory limits at all? Yes, and they should equal the request for anything you care about, per the QoS argument. The counter-position, that limits cause unnecessary kills, is really an argument about limits being set from the wrong metric. With working_set-based sizing, limits are protective rather than hostile, and without them one container's leak becomes a node-level global OOM kill that picks its victim by heuristic.

Is memory.high the better tool? For absorbing transient spikes, yes, and it is under-used. It applies reclaim pressure and slows the process instead of killing it, so a two-second spike is absorbed rather than fatal. Kubernetes exposes it only indirectly through memoryThrottlingFactor on recent kubelets. The ideal is memory.high somewhat below memory.max: throttle first, kill only if that fails.

Should THP be on? madvise, not always, and this is close to unconditional on any node running a database. always mode's synchronous compaction produces latency stalls that are hard to attribute, and madvise lets processes that genuinely benefit opt in. The gain when it applies is real (19 percent throughput in the worked example) and it should be a decision rather than a default.

Does NUMA matter? On single-socket nodes, no. On multi-socket nodes with latency-sensitive workloads, a 2x memory latency penalty on a fraction of accesses is measurable, and the Topology Manager with single-numa-node is the Kubernetes-native answer. For most workloads it is not worth the scheduling constraint, because refusing to admit pods that cannot be NUMA-aligned reduces packing efficiency.

Should swap be enabled on Kubernetes nodes? Cautiously, and mostly no for latency- sensitive services. Swap converts a fast, obvious failure (OOM kill, restart, alert) into a slow, ambiguous one (everything gets slower and nothing reports why). For batch workloads where completion matters more than latency it is defensible, and there swappiness=1 plus a genuine memory limit is the shape.

What is the single most valuable memory metric? Working set against the limit, with pgmajfault rate as the second. The first tells you whether you are near being killed; the second tells you whether you are thrashing. Utilisation of memory.current tells you neither.

Follow-up Q&A

"A container sits at 99 percent of its memory limit and never gets OOM-killed. Explain."

It is holding reclaimable memory, almost always page cache from file I/O. container_memory_usage_bytes includes page cache; the OOM killer acts on the working set, which is usage minus inactive file pages. A container reading large files will approach its limit and stay there indefinitely, because the cache is evicted the moment anything needs the memory. Zero OOM kills alongside a persistent 99-percent alert is a contradiction that resolves immediately once you check memory.stat.

"What is the difference between the two OOM killers?"

The cgroup OOM killer fires when a container exceeds memory.max and kills within that cgroup; the kernel log says "Memory cgroup out of memory" and names the cgroup, and node-level free memory is irrelevant. The global OOM killer fires under system-wide pressure and picks across the machine by oom_score, which Kubernetes biases through oom_score_adj by QoS class. Seeing a global kill on a Kubernetes node means the kubelet should have evicted first, so check the node reservations and eviction thresholds.

"How do you tell thrashing from normal memory pressure?"

Major faults. Minor faults are resolved from memory already present and are normal in large numbers; major faults are disk reads to bring a page back. A steady pgmajfault rate means the working set does not fit and the process is faulting pages in continuously, which presents as high iowait with low CPU and looks like a slow disk. The second signal is the pgscan/pgsteal ratio: scanning ten pages to free one means the system is spending its time looking for memory.

"When do huge pages help?"

When the working set is large and randomly accessed, so the TLB cannot cover it. With 4 KB pages a 32 GB working set needs 8.4 million page-table entries against a TLB of roughly 1,500, and each miss is a page-table walk of up to four memory accesses. 2 MB pages cut the entries by 512x. Use madvise mode rather than always, because always causes synchronous compaction stalls that MongoDB, Redis, Couchbase and Oracle all document as harmful.

"Why did Kubernetes require swap off?"

Because swap makes limit enforcement and eviction ordering ambiguous: a container over its limit can be swapped rather than killed, so a fast obvious failure becomes a slow one where everything is slower and nothing reports why. NodeSwap reached beta in 1.28 with explicit accounting for it. For latency-sensitive services I would still leave it off; for batch work where completion matters more than latency it is defensible with swappiness=1.

"How would you size a memory limit correctly?"

From the p99 of container_memory_working_set_bytes over a week or two, plus about 20 percent, with the request set equal to it for Guaranteed QoS. Not from container_memory_usage_bytes, which includes reclaimable page cache and drifts toward the limit under normal operation. In one case that distinction took a limit from 12 GB to 3 GB and the cluster from 34 nodes to 22, because inflated limits were consuming schedulable capacity for memory nobody used.

Common misconceptions

"Free memory should be high." On a warm Linux system free is near zero and that is correct: unused RAM is wasted RAM, and page cache is evicted on demand. available is the number that matters.

"RSS is memory used." It counts shared pages once per process, so summing across processes over-counts, and it excludes swapped pages. Working set is what the cgroup OOM killer acts on.

"An OOM kill means the node ran out of memory." Usually it means one container exceeded its own limit. The kernel log distinguishes them: "Memory cgroup out of memory" is a limit breach and the node may have been mostly idle.

"Transparent huge pages are a free speedup." In always mode they cause synchronous compaction stalls that every major database vendor documents as harmful. madvise mode makes them opt-in, which is the right default.

"vm.swappiness is the percentage of memory to swap." It is the relative cost the kernel assigns to reclaiming anonymous pages versus file pages. swappiness=1 means strongly prefer dropping page cache, which is what a database with its own buffer pool wants.

Interview delivery note

Say this verbatim: "The number that matters is working set, which is usage minus inactive file pages, because that is what the cgroup OOM killer acts on. A container sitting at 99 percent of its limit for months with zero OOM kills is holding page cache by definition, and in one case fixing that metric took the limit from 12 GB to 3 and the cluster from 34 nodes to 22." The correct metric, the contradiction that identifies the error, and what it was worth.

The senior-versus-staff separator is treating "high memory with no OOM kills" as a contradiction to be resolved rather than a risk to be mitigated. A senior engineer sees a container near its limit and raises the limit. A staff engineer notices that a container genuinely near its limit would have been killed, concludes the memory is reclaimable, checks memory.stat for inactive_file, and finds that the alert was measuring the wrong thing. Reasoning from the absence of an expected failure is the move.

The second signal is distinguishing the two OOM killers by the kernel log line. "Memory cgroup out of memory" is a limit breach and tells you nothing about the node; "Out of memory: Killed process" without the cgroup prefix means the kubelet should have evicted first and the node reservations are wrong. Two different investigations from one word in a log.

The Staff & Team Lead Interview Reference