Namespaces and cgroups v2, hands-on

What it is

A container is not a kernel object. There is no struct container in Linux. A container is a process that has been placed in a set of namespaces, attached to a cgroup, given a different root filesystem, and had its capabilities and syscalls restricted. Remove all of that and you have an ordinary process, which is exactly what it is.

Two independent kernel mechanisms do the work, and conflating them is the most common misunderstanding:

	Namespaces	cgroups
Answer	What can this process see?	How much can this process use?
Mechanism	Separate instances of a global resource	Accounting and enforcement per group
Isolation of	PIDs, mounts, network, users, hostname, IPC	CPU, memory, I/O, PIDs (count)
Failure when wrong	The process sees things it should not	The process starves or is killed

Namespaces are about visibility; cgroups are about quantity. A process can be in a PID namespace where it is PID 1 and still consume every core on the machine, because nothing has put it in a CPU cgroup.

What this is confused with: virtualisation. A VM has its own kernel; a container shares the host kernel and is isolated by kernel features. That single fact explains the entire security discussion on container runtimes: a kernel exploit from inside a container is an exploit of the host, because it is the same kernel.

The problem it solves

Before namespaces, isolating processes on one machine meant chroot (which isolates the filesystem view and nothing else, and was never a security boundary) or a virtual machine (which duplicates a kernel, a boot sequence and hundreds of megabytes of memory per instance).

Namespaces make isolation a property of a process rather than of a machine, so isolation costs a few kilobytes of kernel structures instead of a guest kernel. That is why a node runs 100 containers and not 100 VMs.

The specific problems each namespace solves:

PID     a process should not see, signal, or /proc other tenants' processes
MOUNT   each container needs its own filesystem view and its own mounts
NET     two containers both want to bind port 8080
UTS     each container wants its own hostname
IPC     shared memory segments must not collide
USER    root inside the container must not be root outside
CGROUP  a container should not see the host's cgroup hierarchy
TIME    (5.6+) a container can have its own CLOCK_MONOTONIC offset

The user namespace is the one that matters for security and the one most often unused. Without it, root in a container is UID 0 on the host, and a container escape is immediately a root escape. With it, container-root maps to an unprivileged host UID and an escape lands you as nobody.

Mechanics

Building a container by hand

The clearest way to see that a container is not a thing is to assemble one:

# 1. New namespaces. Nothing else yet: no cgroup, no seccomp, no capabilities work.
sudo unshare --pid --mount --net --uts --ipc --fork --mount-proc bash

# Inside:
$ echo $$
1                                    # PID namespace: we are init
$ hostname container-1               # UTS namespace: our own hostname
$ ip link
1: lo: <LOOPBACK> mtu 65536          # NET namespace: only loopback exists
$ ps aux
USER  PID  COMMAND
root    1  bash                      # PID namespace: the host is invisible
root    9  ps aux

--mount-proc is load-bearing. Without remounting /proc, the PID namespace exists and ps still reads the host's /proc and shows every host process. The namespace changed what PIDs mean and not what /proc contains, which is a good demonstration that namespaces isolate a specific global resource and nothing more.

Then the filesystem:

# 2. A root filesystem. This is what an image is: a tarball of a directory tree.
mkdir -p /tmp/rootfs && cd /tmp/rootfs
docker export $(docker create alpine) | tar -x

sudo unshare --pid --mount --net --uts --ipc --fork \
    chroot /tmp/rootfs /bin/sh
# Inside, / is the alpine tree. pivot_root is the real-world version of this,
# because chroot can be escaped by a process that already holds a fd outside it.

And the limits:

# 3. cgroup v2: create a group, set limits, put the process in it.
sudo mkdir /sys/fs/cgroup/demo
echo "200000 100000" | sudo tee /sys/fs/cgroup/demo/cpu.max      # 2 CPUs
echo "536870912"     | sudo tee /sys/fs/cgroup/demo/memory.max   # 512Mi
echo "100"           | sudo tee /sys/fs/cgroup/demo/pids.max     # anti-fork-bomb
echo $$              | sudo tee /sys/fs/cgroup/demo/cgroup.procs

That is a container. Namespaces for visibility, a root filesystem, cgroups for quantity. A runtime like runc does exactly this, plus seccomp, capabilities and LSM labels, from a JSON spec.

cgroups v2, and what changed

v1 had a separate hierarchy per controller (/sys/fs/cgroup/cpu, /memory, /pids), so a process could be in different groups in each, and reasoning about it was genuinely hard. v2 has one unified hierarchy:

/sys/fs/cgroup/
├── cgroup.controllers          # available here
├── cgroup.subtree_control      # enabled for CHILDREN
├── kubepods.slice/
│   ├── cpu.max
│   ├── memory.max
│   └── kubepods-burstable.slice/
│       └── kubepods-burstable-pod8841.slice/
│           ├── cri-containerd-abc.scope/     # the actual container
│           └── memory.current

The no-internal-process rule: in v2, only leaf cgroups may contain processes. An internal node holds children, not tasks. This removes the v1 ambiguity about how an internal node's tasks compete with its children.

The interfaces worth knowing by name:

cpu.max        "MAX PERIOD"  ->  "200000 100000" = 2 CPUs per 100ms
cpu.weight     1-10000, default 100. PROPORTIONAL share under contention.
cpu.stat       nr_periods, nr_throttled, throttled_usec   <- the throttling evidence
memory.max     hard limit. Exceeding it invokes the cgroup OOM killer.
memory.high    SOFT limit: reclaim pressure, THROTTLES rather than killing.
memory.current what is actually charged
memory.stat    the breakdown: anon, file, slab, ...
memory.events  low/high/max/oom/oom_kill counters
io.max         per-device rbps/wbps/riops/wiops
pids.max       process count cap

memory.high is the v2 feature that should be used more. It applies reclaim pressure and slows the process instead of killing it, which for a service with a transient spike is enormously better than an OOM kill. Kubernetes does not expose it directly, though memoryThrottlingFactor in recent kubelets sets it as a fraction of the limit.

Pressure Stall Information: the metric that says "how badly"

$ cat /sys/fs/cgroup/kubepods.slice/.../memory.pressure
some avg10=12.45 avg60=8.31 avg300=3.02 total=48211934
full avg10=2.10  avg60=1.44 avg300=0.51 total=8841029

some is the fraction of time at least one task was stalled; full is the fraction where every task was stalled. PSI is a direct measure of resource contention, and it is far more useful than utilisation:

CPU at 95% utilisation, cpu.pressure some=2%   -> fine, work is getting done
CPU at 60% utilisation, cpu.pressure some=40%  -> tasks are WAITING. A problem.

Utilisation says how busy the resource is; PSI says how much time was lost waiting for it. A container can be at moderate utilisation and badly starved, and only PSI shows it.

The user namespace, and why it took so long

# Map container UID 0..65535 to host UID 100000..165535.
unshare --user --map-root-user bash
$ id
uid=0(root) gid=0(root)              # inside
$ cat /proc/self/uid_map
         0     100000      65536     # outside, we are 100000

Root inside is unprivileged outside. A process that escapes lands as UID 100000, which owns nothing.

Kubernetes support (spec.hostUsers: false) reached beta in 1.30. The delay was real engineering: every file in the image needs correct ownership under the mapping, which historically meant a chown of the whole rootfs per pod (idmapped mounts, kernel 5.12+, fixed that), and volume ownership has to agree with the mapping.

Without user namespaces, runAsNonRoot is the practical mitigation, and it is weaker: the process is not UID 0, and a capability or a setuid binary can still get you further than you want.

What Kubernetes actually creates

# On a node, for one pod:
$ systemd-cgls /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/
kubepods-burstable-pod8841_....slice
├─cri-containerd-9f2a....scope        # the PAUSE container
│ └─3841 /pause
├─cri-containerd-a1b2....scope        # your app
│ └─3902 /app/server
└─cri-containerd-c3d4....scope        # a sidecar
  └─3945 /usr/bin/envoy

The pause container is what owns the pod's namespaces. It does nothing (it sleeps) and holds the network, IPC and UTS namespaces so the other containers can join them, and so they survive an app container restarting. That is why containers in a pod share an IP and localhost: they are in one network namespace, held open by pause.

Note what is not shared: the mount namespace and the PID namespace, by default. Each container has its own filesystem view, and shareProcessNamespace: true is opt-in.

A worked example: a fork bomb that was not a fork bomb

A CI platform running untrusted build jobs as Kubernetes pods. Nodes were becoming unresponsive roughly twice a week, requiring a hard reboot.

Symptoms:

node becomes unresponsive:      ~2/week
kubelet stops reporting:        yes
SSH:                            times out
console (via cloud provider):   "fork: Cannot allocate memory"
after reboot:                   nothing in the logs, node healthy

"Cannot allocate memory" on fork with plenty of free memory is the PID-exhaustion signature, not a memory problem. The kernel's global PID space (kernel.pid_max, default 4194304 on modern systems but frequently 32768) had been exhausted.

Investigation:

# On a node during the failure (caught via a debug daemonset):
$ cat /proc/sys/kernel/pid_max
32768
$ ls /proc | grep -c '^[0-9]'
32011

# Which cgroup?
$ for c in /sys/fs/cgroup/kubepods.slice/*/*/*/; do
    echo "$(cat $c/pids.current 2>/dev/null) $c"
  done | sort -rn | head -3
28104 .../kubepods-besteffort-pod4471.../cri-containerd-9f2a.scope
  412 .../kubepods-burstable-pod8841.../cri-containerd-a1b2.scope
  188 ...

One container held 28,104 processes. It was a build job whose test suite spawned a process per test case and did not reap them: a zombie accumulation, not a malicious fork bomb.

The reason it took the node down rather than just itself:

$ cat .../cri-containerd-9f2a.scope/pids.max
max                                  # <- NO LIMIT

pids.max was unset, so the container could consume the node's entire PID space. Once exhausted, nothing on the node could fork, including the kubelet's health checks, sshd's session setup and systemd. The node was alive and unable to start any new process.

Fix 1: a PID limit per pod.

# kubelet config
podPidsLimit: 4096

# Which sets, per pod cgroup:
$ cat /sys/fs/cgroup/.../kubepods-besteffort-pod4471.slice/pids.max
4096

node lockups:           2/week -> 0
failing builds:         now fail with "resource temporarily unavailable"
                        inside the container, which is correct and diagnosable

The container now fails instead of the node. That is the entire purpose of a cgroup limit and it had been left at the default of unlimited.

Fix 2: reserve PIDs for the system.

systemReserved: {pid: "1000"}
kubeReserved:   {pid: "1000"}
evictionHard:   {"pid.available": "10%"}

PID is a reservable resource in the kubelet and almost nobody sets it, which is how a misbehaving pod starves the node's own daemons even with per-pod limits in place.

Fix 3: the finding that made the platform actually safe. With PID limits working, the team audited what else was unbounded on these untrusted workloads:

pids.max:        unset       -> 4096
memory.max:      set         (BestEffort pods had none: no memory request or limit)
io.max:          unset       -> per-device caps
cpu.max:         unset       (deliberate, per the CPU-limits argument)
user namespace:  not enabled -> container root was host root
seccomp:         Unconfined  -> RuntimeDefault

BestEffort pods had no memory limit at all, so the same class of failure was available through memory. And seccomp: Unconfined was the default for these pods, so a build job had the full syscall surface.

# The baseline applied to all untrusted workloads:
spec:
  hostUsers: false                      # user namespace (1.30+)
  securityContext:
    runAsNonRoot: true
    runAsUser: 65534
    seccompProfile: {type: RuntimeDefault}
    capabilities: {drop: ["ALL"]}
    readOnlyRootFilesystem: true
    allowPrivilegeEscalation: false
  containers:
  - resources:
      requests: {cpu: "500m", memory: "1Gi"}
      limits:   {memory: "2Gi"}         # memory limit == request would be better

Measured over the following quarter:

                              before      after
node lockups                  2/week      0
noisy-neighbour incidents     ~6/week     0
builds failing for resource
  reasons                     0 (they
                              took the
                              node)      ~14/week (correctly, with clear errors)
mean node PID utilisation     unbounded   under 12%
container escape surface      host root   unprivileged host UID + seccomp

Fourteen builds a week now fail that previously succeeded by taking the node down with them. That is the correct trade and it needed to be communicated as such: the failures were always happening, and previously they were happening to everyone on the node.

The transferable lesson: an unset cgroup limit is unlimited, and the defaults are unlimited for everything except memory. pids.max, io.max and the user namespace are all off by default, and each of them is a path from one container to a dead node. The audit question is not "what did we configure" but "what did we leave unset."

Production evidence

cgroups v2 is the default on Fedora 31+, Ubuntu 21.10+, RHEL 9+ and Debian 11+, and Kubernetes has supported it since 1.25 as GA. The v1-to-v2 migration mattered because tools reading /sys/fs/cgroup/memory/... paths break, which is why some JVM and monitoring agent versions report wrong container limits on v2 hosts.

PSI (Pressure Stall Information) was contributed by Facebook (Johannes Weiner) and is used in production there for oomd, their userspace OOM killer that acts on pressure before the kernel's killer fires. systemd-oomd ships it on desktop Linux, which is a good signal that pressure-based decisions beat threshold-based ones.

memory.high and pressure-based reclaim are used by Facebook's senpai for automated memory sizing: apply pressure, observe, and converge on the smallest working set. That is the same idea as VPA's percentile sizing done through the kernel rather than through metrics.

User namespaces in Kubernetes reached beta in 1.30 after years of work, and the KEP documents exactly the obstacles described above: file ownership under the mapping, which idmapped mounts (kernel 5.12) solved, and volume support.

The pause container is documented in Kubernetes as the namespace holder, and its source is a few dozen lines whose main job is reaping orphaned zombies as PID 1 in pods with a shared PID namespace.

The debate

Are namespaces a security boundary? Weakly. They are a visibility boundary enforced by a shared kernel, so any kernel vulnerability crosses them. The historical escapes (CVE-2019-5736 overwriting the runc binary, CVE-2022-0492 abusing cgroup release_agent, dirty pipe) all worked by exploiting the shared kernel rather than by defeating a namespace. My position: namespaces plus seccomp plus dropped capabilities plus a user namespace is a reasonable boundary for semi-trusted workloads, and for genuinely untrusted code you want a different kernel, which is the container runtimes argument.

Should you enable user namespaces? Yes, where your kernel and Kubernetes version support it, and it is the single largest reduction in escape impact available. The friction is real (file ownership, volumes, some CNI and CSI plugins) and it has been reduced substantially by idmapped mounts. Without it, runAsNonRoot is the fallback, and the difference matters: runAsNonRoot means you are not UID 0 inside; a user namespace means UID 0 inside is unprivileged outside.

Is PSI better than utilisation? For deciding whether a resource is a problem, yes, and it is under-collected. Utilisation at 95 percent with 2 percent pressure is a well-used system; 60 percent utilisation with 40 percent pressure is a starved one. PSI measures lost time, which is what you care about, and it is the metric to reach for when utilisation and observed latency disagree.

memory.high or memory.max? Both. memory.max is the hard backstop that prevents a runaway consuming the node. memory.high set somewhat below it applies reclaim pressure first, so a transient spike slows down instead of being killed. Kubernetes exposes only the limit directly, and recent kubelets set memory.high from memoryThrottlingFactor, which is worth knowing exists. Being killed for a two-second spike is a bad trade when throttling would have absorbed it.

What should you set that you probably have not? pids.max, via podPidsLimit. It defaults to unlimited, one container can exhaust the node's PID space, and the resulting failure is a node that cannot fork anything including its own daemons. It is one kubelet flag and it converts a node outage into a pod failure.

Follow-up Q&A

"What is a container, mechanically?"

A process in a set of namespaces, attached to a cgroup, with a different root filesystem and restricted capabilities and syscalls. There is no container object in the kernel. Namespaces decide what it can see (PIDs, mounts, network, users, hostname, IPC); cgroups decide how much it can use (CPU, memory, I/O, process count). You can build one by hand with unshare, pivot_root and a few writes to /sys/fs/cgroup.

"Namespaces versus cgroups?"

Visibility versus quantity, and they are independent. A process can be in a PID namespace where it is PID 1 and still consume every core, because nothing put it in a CPU cgroup. Conversely a process can be cgroup-limited and see the whole host. Container runtimes apply both, plus seccomp and capabilities, from one spec.

"What does the pause container do?"

It holds the pod's network, IPC and UTS namespaces open so the app containers can join them, and so those namespaces survive an app container restarting. That is why containers in a pod share an IP and can reach each other on localhost. It also reaps orphaned zombies as PID 1 where the PID namespace is shared. It sleeps and does nothing else.

"What is PSI and why is it better than utilisation?"

Pressure Stall Information reports the fraction of time tasks were stalled waiting for a resource: some means at least one task was stalled, full means all were. Utilisation tells you how busy a resource is; PSI tells you how much time was lost waiting for it. 95 percent utilisation with 2 percent pressure is healthy; 60 percent utilisation with 40 percent pressure is starved. It is the metric to reach for when utilisation and observed latency disagree.

"Why do user namespaces matter?"

Without one, root inside the container is UID 0 on the host, so a container escape is immediately a root escape. With one, container UID 0 maps to an unprivileged host UID, and an escape lands you as an account that owns nothing. It reached beta in Kubernetes 1.30; the delay was file ownership under the mapping, which idmapped mounts in kernel 5.12 largely solved. runAsNonRoot is the weaker fallback.

"A node became unresponsive with 'cannot allocate memory' on fork, and memory was free. What happened?"

PID exhaustion. The kernel's global PID space was consumed, so nothing could fork, including the kubelet and sshd, which is why the node looked dead while being alive. The cause is almost always a container with pids.max unset, since the default is unlimited: in one case a build job accumulating zombies held 28,104 processes. The fix is podPidsLimit in the kubelet plus reserving PIDs for the system, and it converts a node outage into a pod failure.

Common misconceptions

"A container is a lightweight VM." A VM has its own kernel; a container shares the host's. Every isolation property comes from kernel features, so a kernel vulnerability crosses the boundary.

"Namespaces limit resources." They limit visibility. A process alone in a PID namespace can still consume every core and all the memory unless a cgroup says otherwise.

"cgroup defaults are safe." Only memory is limited by default in Kubernetes, and only if you set a limit. pids.max and io.max are unlimited unless configured, and each is a path from one container to an unusable node.

"Root in a container is not really root." Without a user namespace it is UID 0 on the host. Capabilities and seccomp narrow what it can do; the UID is the same.

"chroot is isolation." It changes the apparent root and is escapable by a process holding a file descriptor outside it. Real runtimes use pivot_root and unmount the old root.

Interview delivery note

Say this verbatim: "A container is a process in namespaces, in a cgroup, with a different root filesystem. Namespaces decide what it can see and cgroups decide how much it can use, and they are independent: a process can be alone in a PID namespace and still consume every core. The defaults are unlimited for everything except memory, so the audit question is what you left unset, not what you configured." The mechanism, the split, and the diagnostic question.

The senior-versus-staff separator is knowing that pids.max is unlimited by default and what that costs. A senior engineer explains namespaces and cgroups correctly. A staff engineer knows that one container can exhaust the node's global PID space, that the resulting failure is a node that cannot fork anything including the kubelet and sshd (so it looks dead while being alive), and that podPidsLimit converts a node outage into a pod failure. Naming the failure signature, "cannot allocate memory on fork with memory free," is the checkable part.

The second signal is PSI. Using pressure rather than utilisation to decide whether a resource is a problem, and being able to say that 60 percent utilisation with 40 percent pressure is worse than 95 percent with 2 percent, shows you measure lost time rather than busyness.

The Staff & Team Lead Interview Reference