The reconciliation loop as the Kubernetes mental model

What it is

Kubernetes is not an orchestrator that executes your commands. It is a set of control loops, each watching some resource and continuously working to make the world match a declared desired state.

for ever:
    desired  = read from the API server (spec)
    observed = look at the actual world
    if desired != observed:
        take one step toward desired
    write what you saw (status)

Every Kubernetes behaviour follows from that loop. kubectl apply does not create a pod; it writes a Deployment object. The Deployment controller notices a Deployment with no matching ReplicaSet and creates one. The ReplicaSet controller notices a ReplicaSet whose observed pod count is below its desired count and creates Pods. The scheduler notices Pods with no nodeName and assigns one. The kubelet on that node notices a Pod assigned to it and starts containers.

Five independent loops, none of which called the next. They communicate only through the API server, by writing and watching objects.

What it is confused with: an imperative system. kubectl delete pod does not delete a pod in the sense that matters. It marks the Pod for deletion; the ReplicaSet controller then observes a shortfall and creates a replacement. The pod is gone and a pod exists, which is exactly what was declared. People report this as "Kubernetes recreated my pod," and the accurate description is that nothing ever asked for a pod count of two.

The property that follows and that everything else depends on: level-triggered, not edge-triggered. A controller does not react to an event; it reads the current state and acts on the gap. A missed event is harmless because the next sync sees the same gap. This is why Kubernetes recovers from a controller crash, a lost watch connection, or a several-hour outage without any replay or reconciliation log.

The problem it solves

Distributed orchestration by command does not survive failure. A system where a scheduler sends "start this container" to a node has to answer: what if the message is lost, what if the node was restarting, what if the acknowledgement is lost but the container started, what if the scheduler crashes between deciding and sending. Every one of those is a distributed-systems problem requiring at-least-once delivery, idempotency and a recovery log.

Reconciliation makes those questions disappear. The node reads the desired state and compares it to what is running. A lost message is irrelevant because there are no messages, only state. A crashed controller resumes by reading current state. A node that was offline for an hour comes back, reads what it should be running, and converges.

The second problem it solves is composability. Because controllers communicate only through objects, a new controller can be added that watches an existing resource without any existing controller knowing. That is how the ecosystem (cert-manager, external-dns, Argo CD, every operator) exists: they are additional loops on the same objects.

Mechanics

Everything is spec, status and a loop

apiVersion: apps/v1
kind: Deployment
spec:                          # DESIRED: written by you
  replicas: 3
  template: {...}
status:                        # OBSERVED: written by the controller
  replicas: 3
  readyReplicas: 2
  observedGeneration: 7

observedGeneration against metadata.generation is the single most useful debugging field in Kubernetes and is almost never used. generation increments on every spec change; observedGeneration is what the controller has processed. If they differ, the controller has not yet seen your change, which distinguishes "my change is not working" from "my change has not been read."

kubectl get deploy web -o jsonpath='{.metadata.generation} {.status.observedGeneration}'
# 7 5   -> the controller is 2 generations behind. Look at the controller, not the pods.

The chain, in full

kubectl apply -f deploy.yaml
  │
  ▼  API server: validate, admit (webhooks), persist to etcd
  │
  ▼  Deployment controller (watching Deployments)
  │     sees: Deployment with spec.replicas=3, no matching ReplicaSet
  │     does: create ReplicaSet
  │
  ▼  ReplicaSet controller (watching ReplicaSets)
  │     sees: RS wants 3, observes 0 Pods with its ownerReference
  │     does: create 3 Pods
  │
  ▼  Scheduler (watching Pods with spec.nodeName == "")
  │     sees: 3 unscheduled Pods
  │     does: filter nodes, score them, write spec.nodeName
  │
  ▼  Kubelet on each node (watching Pods with spec.nodeName == me)
  │     sees: a Pod assigned to it, not running
  │     does: pull image, create containers via CRI, report status
  │
  ▼  Endpoints/EndpointSlice controller (watching Pods and Services)
        sees: a ready Pod matching a Service selector
        does: add it to the EndpointSlice, which kube-proxy then programs

No step calls the next. Each writes an object and another controller notices. That is why a failure at any stage leaves the system in a consistent, resumable state, and why the diagnostic question is always "which loop has stopped, and what does it observe?"

Ownership and garbage collection

metadata:
  ownerReferences:
  - apiVersion: apps/v1
    kind: ReplicaSet
    name: web-7d4b9c
    uid: 8841-...
    controller: true
    blockOwnerDeletion: true

The garbage collector is itself a reconciliation loop: it watches for objects whose owners no longer exist and deletes them. Cascading deletion is not a delete operation, it is a consequence of ownership plus a loop.

This is why kubectl delete rs --cascade=orphan leaves running pods behind: it strips the owner reference, and now no loop is watching them. Orphaned pods are the clearest demonstration that nothing is executing commands; the pods keep running because no controller has any opinion about them.

Writing a controller

func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // 1. READ desired state. Not-found means deleted: nothing to do.
    var app myv1.Application
    if err := r.Get(ctx, req.NamespacedName, &app); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // 2. OBSERVE the actual world.
    var deploy appsv1.Deployment
    err := r.Get(ctx, types.NamespacedName{Name: app.Name, Namespace: app.Namespace}, &deploy)

    // 3. Take ONE step toward desired. Idempotent by construction.
    switch {
    case apierrors.IsNotFound(err):
        return ctrl.Result{}, r.Create(ctx, r.buildDeployment(&app))
    case err != nil:
        return ctrl.Result{}, err
    case !equality.Semantic.DeepEqual(deploy.Spec, r.desiredSpec(&app)):
        deploy.Spec = r.desiredSpec(&app)
        return ctrl.Result{}, r.Update(ctx, &deploy)
    }

    // 4. Write status. NEVER assume; report what you observed.
    app.Status.ObservedGeneration = app.Generation
    app.Status.ReadyReplicas = deploy.Status.ReadyReplicas
    return ctrl.Result{}, r.Status().Update(ctx, &app)
}

Four properties this code must have, and each corresponds to a real production failure:

Idempotent. Reconcile runs many times for one change, and must converge to the same result. A reconcile that appends to a list rather than setting it grows that list forever.

No memory between invocations. Everything needed is read at the top. A controller holding state across reconciles is wrong after any restart, and the restart is when you need it most.

One step, then return. Do not loop inside Reconcile waiting for something. Return and be called again. A Reconcile that blocks for 30 seconds holds a worker and stalls every other object in the queue.

Report observation, not intent. status.readyReplicas must come from observing pods, not from what you asked for. A controller that writes its intent into status makes the status useless for exactly the debugging it exists for.

Requeue, backoff and the level-triggered guarantee

return ctrl.Result{RequeueAfter: 30 * time.Second}, nil   // check again later
return ctrl.Result{}, err                                  // error: EXPONENTIAL BACKOFF
return ctrl.Result{}, nil                                  // done; wake on the next event

Returning an error triggers rate-limited exponential backoff, typically from 5ms to 1000s. A controller in a hot error loop is throttled automatically, which is why a broken controller degrades rather than melting the API server.

Resync is the safety net. Even with no events, informers do a full resync (default around 10 hours) so every object is reconciled. That is what makes a missed watch event harmless: the loop is level-triggered and the resync guarantees the level is eventually read.

Where the mental model pays off in debugging

"My pod won't start"
  -> WHICH LOOP is stuck, and what does it observe?

Pod has no nodeName            -> scheduler: no node satisfies the constraints
                                  kubectl describe pod -> Events from the scheduler
Pod has nodeName, no container -> kubelet: image pull, CRI, resource admission
                                  check the kubelet on THAT node
Pod is Running but not Ready   -> the readiness probe, and therefore
                                  EndpointSlice will not include it
Pod is Ready but no traffic    -> EndpointSlice controller, or kube-proxy
                                  programming on the client's node
Deployment shows old replicas  -> compare generation to observedGeneration

Every question becomes "which controller, and what is the gap it sees?" That is the practical value of the model and it is why it is worth stating as one.

A worked example: a mutating webhook that stopped everything

A platform team ran a mutating admission webhook injecting sidecars, environment variables and labels. It had run for eighteen months.

The incident:

14:02  webhook deployment rolled to a new version
14:02  new pods fail readiness (a bad config path in the new image)
14:04  all 3 webhook replicas Not Ready
14:04  EndpointSlice for the webhook Service becomes empty
14:05  EVERY pod creation cluster-wide begins failing:
       Internal error occurred: failed calling webhook "inject.platform.io":
       failed to call webhook: no endpoints available

Nothing could be created anywhere, in any namespace, including the webhook's own replacement pods. The webhook's failurePolicy was Fail, and its rule matched pods with no namespace exclusion.

Why this was a deadlock and not merely an outage: the ReplicaSet controller was working correctly. It observed 0 ready webhook pods against a desired 3 and tried to create pods. Each creation went to the API server, which called the admission webhook, which had no endpoints, which failed the request. The loop was running and could not converge, because converging required the thing that was broken.

ReplicaSet controller: "I need 3 pods"  -> create pod
API server:            "let me admit that"  -> call webhook
Webhook Service:       no endpoints
API server:            reject
ReplicaSet controller: still 0 pods. Retry with backoff. Forever.

Resolution took 41 minutes, and only because someone with cluster-admin deleted the MutatingWebhookConfiguration object, which removed the admission step and let the loop converge.

What made it survivable next time:

webhooks:
- name: inject.platform.io
  failurePolicy: Fail
  # 1. Never intercept the namespaces that must work for recovery.
  namespaceSelector:
    matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values: [kube-system, platform-system]
  # 2. Opt-in rather than opt-out: only namespaces that ask for it.
    - key: platform.io/inject
      operator: In
      values: ["enabled"]
  # 3. A short timeout, so a slow webhook degrades rather than hanging.
  timeoutSeconds: 5
  # 4. Only the resources actually needed.
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]

plus:

# The webhook itself must not depend on itself.
# Its own namespace is excluded above; its deployment additionally uses:
spec:
  template:
    metadata:
      labels:
        platform.io/bootstrap: "true"     # excluded from injection
  # And a PDB that cannot take all replicas out at once:
---
apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  minAvailable: 2
  selector: {matchLabels: {app: inject-webhook}}

And the diagnostic that would have caught it in 90 seconds:

- alert: WebhookServiceNoEndpoints
  expr: |
    kube_endpoint_address_available{endpoint=~".*webhook.*"} == 0
  for: 1m
  annotations:
    summary: "An admission webhook has no endpoints. Pod creation may be blocked."

Measured, replaying the same failure in staging:

                              before        after
time to cluster-wide impact   2 minutes     no cluster-wide impact
namespaces affected           all (94)      only opted-in (31)
kube-system affected          yes           no
webhook self-recovery         impossible    yes (own ns excluded)
time to detect                18 min        90 s (alert)
time to resolve               41 min        4 min (rollback proceeds normally)

The lesson is a direct consequence of the reconciliation model. Controllers converge by creating objects, so anything that can block object creation can prevent the system from repairing itself. A failurePolicy: Fail webhook on pods with no namespace exclusion is a cluster-wide single point of failure, and it is one that only manifests when the webhook itself is unhealthy, which is precisely when you need pod creation to work.

The general form, worth carrying beyond webhooks: in a reconciliation system, ask what each control-plane component needs in order to recover, and make sure that thing does not depend on it.

Production evidence

Kubernetes' own design documents describe the controller pattern and level-triggered reconciliation explicitly, and the API conventions document specifies the spec/status split and observedGeneration semantics. The design is stated rather than emergent.

controller-runtime and Kubebuilder are the standard frameworks for writing controllers, and their documented contract is exactly the four properties above: idempotent, stateless between invocations, one step per reconcile, status reports observation. That the framework enforces this shape is evidence it is the load-bearing part.

The operator pattern (CoreOS, then the Operator Framework) is reconciliation applied to application-specific resources, and the large ecosystem (Prometheus Operator, cert-manager, Argo CD, hundreds of database operators) exists because adding a loop over new object types requires no changes to Kubernetes.

Argo CD and Flux implement GitOps as reconciliation with git as the desired state, which is the model taken one level out: the cluster reconciles toward a repository, and drift is detected and corrected by the same mechanism.

The webhook deadlock in the worked example is a widely documented failure mode. Kubernetes' own documentation warns that admission webhooks intercepting resources in kube-system can deadlock a cluster, and recommends namespace selectors excluding system namespaces. It is a known trap with a documented mitigation that is frequently not applied.

The debate

Is level-triggered always right? It is the property that makes Kubernetes robust, and it costs latency and API load. A level-triggered loop must poll or resync to guarantee convergence, so a change can take a resync period to be noticed if a watch is lost, and every controller re-reads state it may already know. Edge-triggered systems react instantly and require exactly-once delivery and a recovery path. My position: for a control plane, level-triggered is clearly correct, because the failure modes of edge-triggered coordination are the hard ones and this design makes them not exist.

Should you write an operator? The bar is higher than the ecosystem suggests. An operator is a distributed system you now maintain, with the four correctness properties above, and getting idempotency or status reporting wrong produces failures that look like Kubernetes bugs. Write one when the operational knowledge is genuinely complex and repeated (database failover, certificate rotation, multi-step upgrades). Do not write one to template YAML, which is what Helm or Kustomize is for.

Is the abstraction worth its cost? The honest criticism is that five loops between kubectl apply and a running container is a lot of indirection, and debugging requires knowing all five. The counter is that the indirection is what gives you self-healing, and the alternative systems that are simpler to trace do not recover from a node being offline for an hour without an operator doing something. The complexity is the failure handling made explicit rather than deferred.

What is the most under-used debugging tool? observedGeneration against generation. It answers "has the controller even seen my change," which distinguishes a controller problem from a configuration problem, and almost nobody looks at it. The second is kubectl describe's Events, which are the controllers reporting what they observed and why they could not converge.

Where does the model break down? When a controller's desired state depends on something outside the API server that is not itself reconciled: a cloud API that rate-limits, an external DNS provider, a manual step. Those introduce edge-triggered dependencies into a level-triggered system, and they are where operators accumulate their worst bugs. The mitigation is to reflect external state into a status field so the loop can observe it, rather than assuming it.

Follow-up Q&A

"Explain the reconciliation loop."

Every Kubernetes controller runs the same loop: read the desired state from the API server, observe the actual world, take one step to close the gap, write what it observed to status. Controllers never call each other; they communicate only by writing and watching objects. kubectl apply writes a Deployment, and five independent loops (Deployment, ReplicaSet, scheduler, kubelet, EndpointSlice) each notice a gap and act, with no step invoking the next.

"What does level-triggered mean and why does it matter?"

A controller acts on the current gap between desired and observed, not on an event. A missed event is harmless because the next sync reads the same state and sees the same gap. That is why Kubernetes recovers from a controller crash, a lost watch connection or an hour of node downtime with no replay log and no reconciliation protocol. An edge-triggered system would need at-least-once delivery, idempotent handlers and a recovery path for each of those.

"Why does deleting a pod recreate it?"

Because nothing asked for that pod. The ReplicaSet declares a replica count, and the ReplicaSet controller observes a shortfall after the deletion and creates a replacement. The pod you deleted is gone and the declared state is satisfied. --cascade=orphan on the ReplicaSet demonstrates the same thing from the other side: strip the owner reference and the pods keep running, because now no loop has an opinion about them.

"How do you debug a pod that will not start?"

Ask which loop is stuck and what it observes. No nodeName means the scheduler could not place it, so read its events for the filter that failed. A nodeName with no container means the kubelet on that node, so check image pull, the CRI and resource admission there. Running but not Ready means the readiness probe, and therefore it will not be in the EndpointSlice. Ready but no traffic means the EndpointSlice controller or kube-proxy programming on the client's node. And before all of it, compare generation to observedGeneration to know whether the controller has even read your change.

"What are the correctness properties of a controller?"

Idempotent, because reconcile runs many times per change and must converge to the same result. Stateless between invocations, because a controller holding memory is wrong after a restart, which is when you need it. One step per reconcile, returning rather than blocking, because a reconcile that waits holds a worker and stalls the queue. And status must report what was observed rather than what was intended, or the field is useless for the debugging it exists for.

"How can a webhook deadlock a cluster?"

Controllers converge by creating objects, and admission webhooks intercept object creation. A webhook with failurePolicy: Fail on pods, with no namespace exclusion, that becomes unhealthy will reject every pod creation cluster-wide, including the creation of its own replacement pods. The loop is running correctly and cannot converge, because converging requires the broken thing. The mitigation is namespace selectors excluding system namespaces and the webhook's own namespace, an opt-in label, and a short timeout.

A pod is stuck in ImagePullBackOff. What is the reconciliation loop actually doing, and how do you diagnose it? ImagePullBackOff is not an error state, it is the backoff state that follows repeated ErrImagePull failures, and the distinction matters because it tells you the loop is still running. The kubelet tried to pull, failed, and is now retrying with exponential backoff capped at five minutes. Nothing will resolve it except the pull succeeding, so the loop will keep retrying forever, which is the reconciliation model working as designed rather than a stuck controller.

The diagnosis is always kubectl describe pod, and the Events section names the actual cause, which is one of four. The image or tag does not exist, usually a typo or a tag that was never pushed. The registry needs credentials the pod does not have, meaning a missing or wrong-namespace imagePullSecrets, and note that a secret is namespaced so copying the pod without the secret is a common cause. The node cannot reach the registry, which is a NetworkPolicy, DNS or egress problem and shows as a timeout rather than a 401 or 404. Or you are being rate limited, which since Docker Hub introduced anonymous pull limits has become a frequent cause and produces a distinctive toomanyrequests message. The signal in an interview is knowing that the message distinguishes these: a 401 is credentials, a 404 is the tag, a timeout is the network, and toomanyrequests is quota. Guessing rather than reading the event is the failure.

Common misconceptions

"kubectl apply creates the resource." It writes an object. Controllers then notice gaps and act. Nothing in Kubernetes executes your command.

"Kubernetes recreated my pod." Nothing asked for a pod. A ReplicaSet asked for a count, and the count was restored. The pod you deleted is genuinely gone.

"Controllers call each other." They communicate only through the API server, by writing and watching objects. That is what allows a new controller to be added without any existing one knowing.

"A missed watch event loses the update." Level-triggered reconciliation means the next sync reads the current state and sees the same gap, and a periodic full resync guarantees it regardless. Events are an optimisation, not the mechanism.

"An admission webhook only affects what it intercepts." A failurePolicy: Fail webhook that becomes unhealthy blocks object creation, and object creation is how every controller converges. It is a cluster-wide dependency that only bites when it is broken.

Interview delivery note

Say this verbatim: "Kubernetes does not execute commands, it runs control loops that compare desired to observed and take one step to close the gap. It is level-triggered, so a missed event is harmless, which is why the system recovers from a crashed controller or an offline node with no replay log. And every debugging question becomes: which loop is stuck, and what gap does it see?" The model, the property that makes it work, and the diagnostic it gives you.

The senior-versus-staff separator is understanding that anything blocking object creation prevents self-healing. A senior engineer explains reconciliation correctly. A staff engineer notices that controllers converge by creating objects, so a failurePolicy: Fail admission webhook with no namespace exclusion is a cluster-wide single point of failure that only manifests when the webhook itself is unhealthy, which is exactly when pod creation must work. Generalising to "ask what each control-plane component needs in order to recover, and ensure that thing does not depend on it" is the staff-level move.

The second signal is observedGeneration. Using it to distinguish "the controller has not seen my change" from "the controller cannot satisfy my change" is a thirty-second check that almost nobody performs, and it splits the diagnosis in half immediately.

The Staff & Team Lead Interview Reference