HPA, VPA, Cluster Autoscaler, Karpenter, KEDA

What it is

Five autoscalers operating on three different axes, and confusing which axis a tool works on is the source of most autoscaling misconfiguration:

                    ┌─────────────────────────────────────┐
   MORE PODS  ──▶   │  HPA          (metric-driven)       │
                    │  KEDA         (event-driven)        │
                    └─────────────────────────────────────┘
                    ┌─────────────────────────────────────┐
   BIGGER PODS ──▶  │  VPA          (right-sizing)        │
                    └─────────────────────────────────────┘
                    ┌─────────────────────────────────────┐
   MORE NODES ──▶   │  Cluster Autoscaler (node groups)   │
                    │  Karpenter          (any instance)  │
                    └─────────────────────────────────────┘

	Scales	Trigger	Reacts to
HPA	Replica count	CPU, memory, custom or external metrics	Load on existing pods
KEDA	Replica count (drives an HPA)	60+ event sources: queue depth, lag, cron	Work waiting to be done
VPA	Requests and limits	Historical usage percentiles	What the pod actually needs
Cluster Autoscaler	Nodes within predefined node groups	Unschedulable pods	Capacity shortfall
Karpenter	Nodes, choosing the instance type itself	Unschedulable pods	Capacity shortfall

The critical relationship: node autoscalers react to pods that cannot be scheduled, which means they are downstream of the pod autoscalers and downstream of resource requests. A cluster that scales nodes badly usually has a requests problem, not a node-autoscaler problem (see requests, limits and QoS).

What this is confused with: HPA and VPA being complementary by default. They conflict on the same resource: HPA sees rising CPU utilisation and adds pods, VPA sees the same and raises the request, which lowers utilisation, which makes HPA remove pods. Running both on CPU is a documented conflict, and the supported combination is HPA on a custom metric with VPA on memory only.

The problem it solves

Fixed capacity is wrong in both directions all the time. Provision for peak and you pay for idle capacity most of the day; provision for average and you fail at peak. The measurable version on a typical business-hours workload:

peak concurrent load:        4,200 rps
trough (03:00):                180 rps
ratio:                          23x

fixed capacity for peak:     100% of the day at peak cost
utilisation at 03:00:        4%

The failures autoscaling introduces if done naively are specific:

Scaling on the wrong signal. CPU-based HPA on an I/O-bound service scales down when latency rises, because a thread blocked on a slow downstream call consumes no CPU. That is the failure on the consumer lag page and it generalises to any I/O-bound workload.

Thrash. Aggressive scale-down removes a pod, load per pod rises, the HPA adds it back, and the cycle repeats with a rebalance or a connection storm each time.

Scaling into a bottleneck. Adding application pods when the database connection pool is the constraint makes things worse, because more pods means more connections to a database that is already the limit.

Mechanics

HPA: the algorithm

desiredReplicas = ceil(currentReplicas x (currentMetricValue / desiredMetricValue))

current:  10 pods at 85% CPU, target 50%
desired:  ceil(10 x (85/50)) = ceil(17) = 17 pods

The tolerance is 10 percent by default (--horizontal-pod-autoscaler-tolerance), so no action is taken while the ratio is within 0.9 to 1.1 of target. That is what prevents constant small adjustments and it is why an HPA at 54 percent against a 50 percent target does nothing.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: api}
  minReplicas: 6
  maxReplicas: 60
  metrics:
  - type: Resource
    resource: {name: cpu, target: {type: Utilization, averageUtilization: 60}}
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0        # react immediately
      policies:
      - {type: Percent, value: 100, periodSeconds: 30}   # double at most every 30s
      - {type: Pods, value: 10, periodSeconds: 30}
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 300      # 5 minutes of stability first
      policies:
      - {type: Percent, value: 10, periodSeconds: 60}    # shed 10% per minute
      selectPolicy: Min

The asymmetry is the whole design. Scaling up late costs availability; scaling down early costs stability. stabilizationWindowSeconds on scale-down uses the maximum recommendation over the window, so a brief dip does not trigger removal.

averageUtilization is a percentage of the request, not of the node. A pod requesting 200m and using 170m is at 85 percent, regardless of node capacity, so the HPA target is meaningless if the request is wrong. That is the dependency between the two pages.

Custom and external metrics: usually the right signal

CPU is a proxy for load and frequently a bad one:

  metrics:
  # Requests per second per pod: what you actually mean.
  - type: Pods
    pods:
      metric: {name: http_requests_per_second}
      target: {type: AverageValue, averageValue: "100"}

  # Queue depth, from outside the cluster.
  - type: External
    external:
      metric:
        name: sqs_approximate_number_of_messages
        selector: {matchLabels: {queue: orders}}
      target: {type: AverageValue, averageValue: "30"}

With multiple metrics the HPA computes a replica count for each and takes the maximum, which is the safe combination: any metric can demand more pods and none can force fewer.

KEDA: scaling on work waiting, and to zero

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
spec:
  scaleTargetRef: {name: order-processor}
  minReplicaCount: 0                 # <- HPA cannot do this
  maxReplicaCount: 40
  cooldownPeriod: 300
  triggers:
  - type: aws-sqs-queue
    metadata:
      queueURL: https://sqs.../orders
      queueLength: "20"              # target messages PER REPLICA
  - type: cron                       # pre-scale before a known spike
    metadata:
      timezone: America/Toronto
      start: "0 8 * * 1-5"
      end: "0 18 * * 1-5"
      desiredReplicas: "10"

KEDA generates an HPA underneath, so the scaling behaviour is the same; what it adds is the metric adapter for 60-plus sources and the activation logic for scale-to-zero.

Scale-to-zero is the differentiator and it is the reason to choose KEDA for queue consumers, batch processors and anything with genuinely idle periods. The cost is cold-start latency on the first message after idle, which is the same trade as Lambda cold starts.

The multi-trigger cron pattern is under-used. Pre-scaling before a known spike removes the reaction lag entirely for predictable traffic, and it composes with the queue trigger because the effective replica count is the maximum across triggers.

VPA: right-sizing, and the eviction problem

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
spec:
  targetRef: {apiVersion: apps/v1, kind: Deployment, name: api}
  updatePolicy:
    updateMode: "Off"          # RECOMMEND ONLY. Start here, always.
  resourcePolicy:
    containerPolicies:
    - containerName: '*'
      minAllowed: {cpu: 100m, memory: 128Mi}
      maxAllowed: {cpu: 4, memory: 8Gi}
      controlledResources: ["memory"]     # memory only, to avoid the HPA conflict

$ kubectl describe vpa api
Recommendation:
  Container: api
    Lower Bound:  cpu: 180m,  memory: 412Mi
    Target:       cpu: 340m,  memory: 780Mi     <- what to set
    Upper Bound:  cpu: 890m,  memory: 1420Mi

updateMode: "Off" is where to start and where most clusters should stay. The recommender's percentile-based sizing is better than most manual values and costs nothing; Auto mode evicts pods to apply new values, which for a stateful or slow-starting service is disruptive.

In-place pod resize (KEP-1287, beta in recent versions) removes the eviction requirement and makes Auto mode substantially more attractive, which is the change to watch.

The HPA conflict, stated precisely:

HPA on CPU:  utilisation rises -> add pods
VPA on CPU:  utilisation rises -> raise the request
             -> utilisation (as a % of request) FALLS
             -> HPA removes pods
             -> utilisation rises again

Supported combination: HPA on a custom metric (requests per second, queue depth), VPA on memory only. That is what controlledResources: ["memory"] above expresses.

Cluster Autoscaler versus Karpenter

Both react to unschedulable pods, which is worth stating twice because it means neither looks at utilisation:

Cluster Autoscaler works with predefined node groups (an ASG, a MIG, a node pool):

1. A pod is Pending with "Insufficient cpu".
2. Simulate: which node group, if scaled up, would fit this pod?
3. Increase that group's desired count.
4. Wait for the node to join, then the scheduler places the pod.

Constraints:
  - node groups must be defined in advance
  - all nodes in a group are the same instance type
  - scale-up is ~1 node group at a time per loop
  - typical time to schedulable: 3-5 minutes

Karpenter provisions instances directly, choosing the type:

apiVersion: karpenter.sh/v1
kind: NodePool
spec:
  template:
    spec:
      requirements:
      - {key: karpenter.sh/capacity-type, operator: In, values: ["spot", "on-demand"]}
      - {key: kubernetes.io/arch, operator: In, values: ["amd64", "arm64"]}
      - {key: karpenter.k8s.aws/instance-category, operator: In, values: ["c","m","r"]}
      - {key: karpenter.k8s.aws/instance-generation, operator: Gt, values: ["5"]}
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s
    budgets:
    - nodes: "10%"                # bound the disruption rate
  limits: {cpu: "2000", memory: 4000Gi}

1. A pod is Pending.
2. Compute the pod's exact requirements (CPU, memory, arch, zone, taints).
3. Choose the cheapest instance type that fits, from the whole catalogue.
4. Launch it directly. Typical time to schedulable: 40-60 seconds.

Consolidation is Karpenter's other half and the larger saving in practice. It continuously evaluates whether workloads could fit on fewer or cheaper nodes and replaces them, which is bin-packing as a continuous process rather than a scale-down decision.

                    Cluster Autoscaler      Karpenter
node groups         required, predefined    none: any instance type
time to schedulable ~3-5 min                ~40-60 s
instance selection  fixed per group         cheapest that fits
bin-packing         only on scale-down      continuous consolidation
spot handling       via node groups         native, with interruption handling
cloud support       AWS, GCP, Azure, more   AWS mature, Azure GA, others emerging
disruption control  PDBs                    PDBs + disruption budgets

The honest cost of Karpenter is churn. Continuous consolidation means nodes are replaced regularly, so PodDisruptionBudgets, graceful termination and karpenter.sh/do-not-disrupt annotations become load-bearing in a way they are not with Cluster Autoscaler.

How they compose

KEDA/HPA:  queue depth rises -> more pods
              ↓
Scheduler: pods are Pending, nothing fits
              ↓
Karpenter: provisions a node sized for exactly those pods (~50 s)
              ↓
Scheduler: places the pods
              ↓
VPA (Off):  recommends better requests for next time

The chain's latency is the sum, and the node provisioning step dominates. That is why minReplicas and headroom matter: an autoscaler that starts from zero spare capacity pays the full node-provisioning time on every spike.

Overprovisioning with low-priority placeholder pods is the standard trick:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: {name: overprovisioning}
value: -10                          # NEGATIVE: evicted by anything real
globalDefault: false
---
# A Deployment of pause containers with real resource requests, at that
# priority. They hold capacity; a real pod preempts them instantly, and
# Karpenter then provisions their replacement in the background.

This converts node provisioning time from the critical path into a background task, and it costs the price of the placeholder capacity.

A worked example: 41 percent utilisation and failing during spikes

A B2C platform, 340 services, business-hours traffic with a 20x peak-to-trough ratio, plus unpredictable marketing spikes.

Baseline:

nodes:                        180 (fixed, 3 node groups)
mean CPU utilisation:         41%
utilisation at 03:00:         6%
HPA coverage:                 61 of 340 services, all on CPU
spike behaviour:              5xx for 4-8 minutes during marketing pushes
monthly compute:              $214,000

Problem 1: CPU-based HPA on I/O-bound services.

checkout-api: CPU-based HPA, target 70%
  during a downstream slowdown:
    latency:      120 ms -> 2,400 ms
    CPU per pod:  62% -> 31%        <- threads blocked, not computing
    HPA action:   SCALED DOWN from 24 to 12 pods

The HPA removed half the capacity during an incident. The fix was to scale on a signal that tracks demand rather than computation:

  metrics:
  - type: Pods
    pods:
      metric: {name: http_inflight_requests}     # concurrency, not CPU
      target: {type: AverageValue, averageValue: "40"}

In-flight requests is the best general signal for a request-serving service, because it rises when the service is slow and when traffic increases, which are both reasons to add capacity. CPU rises for only one of them.

during the same downstream slowdown, after:
  HPA action:   scaled UP from 24 to 58 pods
  5xx rate:     4.1% -> 0.2%

Problem 2: node provisioning on the critical path.

marketing spike, 0 to 3,000 rps in ~90 seconds:
  HPA reacts:                    ~30 s (metric scrape + evaluation)
  pods Pending:                  immediately
  Cluster Autoscaler reacts:     ~60 s
  node ready:                    ~3.5 min
  pods scheduled and warm:       ~4.5 min total
  5xx during that window:        yes

They moved to Karpenter and added overprovisioning:

                          before      after
node provisioning         3.5 min     52 s
overprovision headroom    none        8% of cluster capacity
time to absorb a spike    4.5 min     ~35 s (preempt placeholders, then
                                       provision in the background)
5xx during spikes         4-8 min     none observed

The overprovisioning was worth more than the faster provisioning. Karpenter took node time from 3.5 minutes to 52 seconds; the placeholder pods took the spike off the critical path entirely, because real pods preempt them instantly.

Problem 3: requests were wrong, which made everything else wrong.

VPA in recommendation mode across all 340 services:
  services requesting >2x their p99 usage:   197
  services requesting <p99 usage:             34    <- being throttled/OOMKilled
  mean over-request (CPU):                   2.7x

after applying VPA recommendations:
  cluster allocated (CPU):    91% -> 48%
  nodes:                      180 -> 104
  mean utilisation:           41% -> 63%

Fixing requests removed 76 nodes, and it also made every HPA target meaningful, because averageUtilization is a percentage of the request. An HPA targeting 70 percent of a request that is 2.7x too large was targeting 26 percent of actual capacity.

Problem 4: queue consumers idle overnight.

14 batch and queue-consumer services:
  overnight replicas:      minReplicas: 3 each = 42 pods
  overnight work:          none

# KEDA with scale-to-zero, plus a cron trigger for the known morning batch.
minReplicaCount: 0
triggers:
- type: aws-sqs-queue
  metadata: {queueLength: "20"}
- type: cron
  metadata: {start: "45 5 * * *", end: "0 9 * * *", desiredReplicas: "8"}

overnight pods:     42 -> 0
overnight nodes:    31 -> 6
first-message
  latency after
  idle:             80 ms -> 12 s      <- accepted for these workloads

Twelve seconds of latency on the first message after an idle period was acceptable for batch consumers and would not have been for the API, which is the judgement that decides where scale-to-zero belongs.

Problem 5: the HPA/VPA conflict, hit during the rollout.

Two services had VPA in Auto mode on CPU alongside a CPU-based HPA:

observed oscillation:
  09:00  HPA: 12 pods at 78% CPU -> scale to 18
  09:04  VPA: sustained CPU high -> raise request 400m -> 700m, EVICT pods
  09:06  pods restart with a larger request, utilisation now 45%
  09:11  HPA: 45% < 70% target -> scale down to 12
  09:20  utilisation rises again -> repeat

pod restarts caused by the loop:  ~200/day
fix:  VPA controlledResources: ["memory"] only, HPA on in-flight requests

Final:

                              before      after
nodes                         180         96
mean CPU utilisation          41%         67%
overnight nodes               180         38
5xx during marketing spikes   4-8 min     none observed
HPA coverage                  61/340      284/340
scale-to-zero services        0           14
node provisioning time        3.5 min     52 s
monthly compute               $214,000    $118,000    (-45%)

Forty-five percent off compute with better spike behaviour, and the ordering mattered: fixing requests came first, because every HPA target and every node-autoscaler decision is computed from requests. Autoscaling a cluster whose requests are 2.7x too large scales the error.

Production evidence

Karpenter was built at AWS and donated to the CNCF as part of the Kubernetes Autoscaling SIG. AWS's published case studies report node provisioning in under a minute against several minutes for Cluster Autoscaler, and consolidation is the feature most often credited with the cost reduction.

KEDA is a CNCF graduated project with 60-plus scalers, and it generates a standard HPA rather than replacing it, which is why its scaling behaviour is identical and only the metric source and the scale-to-zero activation differ.

The HPA/VPA conflict on the same resource is documented by the VPA project itself, with the guidance to use VPA for memory and HPA for a custom metric. It is not a subtle interaction and it is still hit regularly.

In-place pod resize (KEP-1287) reached beta and removes VPA's eviction requirement, which is the change that makes Auto mode viable for services that were previously excluded.

Overprovisioning with negative-priority pause pods is documented in the Cluster Autoscaler FAQ as the standard approach to removing node provisioning from the critical path.

Google's Borg paper reported systematic over-requesting by users and described resource reclamation to recover the gap, which is the same finding as the VPA recommendation exercise a decade earlier and at much larger scale.

The debate

What should an HPA scale on? Not CPU, for a request-serving service. In-flight requests or requests per second are better, because they rise both when traffic increases and when the service slows down, and both are reasons to add capacity. CPU rises for only the first, which is why CPU-based HPA scales down during a downstream slowdown. CPU is defensible for genuinely compute-bound workloads.

Karpenter or Cluster Autoscaler? Karpenter for AWS, on the evidence: faster provisioning, instance selection from the whole catalogue rather than predefined groups, and continuous consolidation. The cost is churn, so PodDisruptionBudgets and graceful termination stop being optional. Cluster Autoscaler remains the right answer on clouds where Karpenter is less mature, and for clusters where node stability matters more than cost.

Should you run VPA? In recommendation mode, everywhere, immediately: it is free and its percentile sizing beats manual values. In Auto mode, only for workloads that tolerate eviction, which excluded most services until in-place resize. The recommendations alone were worth 76 nodes in the worked example, without VPA ever changing anything automatically.

Is scale-to-zero worth it? For queue consumers, batch jobs and internal tools with genuine idle periods, clearly. The cost is cold-start latency on the first request, which was 12 seconds in the worked example and acceptable for a batch consumer and not for an API. The question is whether the first request after idle has a human waiting for it.

How much headroom should you keep? Enough to absorb a spike while nodes provision, which means the overprovisioning approach rather than a raw minReplicas bump. Around 5 to 10 percent of cluster capacity in negative-priority placeholder pods removes node provisioning from the critical path entirely and costs that percentage. Compared with 5xx during every marketing push, it is cheap.

What is the ordering? Requests first, always. Every HPA target is a percentage of the request, every scheduling decision uses requests, and both node autoscalers react to pods that cannot be scheduled against requests. Autoscaling a cluster with 2.7x over-requesting scales the error, and the VPA recommendation pass is the cheapest first step.

Follow-up Q&A

"What should an HPA scale on?"

For a request-serving service, in-flight requests or requests per second rather than CPU. CPU is a proxy for load that fails in the specific case you most need it: when a downstream dependency slows down, threads block, CPU per pod falls, and a CPU-based HPA scales down during the incident. In one case it went from 24 pods to 12 while latency was 20x normal. In-flight requests rise both when traffic increases and when the service slows, and both are reasons to add capacity.

"Why do HPA and VPA conflict?"

On the same resource, they form a loop. HPA sees high CPU utilisation and adds pods; VPA sees the same and raises the request, which lowers utilisation as a percentage of request, which makes HPA remove pods. VPA's Auto mode also evicts pods to apply new values, so the loop produces restarts: about 200 a day in one case. The supported combination is HPA on a custom metric with VPA restricted to memory via controlledResources.

"Karpenter or Cluster Autoscaler?"

Karpenter on AWS. Cluster Autoscaler scales predefined node groups, so it is constrained to instance types you configured in advance and takes 3 to 5 minutes; Karpenter computes the pod's exact requirements and launches the cheapest instance that fits from the whole catalogue, in under a minute, and continuously consolidates workloads onto fewer or cheaper nodes. The cost is churn: nodes are replaced regularly, so PodDisruptionBudgets and graceful termination become load-bearing.

"How do you handle a spike faster than nodes can provision?"

Take node provisioning off the critical path with overprovisioning: a Deployment of pause containers at a negative PriorityClass, holding real resource requests. A real pod preempts them instantly and the node autoscaler provisions their replacement in the background. In one case that mattered more than moving to Karpenter: node time went from 3.5 minutes to 52 seconds, and the placeholders took spike absorption to about 35 seconds.

"When would you use KEDA over a plain HPA?"

When the signal is work waiting rather than load on existing pods (queue depth, consumer lag, a cron schedule), or when you want scale-to-zero, which HPA cannot do. KEDA generates an HPA underneath, so the scaling behaviour is identical; what it adds is 60-plus metric sources and the activation logic for zero. Its multi-trigger support is under-used: a cron trigger that pre-scales before a known spike removes the reaction lag for predictable traffic.

"What do you fix first?"

Resource requests. Every HPA target is a percentage of the request, the scheduler places on requests, and both node autoscalers react to pods that cannot be scheduled against requests. A cluster with 2.7x over-requesting has HPA targets that mean something different from what they say and node autoscaling that provisions for capacity nobody uses. Running VPA in recommendation mode is free and, in one case, applying its output removed 76 of 180 nodes before any autoscaling change.

Common misconceptions

"Node autoscalers watch utilisation." Both Cluster Autoscaler and Karpenter react to unschedulable pods. A cluster at 20 percent utilisation with no Pending pods will not scale down under Cluster Autoscaler's default behaviour unless nodes are empty enough to consolidate, and neither will add nodes for high utilisation alone.

"HPA and VPA are complementary." They conflict on the same resource and produce an oscillation with pod restarts. Use VPA for memory and HPA for a custom metric.

"CPU is the natural HPA metric." It is the default and it is wrong for I/O-bound services, where it falls during exactly the incidents that require more capacity.

"averageUtilization is a percentage of the node." It is a percentage of the pod's request, so a wrong request makes the target meaningless.

"Scale-to-zero is free." It costs cold-start latency on the first request after idle, 12 seconds in one measured case. Right for batch consumers, wrong for anything with a human waiting.

Interview delivery note

Say this verbatim: "I would not scale a request-serving service on CPU, because when a downstream dependency slows down the threads block, CPU per pod falls, and the HPA scales down during the incident. In one case it halved capacity while latency was 20x normal. In-flight requests rise both when traffic increases and when the service slows, and both are reasons to add pods." A specific, checkable failure of the default choice.

The senior-versus-staff separator is fixing requests before touching autoscaling. A senior engineer configures HPAs, chooses Karpenter and tunes the behaviour blocks. A staff engineer notices that averageUtilization is a percentage of the request, that the scheduler and both node autoscalers all operate on requests, and that a cluster over-requesting by 2.7x has targets meaning something different from what they say. Running VPA in recommendation mode first removed 76 of 180 nodes before any autoscaling change.

The second signal is overprovisioning with negative-priority pods. Recognising that the autoscaling chain's latency is a sum dominated by node provisioning, and that placeholder capacity converts that into a background task, shows you are optimising the critical path rather than each component.

The Staff & Team Lead Interview Reference