How a do-not-disrupt annotation broke Karpenter consolidation

Our first read was wrong. The Monday FinOps digest fired at 08:15 UTC showing EC2 on-demand up 2.2x for the trailing 72 hours, and the platform lead's instinct was 'someone shipped an HPA that went sideways over the weekend.' We checked requests per second on the top 20 services; nothing moved more than 8% week over week. Total pod count was up about 40 pods against a fleet of 2,400. Not an HPA problem. Something was keeping nodes without a matching demand signal, and the extra capacity was mostly on-demand even though the NodePool allowed spot, which Karpenter prefers when it can choose.

The evidence that pointed us at the right layer came from Karpenter's own counters. We were on v0.37, and we read them for the 60-hour incident window against the same window a week earlier:

# Prometheus, Karpenter v0.37 metric names. Run each as written, then again
# with "offset 7d" after the [60h] for the comparison week.
sum(increase(karpenter_nodes_created[60h]))
sum(increase(karpenter_disruption_nodes_disrupted_total{method="consolidation"}[60h]))

                  incident   a week earlier
nodes created        196          412
consolidated          41          347

Launches were not the signal. They fell too, since part of a normal week's launches are the replacement nodes consolidation itself starts. What collapsed was the rate at which Karpenter took nodes away.

The provisioning path was healthy. There were no failed provisioning or unschedulable events. Karpenter launched 196 nodes against 412 in the comparison week, but it removed far fewer: 41 through consolidation against 347, plus the usual 65 or so lost to spot interruptions. Over 60 hours that gap accumulated into 90 extra nodes, most of them on-demand, sitting quietly and costing money. The on-demand skew is consolidation's absence too: on a normal weekend Karpenter replaces the on-demand nodes it launches during a spot shortfall once spot capacity returns, and that replacement is itself a consolidation.

Once the question was 'why isn't consolidation firing', the answer came out fast. Karpenter's consolidation logic evaluates candidate nodes by checking whether every pod on the node can be safely rescheduled. Any node running a pod with karpenter.sh/do-not-disrupt: true is disqualified as a candidate for as long as that pod runs there, and Karpenter says so in a DisruptionBlocked event on the node, which names the pod. The annotation exists for legitimate reasons: Kafka Streams state, long-running batches with expensive warmup, workloads that lose data on SIGTERM. In our cluster, four workloads owned it and were the only ones that should have had it.

The reality was different:

kubectl get events -A --field-selector reason=DisruptionBlocked,involvedObject.kind=Node \
  -o jsonpath='{range .items[*]}{.message}{"\n"}{end}' | head -3

Cannot disrupt Node: Pod "payments/checkout-api-6f9c7d5b8-x2k4q" has "karpenter.sh/do-not-disrupt" annotation
Cannot disrupt Node: Pod "search/indexer-5b7d9c6f4-p8v2m" has "karpenter.sh/do-not-disrupt" annotation
Cannot disrupt Node: Pod "notifications/dispatcher-7c8f6d9b5-j3n6r" has "karpenter.sh/do-not-disrupt" annotation

kubectl get pods --all-namespaces -o json \
  | jq -r '[.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true")] | length'

1450

kubectl get pods --all-namespaces -o json \
  | jq -r '[.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true") | .metadata.namespace] | unique | length'

38

1,450 pods across 38 namespaces carried the annotation. With pods distributed by the default scheduler, that pinned roughly 85% of nodes.

Our first suspicion was a mutating admission webhook. kubectl get mutatingwebhookconfigurations returned nothing new. The annotation was in the pod specs at the source. We picked one pod, walked back to its ReplicaSet, saw the annotation in the pod template, and ran git blame on the tenant's Deployment manifest. The blame line pointed at an internal chart bump: internal-charts/base-deployment moved from 2.7.4 to 2.7.5 on Friday at 19:47 UTC. The diff was two lines:

+ annotations:
+   karpenter.sh/do-not-disrupt: "true"

Commit message: 'add do-not-disrupt to prevent midday restarts during batch runs.'

The chart owner had one tenant whose Kafka Streams StatefulSet was losing ~90 seconds of state on every consolidation event. The narrow fix was to annotate that one StatefulSet, which required coordinating with the tenant. The wide fix was to put the annotation in the shared chart, which required coordinating with no one. They picked the wide fix. Our Renovate config auto-merged minor and patch bumps from internal charts, and 2.7.4 to 2.7.5 was a patch. Thirty-four tenant repos consumed base-deployment. Over the weekend, thirty-four PRs opened, thirty-four PRs merged, thirty-four ArgoCD syncs rolled deployments, and every rolled pod inherited the annotation.

flowchart LR
  A[Chart 2.7.5 merged<br/>Fri 19:47 UTC] --> B[Renovate opens<br/>34 tenant PRs]
  B --> C[Auto-merge on<br/>minor and patch bumps]
  C --> D[ArgoCD syncs<br/>rolled deployments]
  D --> E[1,450 pods now<br/>carry the annotation]
  E --> F[~85% of nodes<br/>unconsolidatable]
  F --> G[Rollouts and spot replacements<br/>add nodes nothing removes]
  G --> H[Node count nearly doubles<br/>over 60 hours]

A single opinionated chart plus a permissive auto-merge is the shape of the whole incident. Nothing in the loop was malicious. Every step was doing what it was configured to do.

By 09:00 UTC Monday we had the diagnosis and a bad plan. The obvious move was to patch the Deployment template in each of the 34 tenants, removing the annotation at the source:

kubectl patch deployment <name> -n <ns> \
  -p '{"spec":{"template":{"metadata":{"annotations":{"karpenter.sh/do-not-disrupt":null}}}}}'

The command that looked surgical and was not.

We almost ran the loop. Then one of us asked the question that matters: does changing a template annotation trigger a rollout? It does. The Deployment controller's ComputeHash runs DeepHashObject over the entire PodTemplateSpec, and metadata.annotations is part of the template. Change any pod-template annotation and the template hash changes, which creates a new ReplicaSet, which rolls every pod. The cleanest proof of this is that kubectl rollout restart triggers a rollout precisely by stamping kubectl.kubernetes.io/restartedAt into spec.template.metadata.annotations. If template annotations were excluded from the hash, rollout restart could not work.

If we had run the patch loop across 34 tenants at 09:00 UTC on a Monday, we would have rolled about 1,450 pods in flight, cascaded PDB blocks into deploy pipelines, and (worst) triggered even more Karpenter provisioning as the rollouts churned. The recovery would have looked identical to a second, larger incident.

The correct move was to strip the annotation from the *running pods*, not from the template. A ReplicaSet reconciles pod count and pod ownership, not annotation drift on pods that already exist. So mutating a live pod's annotations is safe and does not trigger replacement. Karpenter's disruption loop runs every 10 seconds, and it reconsiders consolidation when pod placement or nodes change (an annotation edit does not count) and at least once every five minutes, so the newly-eligible nodes start draining within a few minutes, on their own.

Here is what we ran. It strips the annotation only from pods outside an explicit keep list, the four workloads that own the annotation plus the fraud-scoring batch, so the pods that need it keep it:

# Pods that keep the annotation: the four workloads that own it, and the
# fraud-scoring batch mid-run. Keyed on namespace and workload-name prefix,
# which every pod carries. A lookalike name is kept too, so check what was
# kept by running the same pipeline with grep -E instead of grep -Ev.
KEEP='^(streams/orders-enricher-|ledger/settlement-writer-|ml/feature-backfill-|media/transcoder-|risk/fraud-scoring-)'

kubectl get pods -A -o json \
  | jq -r '.items[]
      | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true")
      | "\(.metadata.namespace)/\(.metadata.name)"' \
  | grep -Ev "$KEEP" \
  | awk -F/ '{print $1, $2}' \
  | xargs -r -n2 -P8 sh -c 'kubectl annotate pod -n "$0" "$1" karpenter.sh/do-not-disrupt-'

The trailing dash on the annotation key removes it. 1,426 pods, eight calls at a time, took just under three minutes.

Verification was two commands, one for the pod count and one for Karpenter's response:

kubectl get pods --all-namespaces -o json \
  | jq -r '[.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true")] | length'

24

# The chart logs JSON and runs two replicas: read both pods, lift kubectl's
# 10-line default for selectors with --tail=-1, and parse.
kubectl -n karpenter logs -l app.kubernetes.io/name=karpenter --since=10m --tail=-1 \
  | jq -Rr 'fromjson? | select((.message // "") | startswith("disrupting via consolidation")) | .message' \
  | sed 's/ pods).*/ pods)/' | tail -5

disrupting via consolidation delete, terminating 1 nodes (6 pods)
disrupting via consolidation delete, terminating 1 nodes (4 pods)
disrupting via consolidation replace, terminating 1 nodes (11 pods)
disrupting via consolidation delete, terminating 2 nodes (9 pods)
disrupting via consolidation delete, terminating 1 nodes (3 pods)

Down to 24 annotated pods (the four legitimate workloads plus a 20-pod fraud-scoring batch six hours into its run) and Karpenter firing consolidation events within four minutes.

The strip did not stay done. The 34 Deployment templates still carried the annotation until the chart fix shipped, so every pod a ReplicaSet created afterwards came back annotated, including the pods evicted from the nodes consolidation was now removing, and each one re-pinned the node it landed on. We re-ran the strip every 15 minutes until base-deployment 2.7.6 was out, and the annotated count crept back from 24 to around 60 between passes. Over the next 90 minutes node count dropped from 198 to 141 and plateaued. The 24 kept pods were pinning about 22 nodes: 20 fraud-batch pods spread across 18 nodes, and 4 legitimate pods on 4 more. Consolidation could not touch those. We coordinated a restart of the fraud batch with its owning team. It checkpointed cleanly and restarted onto 5 nodes, tight-packed by Karpenter's provisioning-time bin packing, which freed 13 nodes and took us to 128. Over the next 90 minutes the repeated strips and natural pod churn freed another 20 nodes, and Karpenter consolidated those too. Node count landed at 108 by 15:00 UTC, back inside baseline range.

Then we rolled out the fixed chart. base-deployment 2.7.6 removed the blanket annotation and gated it behind an explicit values.yaml opt-in (karpenter.doNotDisrupt: false default, true for the four legitimate tenants). This rollout DID replace pods, because it changed the template hash, and we scheduled it deliberately. Renovate opened the PRs, but we had switched off auto-merge for this chart for the rollout. The platform team merged them in batches of five with a ten-minute soak between batches so any interaction with PDBs or startup probes surfaced before the next wave. Six hours end to end, no SLO impact.

The postmortem produced three durable controls. Each one closes a specific step in the causal chain above.

The first is a Kyverno ClusterPolicy that only permits the annotation on pods that explicitly opt in via a label. It ran in Audit mode for two weeks, we watched the drift metric go to zero, then we flipped it to Enforce:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-karpenter-do-not-disrupt
spec:
  validationFailureAction: Enforce
  rules:
  - name: require-opt-in-label
    match:
      any:
      - resources:
          kinds: [Pod]
    preconditions:
      all:
      - key: "{{ request.object.metadata.annotations.\"karpenter.sh/do-not-disrupt\" || '' }}"
        operator: Equals
        value: "true"
    validate:
      message: "karpenter.sh/do-not-disrupt requires label karpenter.platform/opt-in=true"
      pattern:
        metadata:
          labels:
            karpenter.platform/opt-in: "true"

A future chart change that adds the annotation without the matching label is rejected at admission and never lands.

The second is an SLO alert on Karpenter's own consolidation metric. karpenter_disruption_nodes_disrupted_total was already in our Prometheus scrape config; nobody had put it on a dashboard. We added a Grafana panel showing nodes consolidated per hour, filtered to method="consolidation" (the action label holds delete, replace or no-op, never consolidation), and paged if the rate stayed below 20% of the 7-day rolling median for more than two hours, at any hour. Backtesting against the incident, that alert would have fired around 04:00 UTC Saturday, roughly 8 hours in, instead of 60 hours in. On Karpenter v1 those metric names are gone; alert on karpenter_voluntary_disruption_decisions_total{reason=~"underutilized|empty"} instead, which leaves drift out. It counts decisions, not nodes, so a multi-node consolidation counts once.

The third is a check in the shared chart's own CI pipeline. Any change, minor or patch, that touches a scheduling-related annotation key (karpenter.sh/*, karpenter.k8s.aws/*, scheduler.alpha.kubernetes.io/*, node.kubernetes.io/*, cluster-autoscaler.kubernetes.io/*) fails the chart's release job until the platform team approves it, so a version like 2.7.5 cannot be published, and never reaches Renovate, without that review. Non-scheduling changes still flow through the previous auto-merge path unchanged. The tradeoff we now pay is about one platform-team review per quarter on this chart; the incident that check would have caught cost us roughly $3,500 in on-demand overage across the 60-hour window.

We considered tightening the NodePool's disruption.budgets and did not. The root cause was not consolidation being too eager; consolidation was fine when it was allowed to fire. Tightening budgets would slow legitimate consolidation without preventing another annotation-blast. We put a comment in the NodePool YAML pointing to the postmortem so a future engineer does not try to 'fix' the budgets in response to reading it.

Does `karpenter.sh/do-not-disrupt` pin just the pod or the whole node? It effectively pins the node. Consolidation eligibility is evaluated per-node, and any node running at least one annotated pod is disqualified until that pod is gone or loses the annotation. One annotated pod on a 32-vCPU node keeps the node from consolidating for as long as that pod runs there.

Would tightening PodDisruptionBudgets have prevented this? No. Karpenter checks PDBs at the same gate as the annotation: a node whose pods a PDB would refuse to evict is also kept out of consolidation, with its own DisruptionBlocked event. Tighter PDBs add another way to pin nodes; they do nothing about a chart stamping the annotation on every pod.

Can we detect this via CloudWatch or Cost Explorer alone? Not fast enough. Cost Explorer reports daily unless you enable hourly granularity, and our FinOps digest ran weekly. The signal you want is Karpenter's own consolidation rate from /metrics, which moves within minutes of the problem.

Does this pattern apply to Karpenter v1? Yes. In v1 a running pod with karpenter.sh/do-not-disrupt: "true" still blocks Karpenter from voluntarily disrupting its node. Three things differ. From v1.12 the annotation also accepts a duration such as "30m", which protects a pod only for that long after it starts running; earlier releases honour only "true". A NodePool with terminationGracePeriod set may disrupt a node through drift even with such a pod on it. And the annotation does not exclude a node from the forceful methods, expiration, interruption and manual deletion among them, so expireAfter without terminationGracePeriod can leave an annotated pod blocking a node's drain indefinitely. A NodePool that sets neither field gets exactly that: expireAfter defaults to 720h and terminationGracePeriod has no default. The Deployment controller's PodTemplateSpec hash behavior is a Kubernetes property, not a Karpenter property, and it does not change.

What if we want every pod in a Deployment to be do-not-disrupt by design? Put the annotation in the template and accept that changing it triggers a rollout. Use maxSurge and PDBs to control that rollout the way you would for any other template change. The mistake in our incident was not that a template had the annotation; it was that 34 templates got it without the owning teams knowing.

The awkward thing about this class of Karpenter regression is that every piece looks correct in isolation. The annotation was a real feature intended for real workloads. The chart bump was a legitimate response to a real production pain. The Renovate auto-merge policy was tuned to reduce toil on well-behaved chart bumps. The NodePool was configured the way the docs recommend. It took the combination, plus a weekend, plus a weekly cost digest, to produce a nearly doubled node count and a mid-four-figure overage. The diagnostic move that mattered was reading Karpenter's consolidation rate against its own baseline, then its DisruptionBlocked events, which name the pod in the way. The recovery move that mattered was knowing which kubectl mutations trigger a rollout and which do not.

We have written more on this shape of failure in the Kubernetes and CI/CD stabilization pillar, and the specific pattern of a shared chart change cascading through GitOps sits in the ArgoCD and GitOps recovery cluster. If your Karpenter cluster is growing and you cannot see why, book an infrastructure review and we will pull the consolidation counters and DisruptionBlocked events together, read them the same way we read ours, and get you a diagnosis inside a single working day.