Clear, opinionated recovery notes for SaaS infrastructure
This is the discovery library for teams who want to understand failure patterns, recovery logic, and the kind of review InfraForge runs before they submit.
Read by issue cluster, not by publish order
The useful question is not "what was published last?" It is "which problem class matches the pressure the team is under right now?"
- Terraform and IaC reliability.
- Kubernetes, GitOps, and release stability.
- Migration recovery, audit readiness, and control design.
- If the failure pattern is already familiar, request the review.
- If you need proof, use case studies next.
- If you just need the checklist, download the PDF directly.
Featured recovery notes
A short set of strong entry points before you go broader.
Browse the full library
How to prove a Kubernetes ServiceAccount is unused before deleting it
An auditor sampled 8 privileged ServiceAccounts and 4 were silent for 148 days. Here is how to prove non-use from audit logs and revoke safely.
How to recover a Helm 3 release stuck in pending-upgrade
A Helm release stuck in pending-upgrade blocks helm upgrade but not helm rollback. Roll back first, and send a pending-install on v1 to helm uninstall.
Why S3 traffic goes through NAT: the missing endpoint route
Same-region S3 reads billed as NAT gateway data processing for 22 days. How we traced 1.9 TB a day to an EKS subnet missing the gateway endpoint route.
Every terraform state command and when each is safe to run
A risk-ranked reference for terraform state pull, push, rm, mv, import and the init flags that decide which backend you are actually talking to.
How a do-not-disrupt annotation broke Karpenter consolidation
A shared chart stamped karpenter.sh/do-not-disrupt on every pod template and consolidation stalled cluster-wide. Here is how we found it and rolled it back.
Worker drops jobs intermittently: startup race, env drift, schema skew
Worker silently skips jobs and sometimes crashes with Connection refused? The three causes in order of frequency, the discriminating check, and the safe fix.
How a missing S3 gateway endpoint route quintupled our NAT bill
Our NAT gateway bill jumped from $400 to $2,100 because a private subnet moved onto a route table with no S3 gateway endpoint route. How we found it.
How one stuck PDB doubled our EKS autoscaler bill in six days
An EKS cluster autoscaler held 40 idle m5.4xlarge nodes for six days because one PDB blocked eviction. Here is how we traced it and what we changed.
Recovering a status page from a half-finished schema migration
How we recovered a status page whose Postgres schema was stuck mid-migration after its pod was OOMKilled on startup. Neither Helm rollback nor a re-run worked.
When a hardening rollout breaks 8 layers and your own reconciler fights you
How we recovered five stuck apps after a hardening rollout broke hooks, RBAC, PVCs, quota, and egress while the in-house reconciler fought every patch.
When a validating webhook blocks the ConfigMap that would fix it
A validating admission webhook with failurePolicy Fail crash-looped and blocked the ConfigMap update that would have unblocked the outage. Here is how we got out.
Why a deleted backup Lambda kept billing 9,400 EBS snapshots
Almost all of a $1,830 EBS Snapshot line came from a Lambda deleted a year earlier. How we found the 9,408 orphans, removed them safely, and the tagging rule we wrote.
Why one shared Terraform module made every PR a 14-service change
A consolidated Terraform module turned every PR into 14 service plans with 1,400 resource changes. How we pinned, tiered, and split it back apart.
Why a Terraform apply hangs 90 minutes on a custom provider with no timeout
A 200-entry destroy hung for 90 minutes because a custom Terraform provider skipped context timeouts. How we recovered the half-finished destroy and fixed the provider.
Grafana 'No Data' after migration: 7 reconcilers we had to kill first
Grafana dashboards went blank post-migration and every fix reverted in minutes. Here is how we found the reconcilers and restored the observability stack.
When MinIO Deny Wins Cause Silent Upload Failure
An inverted Deny in a MinIO IAM policy got 12k uploads rejected with 403, and the app swallowed every error and returned 200. How we found it, and the alerts we added.
ArgoCD Drift: Three Namespaces, One JWT Hotfix
A JWT rotation hotfix left three ConfigMaps in three different states and Git stale. Here is how we found the canonical truth and committed it back without breaking auth.
Why terraform plan fails three modules from the cause: the map(any) trap
A 15th map(any) input collided with an existing key three module layers down. The error fired at plan, three files from the code that caused it. Here is how we traced it and untangled the root.
Why a forgotten RDS replica added $8,600 to one AWS bill
A read replica left over from a load test cost $8,600 in instance-hours nobody was using, reported ReplicaLag = -1 for six days, and put nothing at all on the cross-AZ transfer meter. Here is why.
Three init container faults when every kubectl patch reverts in 10 seconds
Three init containers failed one after another and every kubectl patch reverted within ten seconds. Here is how we found the source of truth and fixed it.
Infrastructure change control checklist for audit-ready SaaS teams
A practical checklist for approvals, validation evidence, rollback discipline, and audit-ready change control.
GitOps drift triage checklist for production teams
A fast triage sequence for classifying GitOps drift, comparing rendered output, and restoring sync trust.
Migration blast radius mapping framework for SaaS platforms
A practical framework to map hidden migration dependencies and contain post-cutover reliability risk.
ArgoCD sync failed recovery playbook for production teams
A recovery sequence for repeated ArgoCD sync failures, drift reconciliation, and safer release flow.
Terragrunt vs Terraform for growth-stage SaaS: decision framework
A practical decision framework for selecting IaC structure based on ownership and change safety.
Audit evidence pack for SaaS infrastructure teams
A practical evidence-pack structure for audit readiness without slowing product delivery.
Cloud cost spike triage framework for engineering leads
A systems-first triage flow to isolate spend drivers and reduce cost safely.
Kubernetes release stabilization runbook
A practical runbook to make rollouts deterministic and rollback paths reliable.
Post-migration stabilization checklist for SaaS teams
A 30-day stabilization sequence for teams whose platform got shakier after migration.
Terraform drift detection workflow teams actually maintain
A practical drift detection workflow with ownership, triage, and reconciliation rules that hold up under pressure.
Terraform module refactor strategy for growth-stage SaaS
A phased module refactor strategy that lowers coupling and avoids production disruptions.
Failed Terraform apply incident response checklist
A practical incident sequence to contain impact, reconcile state, and prevent repeat failures.
Terraform state recovery playbook for SaaS teams
A practical sequence to repair state trust, reduce blast radius, and restore predictable infrastructure changes.
Safe Terraform apply guardrails for production SaaS
A guardrail system for CI/CD, review, and rollback that makes Terraform applies boring again.
Infrastructure review checklist for SaaS teams under pressure
A fast decision guide for when to request a review and what to prepare so the response is actionable.
Terraform drift recovery: stabilize IaC without stalling delivery
A practical recovery plan for drift, fear-of-apply, and brittle modules with guardrails that last.
Prefer the PDF?
Use the checklist when you want a short review aid without reading through the full article library.
The Infrastructure Review Checklist is public and ready to download.
Use it to map critical paths, drift signals, release safety questions, and evidence gaps before you request the review.