Recovering a status page from a half-finished schema migration

The chart deploys with the Recreate strategy, and the 0.91.2 binary runs its own migrations on startup, so the upgrade had killed the 0.90.78 pod before the new one began migrating. Helm recorded the upgrade as complete as soon as it had applied the manifests; it does not wait for pods unless asked to. The on-call had done the obvious thing first. The new chart was failing, so they ran helm rollback to the previous revision, revision 13, the last one before 0.91.2. The previous revision's pod came up, hit the database, and crashed too, for a different reason. The new binary expected schema 23 and found the version row marked dirty mid-migration. The old binary expected 21 and found the row already advanced to 23. Each binary checks the version row itself before handing off to golang-migrate, and a row newer than it knows is refused before the dirty flag is even read. One saw a migration that never finished, the other saw a database from the future. Both were sort of right.

A rollback reverts the chart, not the database, so for a binary with a strict version check, like this one, the old binary refusing a newer schema is what any rollback after a forward migration looks like. The new binary refusing too is what told us the database was the problem, not the chart: if both binaries reject the same database, it is in neither of the states they expect. It is in a third state nobody coded for.

$ kubectl logs -n statuspage statuspage-app-7b9f-xq2vk   # 0.91.2 pod, pre-rollback
INFO  starting statuspage v0.91.2
INFO  connecting to postgres at postgres.statuspage.svc:5432
ERROR database schema version 23 is dirty, refusing to start
FATAL refusing to start with a dirty schema version

$ helm rollback statuspage 13 -n statuspage   # 13 is the last revision before 0.91.2
Rollback was a success! Happy Helming!

$ helm history statuspage -n statuspage
REVISION  UPDATED                   STATUS      CHART               APP VERSION  DESCRIPTION
13        Tue May 12 09:12:04 2026  superseded  statuspage-0.90.78  0.90.78      Upgrade complete
14        Fri May 15 14:31:47 2026  superseded  statuspage-0.91.2   0.91.2       Upgrade complete
15        Fri May 15 14:48:12 2026  deployed    statuspage-0.90.78  0.90.78      Rollback to 13

$ kubectl logs -n statuspage statuspage-app-6c4d-7m9pz   # rolled-back 0.90.78 pod
INFO  starting statuspage v0.90.78
ERROR database schema version 23 found, expected 21
FATAL refusing to start with schema version mismatch

Different errors from the two chart revisions, same root cause: the database, not the chart, was the problem.

We dropped into psql against the application database and pulled the migration tracking table. The row said version 23 with the dirty flag set, which in golang-migrate means 'a migration to this version started and never reported success'. That single boolean was the thread we pulled on for the rest of the recovery.

statuspage=> select * from schema_migrations;
 version | dirty
---------+-------
      23 | t
(1 row)

statuspage=> \d incidents
                         Table "public.incidents"
   Column    |            Type             | Collation | Nullable | Default
-------------+-----------------------------+-----------+----------+---------
 id          | bigint                      |           | not null |
 service_id  | bigint                      |           | not null |
 started_at  | timestamp without time zone |           |          |
 resolved_at | timestamp without time zone |           |          |
 title       | text                        |           |          |
Indexes:
    "incidents_pkey" PRIMARY KEY, btree (id)
-- expected per migration 0023: severity column, incident_updates FK, partial index on resolved_at IS NULL
-- present: none of the above

The version row said 23 was in progress. The table structure had none of 23.

We pulled the migration files out of the app image and read them. Migration 0023 was three statements: add a severity column, create an incident_updates table with a foreign key back, create a partial index on unresolved incidents. None of the three were present in the live schema. The OOM had hit after the migration library wrote the version row and before it sent 0023's body to the server. golang-migrate writes version 23 with dirty=true first, runs the body, then clears the flag, and on Postgres a multi-statement body sent in one Exec, the default, runs as a single transaction, so there is no stopping between statements: the body lands whole or not at all. Here it had not landed. That timing is inferred from the state, not observed: a kill after the body reached the server would more likely have left 0023 committed and still marked dirty, and the same IF NOT EXISTS repair covers that case too. None of 0023's DDL was present, and the bookkeeping said 23 was in progress.

This is the specific failure mode that makes partial migrations dangerous. The migration library and the actual schema disagree, and the application trusts the library. The library trusts a row it wrote in a different transaction than the DDL it was supposed to be tracking. Whether the version row and the DDL share a transaction is a per-tool design choice, not a version threshold you can date. The version/dirty pair in the table above is golang-migrate's, and golang-migrate deliberately writes version=N with dirty=true in its own transaction, runs the migration body separately, then writes dirty=false. The dirty column exists precisely because those two are not atomic, and that has not changed. Any tool that ships a dirty flag is telling you the same thing about itself. At the other end, a tool that wraps each migration and its history row in one transaction on Postgres, Flyway among them, does not leave the state described here. Check which one you are running before you trust the version row, because that is what decides whether this failure mode can reach you.

The instinct, and the safe move on most days, is to restore Postgres from the last known-good base backup and replay WAL up to a point just before the migration started. We checked the backup. It was a daily pg_basebackup taken at 08:30, 6 hours old, and WAL archiving had been configured but never tested for PITR. We could probably have done it. We were not willing to bet the status page on 'probably' while the status page was already down.

More importantly, the uptime check history is the product. A PITR replay to just before the migration would have cost only the few minutes since the migration started, but if the untested WAL replay failed we would be down to the base backup alone, and a status page that loses 6 hours of check data after an outage is worse than a status page that takes another hour to come back. We talked it through with the team lead and decided the database in front of us was recoverable, and recovering it was lower risk than the restore path. That decision is worth naming because it goes against the usual 'just restore from backup' instinct. When the data itself is the value, finishing a half-migration by hand is often the right call.

We pulled migration 0023 verbatim from the app image, rewrote each statement with IF NOT EXISTS guards so a re-run could not double-apply, and ran them inside a single transaction so any failure left the database where we found it. Before touching anything we took a pg_dump of the application schema and data to a local file. That dump was our 'we can always undo this' insurance, separate from the production backup system.

-- 1. snapshot first, outside any transaction
$ kubectl exec -n statuspage postgres-0 -- \
    pg_dump -U statuspage -Fc statuspage > /tmp/statuspage-pre-repair.dump

-- 2. complete migration 0023 inside one transaction
BEGIN;

ALTER TABLE incidents
  ADD COLUMN IF NOT EXISTS severity smallint NOT NULL DEFAULT 3;

CREATE TABLE IF NOT EXISTS incident_updates (
  id           bigserial PRIMARY KEY,
  incident_id  bigint NOT NULL REFERENCES incidents(id) ON DELETE CASCADE,
  body         text NOT NULL,
  created_at   timestamp without time zone NOT NULL DEFAULT now()
);

CREATE INDEX IF NOT EXISTS incidents_unresolved_idx
  ON incidents (started_at)
  WHERE resolved_at IS NULL;

-- 3. clear the dirty flag, version row already says 23
UPDATE schema_migrations SET dirty = false WHERE version = 23;

-- 4. sanity-check before commit
SELECT version, dirty FROM schema_migrations;
\d incidents
\d incident_updates

COMMIT;

All four statements committed in 180ms: the three DDL statements plus the version-row correction. The read-only checks below them commit nothing. The dirty flag was the last thing to flip.

After the commit we rolled forward to the chart whose binary expects 23, because the deployment sitting at 0 replicas was still the rolled-back 0.90.78 one and it would have refused a clean schema 23 exactly the way it refused a dirty one: helm upgrade statuspage statuspage/statuspage --version 0.91.2 --reset-then-reuse-values --set replicaCount=0 --set resources.limits.memory=512Mi. Every flag matters. Helm carries a release's existing values forward only when you pass no new ones, so a bare --set would have reset every custom value in the release, the database host and secrets included, to the chart's defaults. --reuse-values is not the answer across a chart version either: it carries the old release's computed values over the new chart's own defaults, so values 0.91.2 introduced can be lost, while --reset-then-reuse-values (Helm 3.14 or later) starts from 0.91.2's defaults and applies the release's values on top. The memory limit went up because the startup warm-up that killed the first 0.91.2 pod would have killed this one too. And replicaCount=0 is what kept the gate. We had scaled the deployment to 0 earlier to stop the CrashLoopBackOff noise, but kubectl scale only changes the live object, and Helm's three-way patch compares the old manifest, the live state and the new manifest, so a plain upgrade would have put the replica count straight back and started a pod before anyone looked at anything. Only then did we admit traffic, with the same pair of flags, helm upgrade statuspage statuspage/statuspage --version 0.91.2 --reset-then-reuse-values --set replicaCount=1, so the stored value is 1 and the next upgrade does not quietly scale the page back to zero. We watched the pod logs. It connected, ran its startup migration check, found 23 clean with nothing to apply, and started serving. We curled the health endpoint, got a 200, then hit /api/services and confirmed all the configured uptime checks were present with their full history intact. Total time from 'database is the problem' to 'status page is back': 51 minutes.

The thing worth saying out loud about this kind of repair: it works because the migration was small and the failure was clean. If 0023 had been a long data migration, or had been split out of its single transaction, the recovery would have looked very different and the backup-restore path would have won. Always read the failed migration before you decide which recovery to attempt. We have written more about that decision in the migration recovery playbook.

Two changes went in within 24 hours. First, the Helm chart now has a pre-upgrade hook that writes two dumps to an object storage bucket, tagged with the chart version it is about to migrate to: pg_dump --schema-only for the structure, plus pg_dump --data-only --table=schema_migrations for the version and dirty values. Both are needed, because --schema-only emits DDL only and dumps no table rows at all, so on its own it would capture everything except the version/dirty pair, which is the single field this recovery turned on. If a future migration goes sideways, the recovery starts from a known structure-level snapshot taken seconds before the migration began, not from the daily backup. The hook adds about 4 seconds to every upgrade and has paid for itself once already.

Second, migrations no longer run at app startup. They run in a pre-upgrade hook Job, weighted to run after the dump hook, with its own requests and limits. Helm waits for a Job hook, up to its --timeout, and fails the release if the hook fails, so a failed migration now stops the upgrade before the Deployment is touched, and the pods already running keep serving, as long as nothing restarts them, instead of being gone before the migration starts. Set that timeout above your longest migration. The OOM itself happened because the app's 256Mi limit was sized for 0.90.78, and 0.91.2 loads its check history into memory at startup; against production's history that peak reached about 380Mi, which is why the roll-forward raised the limit to 512Mi. The Job has no liveness probe: a migration that runs long should finish or fail on its own, not be killed halfway by a probe, because any kill after the version row is written leaves it dirty.

flowchart TD
  A[helm upgrade] --> B[pre-upgrade hook]
  B --> C[pg_dump --schema-only + pg_dump --data-only<br/>-t schema_migrations, to bucket<br/>tagged chart-version target]
  C --> D[migration Job, a pre-upgrade hook<br/>weighted after the dump<br/>own limits, no liveness probe]
  D -->|success| E[app deployment rolls]
  D -->|fail or OOM| F[release fails before the Deployment changes<br/>snapshot exists, recovery starts from known state]
  F --> G[finish by hand OR<br/>restore from base backup + replay WAL]
  H[daily pg_basebackup + archived WAL<br/>separate input, the only PITR target] --> G

The shape of every upgrade now. The branch on the right is the one that has paid for itself since. The logical schema snapshot is a reference for what the structure should look like; only the physical base backup can take a WAL replay.

We have stopped recommending that teams skip the pre-migration schema snapshot just because their database is 'small enough to restore from the daily backup'. The two answer different questions. A base backup plus WAL tells you what the data looked like at a chosen moment, provided the replay works. The schema snapshot tells you what shape the database was in seconds before this specific migration began, which is the thing you need when a migration is the thing that broke.

The reason this kind of incident is hard is not the SQL. The SQL is usually three or four statements you can read off the migration file. The hard part is deciding whether to finish by hand or restore from backup, and that decision depends on details most teams have not catalogued: how their migration library handles the version row, whether PITR has ever been tested, whether the failed migration is structural or data-rewriting, and whether the data between the last backup and now is recoverable some other way.

We run these recovery engagements regularly. We have seen the OOMKilled-migration pattern four times in the last year, two of them on status pages or monitoring tools where the data IS the product, and we have a checklist for the decision now. If you are staring at a CrashLoopBackOff with a schema version mismatch in the logs and a Helm rollback that did not help, book an infrastructure review and we will be on a bridge with you the same day to work through the finish-by-hand versus restore decision before you commit to either.