Fix migrate.yml auto-applies production DB migrations on push with no concurrency guard or deploy ordering - #2703

Fixes: migrate.yml auto-applies production DB migrations on push with no concurrency guard or deploy ordering

Every push to main that touches a database schema file triggers an automatic migration against the production Turso database — the single shared data store behind the API, background jobs, and every customer status page. Nothing stopped two of those migrations from running at the same time, which can corrupt or deadlock a database that all tenants share, and nothing bounded a stalled migration run. This adds the same concurrency guard and 15-minute job timeout the repository already uses on its other release workflows, so production migrations queue one at a time instead of racing.

What caused this

Affected: int_01b0344f3001ufkixqp40d4t · severity medium

Before / After

❌ Before: Two pushes to main touching schema files within a few minutes ran two production migrations at the same time against the one shared Turso database, with nothing queuing them and nothing bounding a hung run. A migration that stalled held the runner open indefinitely.

✅ After: A second push that touches packages/db/drizzle/** waits in a queue until the in-flight migration finishes, then applies the newest migration set; a manually dispatched run waits behind a push-triggered one the same way. A migration that stalls fails after 15 minutes instead of running forever.

What changed

• .github/workflows/migrate.yml: added a workflow-level concurrency group (migrate-db, cancel-in-progress: false) so overlapping drizzle-triggered pushes, or a push plus a manual dispatch, queue instead of running two migrators against the production database at once.
• .github/workflows/migrate.yml: added timeout-minutes: 15 to the migrate job, matching deploy.yml, deploy-checker.yml, deploy-private-location.yml and restart-workflows.yml, so a hung migration fails fast and cannot hold a runner or block queued runs indefinitely.

Why it's safe

• The concurrency group cannot starve the queue: cancel-in-progress: false means every queued run still executes, only later; nothing is dropped.
• Push-triggered runs check out the triggering SHA, so a queued run applies a newer migration set after an older one — exactly the ordering drizzle's journal expects; queueing preserves correctness rather than risking it.
• The static group name means dispatches serialize with pushes intentionally: every run targets the same production DATABASE_URL secret, so there is no legitimate case for two concurrent runs.
• The 15-minute timeout matches the value already in production on the sibling deploy workflows; recent migrations completed well under 5 minutes, so the bound only affects the pathological case.
• No application code, secrets, or deploy steps are touched — the change is inert until the next drizzle-path push, and single-run behavior is identical to today's.

Validation

• pnpm verify (oxfmt + oxlint + doc refs + deno check, all 40 workspace packages): 40/40 tasks successful, run in this thread after pnpm install --frozen-lockfile.
• actionlint .github/workflows/migrate.yml (v1.7.7): clean, no findings.
• YAML structure parse of the edited workflow: concurrency: {group: migrate-db, cancel-in-progress: false} and timeout-minutes: 15 verified present, on: triggers unchanged.
• pnpm verify:test: not run — the change touches only a CI workflow file, no package code or tests; the repo's test suites need a live libSQL database, which this change does not affect.

Detection outcome

The fingerprinted signal is the confirmed issue "migrate.yml auto-applies production DB migrations on push with no concurrency guard or deploy ordering" (iss_08ecdd893001h972tpc1fo8x). Once this deploys, the unguarded-migration aspect it names stops being true: every Migrate DB run — push-triggered or dispatched — joins one migrate-db concurrency group, so a second run queues instead of racing the first against the production Turso DB, and a stalled run fails at 15 minutes instead of holding open indefinitely. The signal's remaining aspects (migrate-vs-deploy interleaving and unvalidated migrations reaching main) are unchanged by design and are recorded as follow-ups on the issue timeline, not silenced here.

Fix chosen

Chosen: cause (removes the mechanism that produces the failure): Add a workflow-level concurrency group (migrate-db, cancel-in-progress: false) and a 15-minute job timeout to migrate.yml, serializing all production migration runs against each other.

Considered and not chosen:

• cause (removes the mechanism that produces the failure): Gate the deploy on migration success: a migrate step or needs relationship inside deploy.yml, or a workflow_run chain that deploys only after Migrate DB succeeds. Not chosen: Changes the release model and deploy latency for every push to main — a maintainer decision, recorded as a follow-up on the issue timeline rather than made unilaterally in this PR.
• cause (removes the mechanism that produces the failure): Apply migrations to a staging Turso first and auto-run prod migrate only on staging success. Not chosen: Requires provisioning a staging Turso instance and new repo secrets that do not exist; an infrastructure decision for the team, not a workflow-file edit.
• disable (turns a feature, guard, check, test, or telemetry off): Drop the push trigger so migrations are only ever run manually via workflow_dispatch. Not chosen: Removes the automation entirely and changes the release process unilaterally; the concurrency guard keeps the automation and removes only the race.

Outcome after fix

Replay the 09-10 burst: pushes land on main at 12:27Z, 12:39Z and 12:40Z, the first two touching packages/db/drizzle/**. Today, the 12:27Z push starts a migrate run immediately and the 12:39Z push starts a second one while the first may still hold the lock window — two drizzle migrators reading the same __drizzle_migrations table and racing to apply the same pending SQL against the production Turso DB. With this change: the 12:27Z run joins migrate-db and runs; the 12:39Z run enters the same group, is marked queued, and GitHub starts it only after the first completes, at which point it applies the newer migration set against a DB already at the older state. If instead a migration hangs (e.g. a network partition to Turso), the job now fails at 15 minutes and the queued run proceeds, where before it held the runner indefinitely and nothing behind it could advance. The deployment itself is untouched: deploy.yml still rolls the API on the same push, as before — that interleaving is explicitly left open on the issue timeline.

Repository conventions present in the repository: AGENTS.md, CLAUDE.md, apps/checker/AGENTS.md, apps/dashboard/AGENTS.md, apps/server/AGENTS.md, apps/status-page/AGENTS.md, apps/web/AGENTS.md, apps/workflows/AGENTS.md, packages/services/AGENTS.md, packages/ui/AGENTS.md.

Repository lint: pnpm run lint (declared in package.json) passed over the whole repository (the declared target takes no file arguments).

Generated by Polylane.