Fix All three Vercel production pointers stuck on CANCELED builds since 09-10; deploy-activity monitoring blind to it - #2706

Fixes: All three Vercel production pointers stuck on CANCELED builds since 09-10; deploy-activity monitoring blind to it

Since 09-10 16:12Z all three production targets have pointed at CANCELED builds and no signal noticed: the docs-only commit triggered turbo-ignore skips on every project, Vercel moved the production pointers onto the canceled deployments, and production silently kept serving the previous build. Failing builds are just as silent as skips — nothing in the repo observes whether a push actually deployed. This adds an hourly watchdog that reads the GitHub deployments API Vercel already writes to and fails CI when production is not shipping.

flowchart LR
    A["push to main"] --> B{"Vercel GitHub app"}
    B -->|"affected app"| C["build READY"]
    B -->|"turbo-ignore: not affected"| D["build CANCELED"]
    C --> E["status success"]
    D --> F["status inactive - Skipped"]
    E --> G["production serves new build"]
    F --> H["production serves prior build"]
    E --> W["watchdog (hourly)"]
    F --> W
    G --> W
    W -->|"green or benign skip"| I["check passes, summary table"]
    W -->|"failure status or no deployment"| J["check fails, annotation names project"]

What caused this

Affected: acc_01b04ba9f00122r4mmzzgv9q · severity medium

Before / After

❌ Before: a push to main either deploys or silently doesn't, and nothing in the repo can tell the difference. Docs-only commits skip all three Vercel builds; broken commits fail them — both leave production serving an older build with every visible signal reading healthy, as it has since 09-10 16:12Z.

✅ After: the watchdog runs hourly and writes a per-project table of which build production is actually serving. If a production build fails, a push never reaches Vercel, or a build gets stuck, the check run fails with a named annotation, and anyone reading CI sees production is behind. The skip-on-docs-commit case stays green and visible in the table.

What changed

• .github/workflows/vercel-production-watchdog.yml (new): hourly scheduled workflow (plus workflow_dispatch), contents: read + deployments: read permissions, only gh and the GitHub REST API — no new secrets. Per project it reads the Vercel-created deployments under Production – and: (1) finds the newest success deployment = what production actually serves; (2) alerts on newer deployments with failure/error status; (3) alerts if a main push past a 30-minute grace has no Vercel deployment (broken webhook/integration); (4) alerts if the newest build sat in queued/in_progress/pending over 60 minutes. Writes a step-summary table per run; skips stay visible but green.

Why it's safe

• Read-only by construction: the job requests contents: read and deployments: read and issues gh api GETs only — it cannot write, deploy, or revert anything; a bug in the script cannot damage production.
• No false-page on the known-benign pattern: the turbo-ignore skip posts inactive, deliberately treated as not-an-alert (the table records what production serves; only failure/error statuses, a dead push, or a stuck build fail the check) — the pattern reviewed impact-none on every prior docs-only wave.
• The en dash in "Production – " is byte-verified U+2013 matching GitHub's environment names, and a zero-row filter result is itself alarmed, so even a Vercel rename surfaces within an hour rather than failing silently.
• sort_by(.id) (not created_at) picks the terminal status even when two statuses share a timestamp, and set -euo pipefail makes any API failure exit non-zero rather than half-evaluating into a false green.

Validation

• oxfmt --check: exit 0, all files correctly formatted (oxfmt targets JS/TS; the YAML is out of scope by design).
• oxlint repo-wide (declared lint): exit 0, 38 pre-existing warnings, 0 errors; no lintable files in the YAML.
• node scripts/check-doc-refs.mts (check:docs leg of pnpm verify): exit 0, no dangling refs in 4171 files.
• YAML parsed via js-yaml: OK; embedded bash extracted and bash -n: syntax OK, before and after the final edit.
• En dash in the environment-name literal byte-verified U+2013, matching GitHub's live environment names.
• Live replay against the real API for f700a85, a6f2b62, c51014f, 10dbb48: the inactive/"Skipped - Not affected" and failure states the logic branches on all observed.
• not run: pnpm test/verify:test — no package code touched; suites need a seeded local database irrelevant to a workflow YAML.

Detection outcome

The finding's signal: "all three Vercel production pointers stuck on CANCELED builds; deploy-activity monitoring blind to it" — the family of production silently not shipping. Once merged, the hourly run re-evaluates that condition against the GitHub deployments API. The pure-skip state stays a summary row (expected turbo-ignore behavior), but the blindness stops: a failing production build (the a6f2b62 dashboard wave), or a main push that never created a Vercel deployment, now fails a visible check run within an hour instead of persisting undetected for 19 hours. The mechanism removed: nothing in the repo read the only deploy-outcome signal that exists (Vercel's GitHub deployment statuses); now one does, hourly, with no new secrets.

Fix chosen

Chosen: cause (removes the mechanism that produces the failure): Add a scheduled watchdog workflow reading the GitHub deployments API Vercel already populates, failing CI on the silent modes: production builds failing while production serves an older build, pushes that never created a Vercel deployment, and builds stuck in the queue.

Considered and not chosen:

• suppress (silences, downgrades, or reroutes the signal without changing what produces it): Alarm directly on the stale productionDeploymentId / CANCELED-pointer state. Not chosen: Alarming on the CANCELED-pointer state itself would fire on every docs-only commit — expected turbo-ignore behavior, already reviewed impact-none on prior waves — teaching the team to ignore the alert while leaving real failure modes unwatched.
• loosen (raises a limit, threshold, or timeout, or widens a retry): Turn off turbo-ignore / enableAffectedProjectsDeployments so every commit builds all three apps. Not chosen: Disabling turbo-ignore forces full production builds on every push — burning build minutes to ship no-op changes — to fix what is an observation gap, not an over-eager skip; the skip is intentional monorepo design.
• cause (removes the mechanism that produces the failure): Watchdog querying the Vercel REST API for productionDeploymentId + readyState per project. Not chosen: No workflow in the repo holds a VERCEL_TOKEN secret (verified), and the workspace's Vercel account is read-only, so it cannot run without an admin first creating a secret — a dependency the change cannot provision itself. The GitHub deployments API carries the same outcome data with permissions the repo already grants.
• capacity (shrinks or reshapes the thing that exceeds a limit so the feature keeps working): Fix the workspace-side monitoring: change deployments_count to count only production-targeted deployments. Not chosen: Fixing the workspace-side deployments_count key query means editing the platform-managed key-query store, unreachable from the repo (established in prior workspace escalations), so it cannot ship as a PR from this run.

Repository conventions present in the repository: AGENTS.md, CLAUDE.md, apps/checker/AGENTS.md, apps/dashboard/AGENTS.md, apps/server/AGENTS.md, apps/status-page/AGENTS.md, apps/web/AGENTS.md, apps/workflows/AGENTS.md, packages/services/AGENTS.md, packages/ui/AGENTS.md.

Repository lint: pnpm run lint (declared in package.json) could not run in the sandbox because its tool is not installed there; run it before merging.

Generated by Polylane.