Fix openstatus-api auto_suspend + min_machines_running=1 makes the public API a single point of failure during traffic lulls - #2701
Open
polylane[bot] wants to merge 1 commit into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 3 Skipped Deployments
|
vercel
Bot
temporarily deployed
to
Preview – openstatus-status-page
September 10, 2026 16:37
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: openstatus-api auto_suspend + min_machines_running=1 makes the public API a single point of failure during traffic lulls
The openstatus-api Fly fleet is configured to suspend idle machines toward
min_machines_running = 1. During traffic lulls, machines in a region suspend and requests landing in that window are rejected with 503 by the edge, and the 2-region fleet collapses to a single serving machine — a single point of failure that escalates to a full outage when a lull coincides with a degraded instance. This change turns auto-stop off and raises the machine floor to 2, so one machine stays serving in each region.What caused this
Affected:
acc_01b058a86001tjsvwd8nll0s· severity mediumWhat changed
apps/server/fly.toml: setauto_stop_machines = "off"andmin_machines_running = 2in the[http_service]block so the public API never suspends below one machine per region.Why it's safe
flyctl deploy; it cannot break the application build or a request path.min_machines_runningto 2 costs one extra machine-hour per region (a shared-4/1GB VM in iad and ams) but restores a real per-region serving machine instead of a suspendable single machine; no new external dependency is introduced.auto_stop_machines = "off"matches the repo's own convention for always-on public workloads (workflows, checker, private-location), so the behavior is already proven in this fleet.GET /ping) and concurrency limits are untouched, so Fly's existing readiness gating continues to apply.Validation
python3 -c "import tomllib; ..."onapps/server/fly.toml: parses,autostop: off | min: 2pnpm format:check: 0 errors (38 pre-existing warnings, none in the changed file)pnpm check:docsandturbo run check:denois not installed in the sandbox, so the Deno-based package checks fail withspawn ENOENT— unrelated to this TOML-only change; no TypeScript or Deno file was touchedRoot cause and scoping notes
Root cause
The public API's Fly config (
apps/server/fly.toml, deployed verbatim by.github/workflows/deploy.ymlviaflyctl deploy --config apps/server/fly.toml) setsauto_stop_machines = "suspend"withmin_machines_running = 1. Live machine683e3d1c400d78confirms the deployed fleet still carries"autostop":"suspend","autostart":true,"min_machines_running":1.Mechanism: during a traffic lull, Fly proxy-initiated suspension cycles begin (documented 02:40–04:10Z on 2026-09-10, ~30s suspend→cancel→start→uncordon across all four machines), and requests landing in a suspend window get 503 from the edge. With only one machine kept running fleet-wide, a single region's suspension leaves one machine serving the whole 2-region (iad + ams) fleet — a single point of failure. This produced a documented 3.56% edge server-error deviation and contributed to the 03:20–05:19Z episode where monitor 771 saw 503s from all 18 Fly regions.
The fix disables auto-stop and raises the floor to 2 (one per region), matching the sibling always-on apps in this repo (
apps/workflows/fly.toml,apps/checker/fly.toml,apps/private-location/fly.tomlall useauto_stop_machines = "off"). This removes the suspend-cycle race at its producer rather than tolerating the resulting 503s.Causal chain
Detection outcome
The edge server-error ratio deviation and suspension-induced 503s stop firing once this change deploys. The signal's producer is the
auto_stop_machines = "suspend"+min_machines_running = 1combination inapps/server/fly.toml; changing those to"off"and2removes the suspend cycle that generated the edge rejections, so requests no longer land in a suspend window and each region keeps a started machine serving.Fix chosen
Chosen: cause (removes the mechanism that produces the failure): Disable auto-stop and raise min_machines_running to 2 in apps/server/fly.toml, removing the suspension mechanism that produces 503s during lulls.
Considered and not chosen:
Outcome after fix
Replaying the failing scenario with the change applied:
min_machines_running = 1; requests landing mid-cycle got 503 from the edge, and a region's suspension left one machine serving the fleet.auto_stop_machines = "off", no suspend cycle is initiated, so no request lands in a suspend window and the edge has a started machine in each region at all times.min_machines_running = 2, even a machine failure or a mid-deploy swap leaves one started machine per region serving, so the fleet never collapses to a single machine.1 file changed (+2/-2)
apps/server/fly.toml: modified, +2/-2Repository conventions present in the repository: AGENTS.md, CLAUDE.md, apps/checker/AGENTS.md, apps/dashboard/AGENTS.md, apps/server/AGENTS.md, apps/status-page/AGENTS.md, apps/web/AGENTS.md, apps/workflows/AGENTS.md, packages/services/AGENTS.md, packages/ui/AGENTS.md.
Repository lint:
pnpm run lint(declared in package.json) could not run in the sandbox because its tool is not installed there; run it before merging.Generated by Polylane.