Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix(coderd): reject API keys of soft-deleted users during authentication - #28634

Draft
ThomasK33 wants to merge 7 commits into
mainfrom
fix-deleted-user-api-auth
Draft

fix(coderd): reject API keys of soft-deleted users during authentication#28634
ThomasK33 wants to merge 7 commits into
mainfrom
fix-deleted-user-api-auth

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Aug 26, 2026

Copy link
Copy Markdown
Member

Problem

An api_keys row can outlive its user: the insert-vs-soft-delete race (#28538) can leave a key committed after delete_deleted_user_resources() ran, and restored backups or inserts that bypass trigger_insert_apikeys reconstruct the same state. Such an orphaned credential currently authenticates normally, because role resolution does not consider users.deleted.

Change

Authentication rejects credentials of soft-deleted users, and every other UserRBACSubject consumer handles the deleted-user sentinel explicitly:

  • GetAuthorizationUserRoles returns the user's deleted flag. It deliberately does not grow a WHERE users.deleted = false filter: the query is :one, and its non-authentication consumers (provisionerdserver job acquisition, dynamic parameter rendering) must keep resolving roles for a soft-deleted owner, otherwise every build for that owner fails permanently, including the delete build that reclaims the workspace, with recovery requiring manual DB surgery. The query comment now names those dependents so the contract survives refactors.
  • httpmw.UserRBACSubject returns a sentinel ErrUserDeleted for soft-deleted users, so every subject construction fails closed.
  • httpmw.ValidateAPIKey maps ErrUserDeleted to 401 (signed-out), not 500: a deleted user's key is a correctly rejected credential, not a server error. The 401 is a soft failure, so optional-auth routes (public workspace apps) treat the stale cookie as anonymous rather than locking the browser out.
  • The rejection happens before every write, including the OIDC/GitHub token refresh: a request the server is about to reject no longer calls the IdP with the deleted user's refresh token, rewrites user_links, bumps the deleted user's last_seen_at, or re-fires the cleanup trigger as a side effect of whoever holds the stale token.
  • httpmw.ExtractWorkspaceAgent maps ErrUserDeleted to 401 so an agent whose owner was soft-deleted stops retrying instead of hammering a 500.
  • The remaining UserRBACSubject call sites route the sentinel explicitly rather than surfacing opaque user is deleted errors: OAuth2 token grants return RFC 6749 invalid_grant; chatd and chat tool paths (reachable because chats are not purged on soft-delete) fail closed with explicit "chat owner has been deleted" responses; token create/config endpoints return 400/404 for a deleted target user; the two sites where a deleted user is unreachable (password login) or already fails closed (signed chat-file downloads) are documented in place.

Tests

  • TestAPIKey/DeletedUser reconstructs the orphaned-credential state (soft-delete with triggers suppressed via SET LOCAL session_replication_role = replica) and asserts the 401 signed-out rejection, that the failure is soft (an optional-auth route continues unauthenticated), and that the rejected request leaves api_keys.last_used/expires_at and users.last_seen_at untouched.
  • TestAPIKey/DeletedUserExpiredOAuth gives the deleted user a valid OIDC key with an expired OAuth link and asserts the middleware rejects before the refresh: zero IdP calls and an untouched user_links row.
  • TestWorkspaceAgent/DeletedOwner asserts an agent whose owner was soft-deleted receives a terminal 401, not a retryable 500.
  • TestGetAuthorizationUserRolesDeletedUser pins the query contract: deleted users still resolve roles, with deleted set, so the provisioning consumers stay unaffected.

Relationship to #28546

Extracted from #28546 per its review panel (read-side half of the soft-delete guards; findings CRF-11/CRF-29/CRF-33/CRF-34). The write-side race fix stays in #28546; this PR makes any row that slips past cleanup inert as a credential regardless of its source.

Intended merge order: this PR lands first, then #28546. Until its write-side guards make the protection source-agnostic, this PR already neutralizes any orphaned row however it was created.

Note: the branch temporarily carries #28874's storybook deflake commits so the local full-suite pre-push validation can run; they drop out on rebase once #28874 merges.


Generated with mux • Model: anthropic:claude-fable-5 • Thinking: xhigh

@ThomasK33

Copy link
Copy Markdown
Member Author

/coder-agents-review

@coder-agents-review

coder-agents-review Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Chat: Review posted | View chat
Requested: 2026-09-01 13:48 UTC by @ThomasK33

Review history
  • R1 (2026-08-26), 1 Nit, 2 Note, 2 P2, 2 P3, COMMENT. Review
  • R2 (2026-08-26), 1 Nit, 3 Note, 3 P2, 3 P3, COMMENT. Review
  • R3 (2026-09-01): 19 reviewers, 2 Nit, 6 Note, 1 P1, 4 P2, 13 P3, REQUEST_CHANGES. Review

deep-review v0.9.0 | Round 3 | b5c49e1..6732127

Last posted: Round 3, 26 findings (1 P1, 4 P2, 13 P3, 2 Nit, 6 Note), REQUEST_CHANGES. Review

Finding inventory

Finding inventory - PR #28634

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P2 Author fixed (9a09ace) coderd/httpmw/apikey.go:489 Hard: true on deleted-user 401 contradicts Hard semantics and locks stale-cookie holders out of public workspace apps R1 Netero Yes
CRF-2 P2 Author fixed (6732127); verify coderd/httpmw/apikey.go:278 Deleted-user rejection placed after the mutate block, so a rejected credential still drives writes and a trigger DELETE cascade R1 Netero Yes
CRF-3 P3 Author fixed (6732127); verify coderd/httpmw/apikey.go:929 UserRBACSubject gained an error return that most callers translate to 500 or opaque error; none updated or tested R1 Netero Yes
CRF-4 P3 Author fixed (9a09ace) coderd/httpmw/apikey.go:479 Comment/test/PR claim "manual insert" can orphan a key; trigger_insert_apikeys already prevents it R1 Netero Yes
CRF-5 Nit Author fixed (9a09ace) coderd/httpmw/apikey.go:915 ErrUserDeleted sentinel declared 900 lines in; convention puts sentinels near top R1 Netero Yes
CRF-6 Note Author fixed (9a09ace) coderd/database/querier_test.go:3442 Comment says "role set intact" but cleanup trigger already deleted org memberships R1 Netero Yes
CRF-7 Note Author fixed (6732127); verify coderd/httpmw/apikey_test.go:270 ALTER TABLE ... DISABLE TRIGGER in a parallel test stalls other tests under shared CODER_PG_CONNECTION_URL R1 Netero Yes
CRF-8 P2 Author fixed (6732127); verify coderd/httpmw/apikey_test.go:250 TestAPIKey/DeletedUser passes against the pre-fix implementation; neither the soft-401 (CRF-1) nor the reject-before-write ordering (CRF-2) is pinned R2 Netero Yes
CRF-9 P3 Author fixed (6732127); verify coderd/httpmw/workspaceagent.go:126 The new ErrUserDeleted 401 branch has no test R2 Netero Yes
CRF-10 Note Author fixed (6732127); verify coderd/database/querier_test.go:3442 Test pins the returned column but not the consumer contract (provisionerdserver/dynamicparameters still resolve roles for a soft-deleted owner) R2 Netero Yes
CRF-11 P2 Open coderd/apikey.go:160 Deleted-user token creation guard nested in if Lifetime != 0; default token path and postAPIKey (no guard) 500 with raw pq error; also inconsistent 400/404 status and deleted-state disclosure R3 Knov P2, Chopper P2, Bisky P2, Leorio P2, +11 P3 Yes
CRF-12 P3 Open coderd/httpmw/workspaceagent.go:126 Comment/PR desc/test claim the 401 makes the agent stop retrying; agent retries 401 identically to 500 (no status inspection) R3 Knov, Razor, Kite, Hisoka, Chopper, Mafuuu, Takumi P3 Yes
CRF-13 P3 Open coderd/x/chatd/chatd.go:1232 chatd branches use xerrors.New, dropping the %w sentinel chain and leaving a permanent condition unmarked terminal, so retryGenerationPhase retries 3x R3 Ryosuke, Meruem, Kite, Mafuuu, Ging-go, Zoro, Pariston, Takumi, Chopper, Melody P3 Yes
CRF-14 P3 Open coderd/x/chatd/chatd.go:3857 userSkillContext synthesizes an rbac.Subject for the chat owner directly, bypassing UserRBACSubject/ErrUserDeleted; invariant has an unlisted hole R3 Knov, Ryosuke, Meruem P3; Kite P4; Kurapika, Pariston Note Yes
CRF-15 P3 Open coderd/exp_chats.go:5214 Model-override deleted-member guard only fires for mode=model and non-self; normal soft-delete path 500s in OrganizationMemberParam middleware; 400+disclosure R3 Knov, Mafuuu P3; Razor Note Yes
CRF-16 P3 Open coderd/httpmw/workspaceagent.go:130 Deleted-owner 401 uses hard httpapi.Write not optionalWrite, inconsistent with the CRF-1 soft decision; hard-fails the Optional GET /appearance mount R3 Ryosuke, Melody P3; Kurapika Note Yes
CRF-17 P3 Open coderd/oauth2provider/tokens.go:406 11-13 new ErrUserDeleted routing branches untested; oauth2 pair reachable in normal operation (codes not purged) and RFC-constrained; one branch (CRF-11) already broken R3 Mafu-san P2; Kite, Chopper, Bisky, Meruem P3; +5 Note Yes
CRF-18 P1 Open site/vite.config.mts:250 Branch carries 4 unrelated #28874 storybook commits (not in origin/main): red required title check now, and merging ships a repo-wide retry: 2 flake-mask unreviewed under a fix(coderd) title R3 Mafu-san P1; Ryosuke, Leorio, Komugi P2; +6 lower Yes
CRF-19 P3 Open coderd/httpmw/apikey.go:433 TOCTOU: a soft-delete committing during the OIDC refresh makes UpdateUserLink return sql.ErrNoRows -> Hard 500, resurrecting the CRF-1 lockout in the window; block comment overstates an unconditional invariant R3 Komugi P2; Takumi Note Yes
CRF-20 P3 Open coderd/httpmw/apikey.go:63 Structural: ErrUserDeleted carries no HTTP disposition, so correctness depends on 14 callers hand-mapping it (500 default); httpmw owns a policy that service packages reach sideways for R3 Ryosuke, Meruem P3 Yes
CRF-21 P3 Open coderd/database/querier_test.go:3442 Test comment names "prebuilds" as a dependent that does not call GetAuthorizationUserRoles (drifted from users.sql:588); test does not pin the org-role removal it describes R3 Gon P2; Zoro Nit; Melody, Mafuuu, Meruem Note Yes
CRF-22 P3 Open coderd/exp_chats.go:3556 Three byte-identical 12-line ErrUserDeleted prologues in chatCreate/Start/StopWorkspace; extract a chatOwnerContext helper R3 Bisky, Zoro P3 Yes
CRF-23 Nit Open coderd/x/chatd/chatd.go:1228 Comment hygiene: "chats are not purged" rationale duplicated 6x across 3 packages; "pre-fix" review-history comment in apikey_test.go:341; verbose comments restating code R3 Gon, Ryosuke, Leorio, Mafu-san, Zoro Yes
CRF-24 Note Open coderd/oauth2provider/tokens.go:524 UserRBACSubject signals "may not act" via a forced error (deleted) and a discardable value (suspended); 12 sites drop UserStatus, so a suspended user's refresh token still mints an (inert) key R3 Knov P3; Chopper, Zoro Note Yes
CRF-25 Note Open coderd/httpmw/apikey.go:290 Reject-on-read never removes the orphaned api_keys row (dbpurge only removes expired keys); a purge over api_keys joined to users.deleted would eliminate the bad state R3 Meruem Note Yes
CRF-26 Note Open coderd/httpmw/apikey_test.go:58 SET LOCAL session_replication_role = replica is superuser-only; a non-superuser CODER_PG_CONNECTION_URL yields an opaque permission error; document in the helper R3 Kite, Ryosuke, Meruem, Hisoka Note Yes

Contested and acknowledged

None.

Re-raised (partial fixes)

CRF-2 (P2, apikey.go:422) - reject-before-write still violated by the refresh block

  • Original (R1): rejection sat after the LastUsed/expiry mutate block, so a rejected credential drove writes and a trigger DELETE cascade.
  • R1 fix: author moved the UserRBACSubject call and rejection above the LastUsed/expiry (changed) block.
  • Re-raise (R2): the OIDC/GitHub token-refresh block (apikey.go:280-412) still runs before the deleted check. For an orphaned OIDC/GitHub credential whose OAuth token has expired, the middleware makes an outbound IdP call (~366) and attempts UpdateUserLink (~390), which hits fail_if_user_deleted() and returns a Hard 500, not the soft 401 the PR promises. Author's reply ("user_links rows are gone") is false in the orphan scenario the PR targets, where cleanup did not run and the link row survives alongside the api_keys row. Fix: do the deleted check immediately after the expiry check (~278), before the refresh block.

CRF-3 (P3, apikey.go:929) - routing incomplete

  • Original (R1): new ErrUserDeleted return unrouted at nine of ten UserRBACSubject callers.
  • R1 fix: author routed one additional caller (workspaceagent.go).
  • Re-raise (R2): 12 of 14 non-test call sites remain unrouted. oauth2provider/tokens.go:391,508 wrap into a generic 500 rather than RFC 6749 invalid_grant (AGENTS.md OAuth2 rule; sibling sentinels errBadCode/errBadToken show the intended shape). chatd owner resolution (chatd.go:1215,1632, listtemplates.go:438, exp_chats.go) now fails with an opaque "user is deleted" for a soft-deleted owner; chats are not purged by delete_deleted_user_resources, so the no-workspaces bound does not protect this path. Also makes the PR's "authentication path only" claim inaccurate: the guard lives in the shared UserRBACSubject.

CRF-7 (Note, apikey_test.go:270) - lock-free alternative exists

  • Original (R1): ALTER TABLE ... DISABLE TRIGGER takes an ACCESS EXCLUSIVE lock that stalls parallel tests under shared CODER_PG_CONNECTION_URL.
  • R1 fix: author added a comment documenting the hazard but kept the ALTER TABLE.
  • Re-raise (R2): SET LOCAL session_replication_role = replica suppresses the trigger for the transaction with no DDL and no table lock (verified on the current schema). Replacing the two ALTER TABLE statements removes the hazard the comment describes.

Round log

Round 3 (panel)

First panel round (Netero-only cap reached). 19 reviewers. All 6 round-2 findings verified genuinely fixed by the panel (CRF-2 ordering pinned by mutation, CRF-7 lock-free helper, CRF-8/9/10 tests real). Event REQUEST_CHANGES (CRF-18 P1). New findings CRF-11..CRF-26. Heaviest convergence: CRF-11 (token-creation 500, ~15 reviewers), CRF-12 (false retry claim, 7 P3), CRF-13 (chatd sentinel drop + retried permanent error, 10 P3), CRF-18 (carried storybook commits / red title check). Orchestrator verified CRF-11 (nested guard + postAPIKey no guard, apikey.go:155/208) and CRF-16 (workspaceagent hard Write vs optionalWrite) against the worktree. Disagreement resolved: CRF-19 race rated P3 (Komugi P2 empirical repro vs Takumi Note inherent read-then-act; kept actionable at P3, narrower than the always-on CRF-1).

Round 3 (pre-panel bookkeeping)

Churn guard PROCEED. All 6 round-2 findings author-fixed in 6732127 (rebased onto b5c49e1; PR commits 84f5783, be7531c, 6732127). PR grew to +508 effective (174 prod, 334 test, 22 files) as the author routed all 14 UserRBACSubject call sites, added tests, and moved the reject above the refresh block. Netero-only cap (2 consecutive) reached, so this is the first PANEL round. Fixes are author claims until the panel verifies them.

Round 2

Churn guard PROCEED. All 7 round-1 findings author-fixed in 9a09ace (claims verified when reviewers encounter the code). Head 9ee3dae..9a09ace. Netero re-run: 3 P2, 2 P3, 1 Note new. Pre-panel Netero gate: P0-P2 present, so 2nd consecutive Netero-only round (COMMENT), panel deferred. Orchestrator verified: refresh-block ordering (apikey.go:280-412 precedes the deleted check at ~415; UpdateUserLink at ~390 hits fail_if_user_deleted for surviving user_links); 14 UserRBACSubject call sites with only apikey.go:422 and workspaceagent.go:112 routed; tokens.go:391 wraps into a generic 500. CRF-1, CRF-4, CRF-5, CRF-6 confirmed genuinely fixed. CRF-2, CRF-3, CRF-7 fixed only partially, re-raised R2 on their threads. New: CRF-8 (test does not pin the CRF-1/CRF-2 behaviors), CRF-9 (workspaceagent branch untested), CRF-10 (querier test does not pin the consumer contract).

Round 1

Netero-only first-pass. 2 P2, 2 P3, 1 Nit, 2 Notes. Reviewed against 845790e..9ee3dae. Pre-panel Netero gate: P0-P2 findings present, so panel deferred until mechanical findings are addressed. COMMENT event. Orchestrator verified all findings against the worktree: Hard-flag routing (write vs optionalWrite at apikey.go:663/596), the mutate-then-reject ordering (changed block at 442-473 precedes UserRBACSubject at 476), and the trigger_update_users/delete_deleted_user_resources DELETE cascade (dump.sql:5236, 962). Netero empirically confirmed the row deletion and last_seen_at bump with a probe test (since removed).

About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Komugi flake/determinism
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

🤖 Managed by Coder Agents.

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First-pass review only. These are mechanical findings from Netero; the full review panel has not yet reviewed this PR and will do so once the findings below are addressed. Posting now so the defects are fixed before the panel spends parallel review time.

The change is well-scoped and the tests are honest: reverting apikey.go to base makes TestAPIKey/DeletedUser fail with 200 instead of 401, and TestGetAuthorizationUserRolesDeletedUser is compile-coupled to the new column. The decision to have GetAuthorizationUserRoles return the deleted flag rather than filter on it (keeping the provisioner path alive for soft-deleted owners) is the right call and is well argued in the description.

Severity count: 2 P2, 2 P3, 1 Nit, 2 Notes.

The two P2s are worth pausing on. CRF-1: the new 401 sets Hard: true, which routes through write() instead of optionalWrite(), so a browser holding the session cookie of a since-deleted user gets a JSON 401 on a public workspace app instead of being treated as anonymous. CRF-2: the rejection sits after the block that writes the API key and bumps last_seen_at, and that UPDATE re-fires trigger_update_users, so a request the server is about to reject still drives a DELETE cascade on api_keys and advances the deleted user's last_seen_at. Both undercut the description's claim that an orphaned row is "inert."

Netero on the ordering: "An unauthenticated request on the read path therefore deletes rows as a side effect."

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/httpmw/apikey.go Outdated
Comment thread coderd/httpmw/apikey.go Outdated
Comment thread coderd/httpmw/apikey.go
Comment thread coderd/httpmw/apikey.go Outdated
Comment thread coderd/httpmw/apikey.go Outdated
Comment thread coderd/database/querier_test.go
Comment thread coderd/httpmw/apikey_test.go Outdated
@ThomasK33 ThomasK33 added the experimental Changes that might not necessarily be merged, until its approved to proceed with. label Aug 26, 2026
@ThomasK33

Copy link
Copy Markdown
Member Author

Round 2 pushed as 9a09ace, addressing all seven first-pass findings:

  • CRF-2 (P2): the roles fetch and deleted-user rejection moved above the changed block, so a rejected credential drives no writes: no UpdateAPIKeyByID, no UpdateUserLastSeenAt, no cleanup-trigger re-fire. (The OAuth-refresh block already 401s earlier for deleted users because their user_links rows are gone.)
  • CRF-1 (P2): Hard dropped from the new 401, so optional-auth routes treat a since-deleted user's cookie as anonymous, matching every other rejected-credential 401.
  • CRF-3 (P3): httpmw/workspaceagent.go maps ErrUserDeleted to a 401 so an agent whose owner was soft-deleted stops retrying.
  • CRF-4 (P3) / CRF-6 / CRF-7: "a manual insert" replaced with "an insert that bypassed trigger_insert_apikeys" in both comments and the description; the querier-test comment now says site-level roles survive while org-scoped roles are already gone with the memberships; the shared-database ACCESS EXCLUSIVE stall is documented at the site.
  • CRF-5 (Nit): ErrUserDeleted declared at the top of the file above ValidateAPIKeyError.

Validation: build + vet, full coderd/httpmw package, and the querier contract test against live PostgreSQL.

@ThomasK33

Copy link
Copy Markdown
Member Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second first-pass (Netero) round; the full panel still hasn't reviewed and will once the mechanical floor is clean. Round 1's fixes are real progress: CRF-1 (soft 401 for optional-auth routes), CRF-4 ("insert that bypassed trigger_insert_apikeys"), CRF-5 (sentinel moved to the top), and CRF-6 (org-membership comment) are all genuinely fixed and verified.

But three of the seven were only partially fixed, and the two headline behaviors this commit is named for still aren't nailed down. Severity count this round: 2 P2 open (one re-raise, one new), 1 P3 re-raise, 1 P3 new, 1 Note re-raise, 1 Note new.

The core issue: the "reject before any write" property still does not hold across ValidateAPIKey. CRF-2's fix moved the check above the LastUsed/expiry block, but the OIDC/GitHub token-refresh block runs even earlier, so an orphaned OIDC/GitHub credential (exactly the restored-backup / cleanup-didn't-run scenario the PR targets) still triggers an outbound IdP call and an UpdateUserLink write, and comes back as a Hard 500, not the soft 401. Moving the deleted check to immediately after the expiry check fixes all of this in one place. Separately, CRF-3's routing fix touched one caller; 12 of 14 UserRBACSubject call sites remain unrouted, including an OAuth2 token endpoint that should return RFC invalid_grant and chatd owner resolution for soft-deleted owners (chats aren't purged, so that path is reachable).

And the tests don't pin what was fixed. Netero: "Both CRF-1 (soft 401) and CRF-2 (reject before writes) can be reverted by any future refactor without a single test failing." TestAPIKey/DeletedUser passes unchanged against the round-1 pre-fix implementation because it only asserts status/message/detail, which were identical under the old ordering and old Hard value.

Nothing here is silent; all three re-raises are the same concerns from round 1 with the fix incomplete.

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/httpmw/apikey_test.go
Comment thread coderd/httpmw/workspaceagent.go
Comment thread coderd/database/querier_test.go
…r arming

Six interaction stories flake when the suite runs under CPU load (as it
does inside make pre-push -j24). Four click through a just-closed Radix
Select before Radix asynchronously restores pointer events and unmarks
the page aria-hidden; a shared waitForRadixLayerClose helper now re-
queries the next interaction target until it is interactive again.
Leave With Unsaved Changes raced deeper: the identifier combobox only
commits the typed value to formik on blur, so the story's dialog relied
on the link click's own pointerdown blur beating react-router's render-
armed navigation blocker. The story now blurs explicitly and waits for
the prompt's beforeunload leg to prove the blocker is armed before
navigating, and the validation-error assertion waits for formik's async
validation pass.
The interaction project runs under full CPU contention inside make
pre-push, which keeps surfacing timing races that never fire standalone.
Deterministic failures still fail every retry and block the push.
…istbox close

fillForm clicks a login-type option and immediately types into the
password field or clicks Save, racing Radix's asynchronous pointer-events
and aria-hidden cleanup after the listbox closes. Fast runs (isolated
files or a loaded pre-push) lose the race deterministically; idle
full-suite runs pass only because per-story gaps give the cleanup time.
A full-suite discovery run under CPU load with retries and bail disabled
surfaced the complete tail of the same class: TaskPrompt, MCPServersPage,
AddMCPServerPageView, and WorkspaceSettingsPageView stories all interact
immediately after an option click closes a Radix listbox. Retries cannot
absorb these under sustained pre-push contention, so each is fixed at the
root with waitForRadixLayerClose.
@ThomasK33
ThomasK33 force-pushed the fix-deleted-user-api-auth branch from 9a09ace to 6732127 Compare September 1, 2026 13:46
@ThomasK33

Copy link
Copy Markdown
Member Author

Round 3 pushed as 6732127, addressing all six round-2 findings:

  • CRF-2 (P2, re-raise): the UserRBACSubject fetch and deleted-user rejection moved above the OIDC/GitHub token-refresh block, directly after the API-key expiry check, so reject-before-write now holds for every login type in one place: no IdP call with the deleted user's refresh token, no UpdateUserLink, no LastUsed/last_seen_at writes. (Verified red against the round-2 ordering: the counting fake IdP recorded a call and the middleware attempted the user_links rewrite.)
  • CRF-3 (P3, re-raise): every remaining UserRBACSubject call site routes ErrUserDeleted explicitly: OAuth2 grants → RFC 6749 invalid_grant (errBadCode/errBadToken); chatd/chat-tool paths → explicit fail-closed "chat owner has been deleted" responses (403 responder for the workspace tools); model override → 400; token create/config (getMaxTokenLifetime, the unenumerated 14th site, reachable via GetUserByID) → 400/404; downloadChatFile (fail-closed 404) and loginRequest (unreachable, GetUserByEmailOrUsername filters deleted) documented in place. PR description no longer says "authentication path only".
  • CRF-7 (Note, re-raise): the fixture is now a shared softDeleteUserKeepRows helper using SET LOCAL session_replication_role = replica — no DDL, no ACCESS EXCLUSIVE lock; comment updated.
  • CRF-8 (P2): TestAPIKey/DeletedUser now pins the soft 401 (optional-auth route continues unauthenticated) and reject-before-write (last_used/expires_at/last_seen_at unchanged); new TestAPIKey/DeletedUserExpiredOAuth pins the CRF-2 ordering (zero IdP calls, untouched user_links). All pins verified red via reversible neuters after committing the fix.
  • CRF-9 (P3): new TestWorkspaceAgent/DeletedOwner asserts 401 Workspace owner has been deleted. instead of a 500; verified red.
  • CRF-10 (Note): the GetAuthorizationUserRoles comment names both dependent consumers (provisionerdserver.go role resolution, dynamicparameters/render.go) and warns against adding a deleted filter without migrating them; regenerated via make gen/db.

Also rebased onto current origin/main (clean, no conflicts); the branch temporarily carries #28874's storybook deflake commits so the local full-suite pre-push validation can run — they drop out on rebase once #28874 merges. Validation: go build ./..., vet + gofmt, and -count=1 runs of coderd/httpmw, coderd/oauth2provider, coderd/x/chatd, coderd/x/chatd/chattool, focused coderd (TestChat|TestToken|TestAPIKey|TestUserOIDC), and the querier contract tests against live PostgreSQL — all green.

@ThomasK33

Copy link
Copy Markdown
Member Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Full panel review (rounds 1-2 were Netero-only first-pass gates; this is the panel's first and required pass). The core design is sound and the round-2 fixes hold up under scrutiny: multiple reviewers independently moved the UserRBACSubject fetch back below the refresh/write blocks and confirmed TestAPIKey/DeletedUser and DeletedUserExpiredOAuth fail against that ordering, so CRF-2 and CRF-8 are pinned by assertions that bite (zero IdP calls, untouched user_links, untouched last_used/last_seen_at). CRF-7 is fixed at the root with session_replication_role instead of a table lock, and CRF-10's query comment now names its real dependents (provisionerdserver.go, dynamicparameters/render.go, both verified). Choosing to return the deleted flag rather than filter the :one query, and documenting the two consumers at the query definition, is the right call.

The problem is the routing sweep this round produced: one policy became fourteen hand-written dispositions, and several are wrong or untested. Severity count: 1 P1, 1 P2, 11 P3, 1 Nit, 3 Notes.

Most-converged findings: (CRF-11, P2) creating a token or key for a soft-deleted user returns a 500 with a raw Postgres string on the default no-lifetime path and on POST /keys entirely, because the guard is nested inside if Lifetime != 0; ~15 reviewers found this, several with live probes. (CRF-12, P3, 7 reviewers) the comment, PR description, and test all claim the 401 makes the agent stop retrying, but the agent has no status-code inspection and retries 401 exactly like the 500 it replaced. (CRF-13, P3, 10 reviewers) the three chatd branches use xerrors.New, dropping the ErrUserDeleted chain and leaving a permanent condition unmarked terminal, so retryGenerationPhase retries it three times.

Blocking this round on CRF-18 (P1): the branch carries four #28874 storybook commits that are not in origin/main, which is why the required title check is red, and merging as-is would ship a repo-wide storybook retry: 2 flake-mask plus a shared helper unreviewed under a fix(coderd) title. The PR body's "they drop out on rebase once #28874 merges" is a promise with no enforcing mechanism. Rebase them out (the remaining files are all under coderd/, so the title becomes valid) or land #28874 first; a disclosed rule violation is still a red required check.

Kite on the decomposition: "the artifact under review is not the artifact that ships."

CRF-24 through CRF-26 are attached inline as Notes: worth knowing, not blocking.


coderd/x/chatd/chatd.go:3857

P3 [CRF-14] userSkillContext synthesizes an rbac.Subject for the chat owner directly, bypassing UserRBACSubject, so the "a deleted user may not act" invariant has an unlisted hole in a package this PR touched. (Knov, Ryosuke, Meruem P3; Kite P4; Kurapika, Pariston Note)

It builds rbac.Subject{Type: SubjectTypeUser, ID: userID, Roles: {member}, Scope: ScopeAll} from a bare UUID and hands it to dbauthz.As, never consulting users.deleted. Chats are not purged on soft-delete, which is the exact premise the PR uses to justify guarding the three chatd sites three functions away, so this path is reachable for a deleted owner; explore-subagent turns return before the guarded effectiveMCPServerConfigs check, so they stay reachable. Blast radius is bounded (dbauthz scopes it to ResourceUserSkill.WithOwner(userID), the deleted user's own skills), which is why this is P3 and not higher. But every other exception in this PR (loginRequest, downloadChatFile) got a comment; this one is silent. Route it through UserRBACSubject (or roles.Subject(scope) on the row) so the invariant holds by construction, or document why it is safe.

🤖

coderd/httpmw/apikey.go:433

P3 [CRF-19] A soft-delete that commits during the OIDC/GitHub refresh turns the promised soft 401 into a hard 500, and the block comment states an unconditional invariant a concurrent delete breaks. (Komugi P2; Takumi Note)

The deleted check at :290 and the UpdateUserLink write at :421 are separated by an outbound IdP round trip (:397) with no ordering against a concurrent UpdateUserDeletedByID. Komugi constructed the losing schedule (soft-delete committed from inside TokenSourceFunc): the user_links row is gone, UpdateUserLink returns sql.ErrNoRows, and :433 maps it to Hard: true / 500, resurrecting the CRF-1 optional-route lockout inside a window measured in IdP-round-trip time. The trigger is the ordinary "admin deletes a user who has a tab open."

Rated P3 rather than P2: the window is narrow and OIDC/GitHub-only, materially smaller than the always-on CRF-1, and Takumi notes the point-in-time read is inherent to any read-then-act auth check. But the fix is small and worth it: map UpdateUserLink returning sql.ErrNoRows to the same soft 401 (a credential revoked under us is not a server error). Separately, soften the comment at :281-289, which promises "keeps the stale token from bumping last_seen_at" as an unconditional guarantee that only holds for a delete committed before the read.

🤖

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/apikey.go
if err != nil {
// The {user} param can resolve a soft-deleted user; creating
// a token for one is a bad request, not a server error.
if errors.Is(err, httpmw.ErrUserDeleted) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 [CRF-11] Creating a token or API key for a soft-deleted user returns a 500 with a raw Postgres string, because the deleted-user guard is nested inside the optional lifetime branch and postAPIKey has no guard at all. (Knov P2, Chopper P2, Bisky P2, Leorio P2; +11 at P3)

The errors.Is(err, httpmw.ErrUserDeleted) check sits inside if createToken.Lifetime != 0, and validateAPIKeyLifetime is the only thing on this path that calls UserRBACSubject. With lifetime omitted (the default), and for POST /users/{user}/keys (no guard at all), control falls through to createAPIKey -> InsertAPIKey, where insert_apikey_fail_if_user_deleted (dump.sql:1170) raises. Verified live by several reviewers:

POST /users/{uuid}/keys/tokens (no lifetime) -> 500 "Failed to create API key." detail="insert API key: pq: Cannot create API key for deleted user"
POST /users/{uuid}/keys/tokens (lifetime set) -> 400 "Cannot create a token for a deleted user."
POST /users/{uuid}/keys                       -> 500, same pq detail

Reachable without any orphan race: ExtractUserContext resolves {user} by UUID via GetUserByID, which has no deleted = false filter. The PR description's "token create/config endpoints return 400/404 for a deleted target user" is false for two of three shapes, and the 500 leaks a driver string and burns the server-error budget.

Fix (also resolves CRF-15's status split): httpmw.UserParam(r) already returns the database.User with Deleted. Check if user.Deleted at the top of both postToken and postAPIKey, next to the existing user.IsSystem guard, and return one consistent status. Prefer httpapi.ResourceNotFound to match tokenConfig and to avoid disclosing the user's deleted state (see CRF-15). Then delete the nested ErrUserDeleted branch.

🤖

}),
)
if err != nil {
// A soft-deleted owner is an expected, terminal condition

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 [CRF-12] The comment, PR description, and test all justify the 401 by claiming the agent stops retrying; the agent retries a 401 exactly as hard as the 500 it replaced. (Knov, Razor, Kite, Hisoka, Chopper, Mafuuu, Takumi P3; Kurapika Nit)

The comment: "401 tells it to stop retrying instead of hammering a 500." But agent.runLoop (agent/agent.go:604) wraps run() in retry.New(100ms, 10s) and exits only on context cancellation or a.isClosed(); nothing in agent/ or codersdk/agentsdk/ inspects an HTTP status on the connect path. Takumi adds that the only permanent-status list (workspacesdk/dialer.go) lists both 401 and 500 as permanent, and that this middleware runs once at the websocket upgrade so an already-connected agent keeps its session regardless.

The 401 status is correct on its own terms (a deleted owner is a client condition, not a server error, and it keeps deleted-owner agents out of the 5xx rate). The stated mechanism does not exist, and workspaceagent_test.go:99 encodes the false claim ("terminal 401, not a retryable 500") while asserting only the status code. Drop the retry claim from the comment, the test comment, and the PR description, or make it true in runLoop in a separate change.

🤖

Comment thread coderd/x/chatd/chatd.go
// path stays reachable. Fail closed with an explicit message
// instead of an opaque authorization error.
if errors.Is(err, httpmw.ErrUserDeleted) {
return nil, xerrors.New("chat owner has been deleted")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 [CRF-13] The chatd fail-closed branches use xerrors.New, which drops the %w sentinel chain and leaves a permanent condition unmarked terminal, so retryGenerationPhase retries it three times. (Ryosuke, Meruem, Kite, Mafuuu, Ging-go, Zoro, Pariston, Takumi, Chopper, Melody P3)

forcedMCPServerConfigsForOwner returns xerrors.New("chat owner has been deleted"), replacing xerrors.Errorf("load chat owner authorization: %w", err). Two costs. First, errors.Is(err, httpmw.ErrUserDeleted) is now false for every caller above, so the sentinel this PR introduced to make classification possible is destroyed at these sites; the pre-change wrapped error carried strictly more information. Second, the error reaches prepareGeneration under retryGenerationPhase (generation.go:462), which only short-circuits on isTerminalGeneration (errTerminalGeneration). A soft-deleted owner is permanently unauthorizable, but the turn burns generationPhaseMaxAttempts = 3 with 200ms/400ms backoff and logs misleading "retrying" warnings.

Siblings at chatd.go:1655 and chattool/listtemplates.go:445. Fix: keep the chain and mark terminal, e.g. terminalGeneration(xerrors.Errorf("chat owner has been deleted: %w", err)), or delete the branches since the wrapped fallthrough already fails closed. Note (Zoro): at listtemplates.go:445 the reworded string does reach the model via asOwner, so keep the wording there but keep the %w.

🤖

Comment thread coderd/exp_chats.go
if apiKey.UserID != member.UserID {
memberSubject, _, err := httpmw.UserRBACSubject(ctx, api.Database, member.UserID, rbac.ScopeAll)
if err != nil {
// A deleted member is a bad request target, not a server

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 [CRF-15] The model-override deleted-member guard only fires for mode=model and non-self callers, and the normal soft-delete path 500s in middleware before it; the response also uses 400 and discloses the deleted state, unlike the PR's sibling sites. (Knov, Mafuuu P3; Razor Note)

putUserChatPersonalModelOverride resolves member := httpmw.OrganizationMemberParam(r), then reaches UserRBACSubject only inside case ChatPersonalModelOverrideModeModel under if apiKey.UserID != member.UserID. Requests with mode=chat_default/deployment_default skip it. And in the normal path, delete_deleted_user_resources removes the user's organization_members rows, so ExtractOrganizationMember returns 0 rows and organizationparam.go 500s ("Expected exactly one organization member, but got 0", labeled "should never happen") before this branch runs. So the site is only reachable in the orphan state, and the guard is placed where a UserRBACSubject call happened to exist rather than where member is resolved. Check member.UserID's deleted flag once after it is resolved, and pick a status consistent with CRF-11 (404). The 400 message here also discloses the target's deleted state; userparam.go deliberately uses a constant message so no state about the queried user leaks (see CRF-11).

🤖

// for the agent, not a server error: 401 tells it to stop
// retrying instead of hammering a 500.
if errors.Is(err, ErrUserDeleted) {
httpapi.Write(ctx, rw, http.StatusUnauthorized, codersdk.Response{

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 [CRF-16] The deleted-owner 401 uses hard httpapi.Write, not optionalWrite, contradicting the CRF-1 decision one file over; on the one Optional: true mount it hard-fails instead of degrading to anonymous. (Ryosuke, Melody P3; Kurapika Note)

The middleware's own rule (workspaceagent.go:68-69) reserves optionalWrite for "token is not provided or is invalid"; the adjacent sql.ErrNoRows branch obeys it. An api-key/agent-token row whose owner is deleted is exactly that case, and apikey.go:293 deliberately omits Hard for it so public workspace apps fall through anonymous (CRF-1). Here the same condition writes hard. enterprise/coderd/coderd.go:681 mounts this middleware with Optional: true on GET /appearance.

Honest caveat: this branch is consistent with the inactive-user sibling two lines below (which also uses hard Write), so the inconsistency is with the apikey path, not within this file (Kurapika). This needs a deliberate call: either soft (match CRF-1, since a deleted owner is the same "credential belongs to a gone user" case) or hard (match the inactive sibling) and say which and why. The DeletedOwner test uses Optional: false, so the differing branch is uncovered.

🤖

Comment thread coderd/exp_chats.go
) (codersdk.Workspace, error) {
actor, _, err := httpmw.UserRBACSubject(ctx, api.Database, ownerID, rbac.ScopeAll)
if err != nil {
// Chats are not purged when their owner is soft-deleted, so a

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 [CRF-22] Three byte-identical twelve-line ErrUserDeleted prologues in chatCreateWorkspace, chatStartWorkspace, and chatStopWorkspace; extract one helper. (Bisky, Zoro P3)

The three functions differ only in return type: identical comment, UserRBACSubject call, errors.Is branch, 403 body, and ctx = dbauthz.As(ctx, actor). Triplicated fail-closed branches drift independently, and the next status or message change will touch three places and miss one. Extract a chatOwnerContext(ctx, ownerID) (context.Context, error) (the shape chattool/listtemplates.go asOwner already demonstrates), then each call site is four lines, and there is one place to test. TestChatStopWorkspace_BypassesRequireActiveVersion already drives one of them and can be copied for the helper test. The fourth model-override site (5214) is a different unit; keep it separate (see CRF-15).

🤖

Comment thread coderd/x/chatd/chatd.go
func forcedMCPServerConfigsForOwner(ctx context.Context, store database.Store, organizationID, ownerID uuid.UUID) ([]database.MCPServerConfig, error) {
owner, _, err := httpmw.UserRBACSubject(ctx, store, ownerID, rbac.ScopeAll)
if err != nil {
// Chats are not purged when their owner is soft-deleted, so this

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit [CRF-23] Comment hygiene across the routing sweep. (Gon, Ryosuke, Leorio, Mafu-san, Zoro)

Three things, all against AGENTS.md's "put each fact in one place" and "no review-history comments": (1) the "Chats are not purged when their owner is soft-deleted" rationale is pasted verbatim at six sites (exp_chats.go:3556/3641/3725, chatd.go:1228/1651, listtemplates.go:441); state it once on the ErrUserDeleted declaration and leave one-line pointers. (2) apikey_test.go:341 narrates this PR's review history ("pre-fix, the middleware would call the IdP..."); there is no "pre-fix" in the merged tree, so say what the fixture constructs instead. (3) several new comments restate the status code on the next line ("not a server error" above a non-5xx return); keep only the part a reader cannot see, e.g. that GetUserByID resolves soft-deleted users. Also (Leorio): the commit subjects exceed 72 chars and one joins two changes with "and"; move the mechanism to the body.

🤖

@@ -518,6 +523,11 @@ func refreshTokenGrant(ctx context.Context, db database.Store, app database.OAut

actor, _, err := httpmw.UserRBACSubject(ctx, db, prevKey.UserID, rbac.ScopeAll)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note [CRF-24] UserRBACSubject signals "this subject may not act" through two channels, and callers are forced to handle only one. (Knov P3; Chopper, Zoro Note)

The deleted case is a forced error return; the suspended case is the UserStatus value, which 12 of 14 call sites discard (actor, _, err :=). So refreshTokenGrant correctly maps ErrUserDeleted to invalid_grant but a suspended user's refresh token still proceeds to apikey.Generate and gets a fresh session token. Not exploitable, because ExtractAPIKeyMW re-checks status downstream and the minted key is inert. It is a contract asymmetry, pre-existing but now concentrated in the shared helper this PR makes the decision point: if a deleted user may not act, neither may a suspended one, and the API lets a caller silently forget the second. Recording, not blocking.

🤖

Comment thread coderd/httpmw/apikey.go
// the stale token from calling the IdP with the deleted user's refresh
// token, rewriting user_links, bumping the deleted user's last_seen_at,
// or re-firing the cleanup trigger.
actor, userStatus, err := UserRBACSubject(ctx, cfg.DB, key.UserID, key.ScopeSet())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note [CRF-25] Reject-on-read never removes the orphaned api_keys row, so the bad state is tolerated on every request instead of eliminated. (Meruem Note)

Nothing sweeps api_keys rows whose user is deleted; dbpurge deletes only expired keys, so an orphaned row with a live expires_at is presented and rejected on every request until it expires, each rejection costing a GetAuthorizationUserRoles round trip. DeleteAPIKeysByUserID already exists. A purge pass over api_keys joined to users.deleted would make the bad state stop existing rather than being permanently tolerated, and it does not depend on #28546. Worth considering as the other half of the same fix; not blocking this PR.

🤖

// shared database). This reconstructs the orphaned credentials the
// middleware must reject (rows that survived cleanup past a race, a
// restored backup, an insert that bypassed trigger_insert_apikeys).
func softDeleteUserKeepRows(t *testing.T, sqlDB *sql.DB, userID uuid.UUID) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note [CRF-26] softDeleteUserKeepRows silently requires a superuser test role. (Kite, Ryosuke, Meruem, Hisoka Note)

SET LOCAL session_replication_role = replica is a superuser-only GUC (the previous ALTER TABLE ... DISABLE TRIGGER needed only table ownership). It works in CI and against the dockerized postgres here, but a developer pointing CODER_PG_CONNECTION_URL at a least-privilege role gets a raw permission denied to set parameter from require.NoError with no hint about the cause. One clause in the helper comment naming the requirement resolves it. Not a regression, just an unnamed environmental assumption.

🤖

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

experimental Changes that might not necessarily be merged, until its approved to proceed with.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant