First-pass review only. These are mechanical findings from the pre-panel reviewer (Netero); the full review panel has not yet reviewed this PR and will do so after these findings are addressed.

The change is well-constructed: the FOR NO KEY UPDATE choice is correctly reasoned against FOR KEY SHARE from FK validation, the INSERT-only gate is deliberate and documented, the isolation-level analysis is sound, and the tests are genuine (the concurrent-insert test blocks on the real row lock via pg_stat_activity rather than sleeping, and the migration backfill test asserts a non-zero pre-state with a control user so the post-state is not vacuous). Migration numbering and dump.sql are consistent.

Severity count: 1 P2, 1 P3, 1 Note.

The P2 is the one to resolve before the panel spends parallel review time: the migration comment claims the unlocked UPDATE path avoids the delete_deleted_user_resources deadlock, but for user_secrets that deadlock is still reachable through the separate per-user cap trigger, which takes FOR UPDATE on the users row on BOTH INSERT and UPDATE with no TG_OP gate. Netero reproduced the deadlock. Either gate the cap trigger's lock to INSERT-only, or correct the migration comment to state the hazard remains for user_secrets. As Netero put it: the comment "is true for user_links and user_skills and false for user_secrets."

The P3 flags that delete_deleted_user_resources wipes six tables but only four have insert guards; user_ai_provider_keys and organization_members have neither a guard nor a backfill entry here. That is a pre-existing gap this PR does not widen, but it needs a human decision: file a ticket or explicitly accept it.

🤖 This review was automatically generated with Coder Agents.

First-pass review only (round 2). These are mechanical findings from the pre-panel reviewer (Netero); the full review panel still has not reviewed this PR and will do so once the P1 below is resolved. The panel runs next round regardless.

What improved since round 1: CRF-1 is genuinely fixed (the user_secrets cap trigger now gates its users-row lock to TG_OP = 'INSERT', and the UPDATE path completes immediately), and CRF-2 is fixed (new BEFORE INSERT guards plus backfill for user_ai_provider_keys and organization_members, and the race and migration tests now cover all six tables). CRF-3 is accepted as an intentional constraint contract. Netero also positively verified the backfill's "only race products survive" claim by confirming every table added to delete_deleted_user_resources shipped its own backfill in an earlier migration.

But the round-1 fix opened a new regression. Severity count: 1 P1, 1 P2.

The P1 is a lock-order inversion introduced by making the INSERT-path guard take users FOR NO KEY UPDATE. The two OAuth2 token transactions (authorization-code exchange and refresh grant) DELETE the previous api_keys row and then InsertAPIKey in the same transaction, so they now acquire the old api_keys tuple before users, while delete_deleted_user_resources acquires users before deleting api_keys. Netero reproduced a live deadlock at HEAD (Postgres aborted the soft-delete with SQLSTATE 40P01) that does not exist at base, where the soft-delete merely waits. The end state is the same one this PR exists to prevent: a user left active with a freshly minted token, now surfaced as a 500 on DELETE /api/v2/users/{user}. The migration comment's claim that gating the lock to the INSERT path avoids the deadlock is false as stated: the inversion is per-transaction statement order, not per-trigger-operation. In Netero's words: "the same end state the PR exists to prevent, reached through a different door." Fix direction: have the token-writer transactions take the parent users lock first (before DeleteAPIKey*), matching the cleanup's order, and correct the migration comment.

The P2 is that the regression test offered as proof of the CRF-1 fix, TestSoftDeleteGuardUpdatePathTakesNoUserLock, never bounds the wait: when the shared context deadline fires, database/sql rolls back the lock-holding transaction and the previously blocked UPDATE then succeeds, so the test passes whether or not the TG_OP gates exist (Netero measured 25s at round-1 base vs 0.14s at HEAD, with no assertion looking at the delay). The CRF-1 fix is real; the guard shipped to protect it is not.

Process note, out of scope: make lint/go is red on this repo state with three staticcheck SA5011 issues in scripts/clidocgen/main.go, a file this PR does not touch. Netero could not map it to the named failing CI jobs (gh returned 401 in the review sandbox), so treat the CI mapping as unverified, but it should not be left red.

🤖 This review was automatically generated with Coder Agents.

First full panel round (17 reviewers). This is a genuinely well-built change and several parts earned praise on their own: the FOR NO KEY UPDATE lock strength is correctly chosen against the FK FOR KEY SHARE, the isolation-level analysis is sound, the backfill runs before CREATE TRIGGER to minimize the ACCESS EXCLUSIVE window, the migration header is (in Leorio's words) "the best migration chart note I have read in this repo," and the race tests are real: multiple reviewers mutation-tested them (remove a guard's lock and TestSoftDeleteGuardWinsConcurrentInsert fails by timing out in waitForBackendBlocked). Prior-round fixes hold up: CRF-5's lock_timeout witness is verified real (removing a TG_OP gate now fails the test with 55P03 instead of being rescued by the context rollback), and CRF-2's six-table guard+backfill coverage is confirmed.

Severity count: 2 P1, 4 P2, 12 P3, 3 Nits.

The headline: the CRF-4 fix is incomplete. CRF-4 (the OAuth2 lock-order inversion) was closed as author-fixed, but the fix reordered only the two tokens.go call sites, and the underlying invariant, take the users lock before touching any guarded child row, is enforced by a migration comment and nothing else. Two more transactions violate it and were reproduced as live deadlocks against a concurrent user soft-delete, PR-introduced (dropping the new trigger or checking out base removes the deadlock in every reviewer's control run): regenerateSessionToken on the workspace-start path (CRF-6) and the OIDC/SSO login org-sync transaction (CRF-7). The consequence in both is the exact state this PR exists to prevent: DELETE /api/v2/users/{user} returns 500 (Postgres picks the victim, so a build or login can 500 instead), and 40P01 is not auto-retried (only 40001 is), so it surfaces as a hard error. This is a fix applied per-instance instead of to the class. As Ryosuke put it: "A contract enforced by prose is a contract that decays."

Three structural directions came up repeatedly and are worth a human decision before another Acquire call is bolted on (CRF-8): (1) put the invariant on the read side, GetAuthorizationUserRoles and ValidateAPIKey do not filter users.deleted today, so a resurrected api_keys row authenticates with full roles (CRF-11, verified end-to-end: a suppressed-cleanup orphan key returned 200 on GET /users/me). One AND NOT users.deleted predicate makes every orphan inert regardless of source and imposes no lock ordering on anyone. It does not deliver the per-table constraint contract #28423 wants, so it is a complement, not a drop-in. (2) Make the guard itself order-safe: Knov's suggestion of BEFORE ROW DELETE triggers that take the users lock (BEFORE ROW fires before heap_delete's tuple lock, so every deleter takes users first unconditionally) makes the inversion unrepresentable and removes the caller contract entirely, at the cost of serializing child deletes on the users row. (3) At minimum, add a deadlock-regression test per delete-or-update-then-insert path using the harness already in this PR (reviewers' repros ran ~1s each, deterministic), since removing either tokens.go lock call currently breaks no test.

The CRF-1 fix also has fallout on the user_secrets cap trigger: gating its lock to INSERT removed UPDATE-path serialization, so concurrent per-row PATCHes now blow past the byte caps (CRF-9, reproduced at ~300-400 KiB against a 204 KiB cap and 72 KiB against a 24 KiB env cap; the "bounded overshoot" the comment promises scales with row count, and the over-cap state is sticky, the user can no longer update any secret). Separately the INSERT-path FOR UPDATE the cap trigger keeps is the wrong lock mode: it upgrades the guard's FOR NO KEY UPDATE, conflicts with FK FOR KEY SHARE across the 36 tables referencing users, and adds a deadlock edge (CRF-10). A per-user advisory lock (pg_advisory_xact_lock keyed on user_id, the pattern already used at chatd/synthetickey.go) on both paths closes the bypass and drops the users-row lock, resolving both.

Process notes, not inline: the PR description is stale, it still says migration 000585, never mentions AcquireUserSoftDeleteGuardLock, the tokens.go change, or the ordering contract, which is the single most important thing a future writer needs (CRF-12). The required CI check title is red; the current title parses as valid, so the run is most likely stale from the earlier scoped title (a scope like coderd/database would now fail on the oauth2provider/ and enterprise/ files), but nobody in the panel could read the log (gh returned 401 in the sandbox), so please confirm rather than assume. New per-user serialization on the hot users row is acceptable but on the record: every api_keys insert now waits on any open write to that user's row. And Pariston's note that the migration comment's stated UPDATE-path deadlock mechanism is imprecise (single-row BEFORE ROW triggers fire before the tuple lock; the real hazard is multi-row UPDATE and ON CONFLICT) is worth a wording pass, though the gates themselves are justified.

coderd/provisionerdserver/provisionerdserver.go:3320

P1 [CRF-6] The CRF-4 fix covered tokens.go but missed regenerateSessionToken, which is the same delete-then-insert on api_keys and still deadlocks against a concurrent user soft-delete. (Knuckle, Meruem, Takumi, Killua, Hisoka, Razor, Knov, Mafu-san P1; Ryosuke, Melody, Mafuuu, Pariston P2; Kurapika P3)

The transaction at 3319 runs deleteSessionToken (nested InTx reuses the outer transaction, db.go:186), which holds the api_keys tuple, then InsertAPIKey at 3325, whose guard now takes users FOR NO KEY UPDATE. Lock order: api_keys tuple, then users. delete_deleted_user_resources takes users then api_keys. Cycle. (Knuckle)

Reproduced by five reviewers with controls (deadlock at HEAD, none at base). Postgres picks the victim, so either DELETE /api/v2/users/{user} 500s and the user stays active with a fresh token, or the build fails at regenerate session token. Runs on WorkspaceTransitionStart.

Reachability debate, resolved toward P1: deleteUser rejects users with non-deleted workspaces (users.go:697), so for a real user the race needs the TOCTOU between that unlocked GetWorkspaces check and UpdateUserDeletedByID (verified: no lock between them). That narrows probability but the path is externally reachable and reintroduces a closed P1 with no 40P01 retry, so consequence sets the floor. Fix: tx.AcquireUserSoftDeleteGuardLock(ctx, workspace.OwnerID) as the first statement inside the InTx at 3319.

🤖

coderd/userauth.go:1986

P1 [CRF-7] The new organization_members guard makes the OIDC/SSO login transaction deadlock against a concurrent soft-delete of the same user, an inversion that did not exist before this PR. (Komugi, Meruem, Melody, Mafuuu, Pariston, Hisoka P1; Takumi, Knuckle, Kurapika, Ryosuke, Mafu-san P2)

UpdateUserLink at 1943 locks the user_links tuple (the UPDATE path takes no users lock by design), then SyncOrganizations reaches InsertOrganizationMember (idpsync/organization.go:143), whose new guard takes users FOR NO KEY UPDATE. Cleanup holds users and waits on the user_links tuple. (Meruem)

Reproduced by six reviewers, each with the control that dropping only trigger_insert_organization_members removes the deadlock, so the PR introduces it (before, the insert took only FK FOR KEY SHARE, which does not conflict). Consequence: DELETE /users/{id} or the login 500s; no 40P01 retry. The trigger is an ordinary OIDC login that adds a membership, concurrent with an admin offboarding that user, which is the common deletion scenario (deletion requires no workspaces, not being logged out). The contract text only names DELETE; an UPDATE inverts identically. Fix: AcquireUserSoftDeleteGuardLock(ctx, user.ID) at the top of the oauthLogin InTx (line 1755), and restate the contract as "holds any lock on a guarded child row, then inserts."

🤖

coderd/httpmw/apikey.go:476

P2 [CRF-11] Authentication and authorization never check users.deleted, so the six write-side guards are the only line of defense against a resurrected row, and the read-side check is the cheaper, source-agnostic half that is missing. (Ryosuke P2; Pariston, who rated the underlying bug P0-worthy)

Verified end-to-end: with a soft-deleted user's api_keys row preserved (cleanup trigger suppressed), the same token returned 200 on GET /api/v2/users/me and GET /api/v2/workspaces. GetAuthorizationUserRoles selects roles with no deleted filter (confirmed) and ValidateAPIKey branches only on UserStatus (suspended/dormant), which stays active through a soft-delete.

This is the security consequence the PR's summary leads with, and it is real. AND NOT users.deleted on the authorization lookup makes every resurrected row inert whatever its provenance (this race, a future unguarded table, a restored backup, a manual insert), and imposes no lock ordering on anyone. It does not deliver the per-table constraint contract #28423 wants, so it is a complement to the guards, not a replacement. This needs a human decision: adopt the read-side check as defense in depth, or explicitly accept that the write-side triggers plus the ordering contract are the whole invariant.

🤖

coderd/members.go:74

P3 [CRF-20] The two new guards surface as raw HTTP 500 with Postgres text for caller errors, instead of a mapped 400. (Leorio)

postOrganizationMember handles 404 and the unique-violation, then falls through to InternalServerError. GetUserByID has no deleted filter, so ExtractUserParam resolves a soft-deleted user and the insert hits organization_member_user_deleted; the admin gets a 500 whose only clue is "Cannot create organization_member for deleted user" in Detail. Map it like the unique-violation branch to a 400 ("Cannot add a deleted user to an organization"). Sibling: exp_chats.go:6656 answers user_ai_provider_key_user_deleted with a 500 and logs server-side, so the caller learns nothing. Both new guards need a mapped branch or the PR ships two new 500s.

🤖

🤖 This review was automatically generated with Coder Agents.

Round 4, first full panel on the rework (16 reviewers, plus Netero and Law). Note on process: Law returned a mandatory split (extract the read-side filter, CRF-33), which normally skips the panel, but the tooling requires panel reviewers on a post-panel round, so the panel ran and Law/Netero are folded in here.

The rework is genuinely strong and the panel verified it rather than trusting it: every round-3 fix holds at the SQL and state level (advisory-lock cap rewrite closes the CRF-9 bypass and CRF-10 lock-mode edge, the shared fail_if_user_deleted() passes the right TG_ARGV per trigger, the read-side parity holds, the backfill covers eight tables, and multiple reviewers mutation-tested the race suite and it fails for the right reasons). Several reviewers independently ran the full missed-caller sweep and confirmed the four locked transactions are the complete set.

Severity count: 1 P0, 1 P1, 3 P2, 12 P3, plus nits.

The P0 (CRF-27): the CRF-15 fix authorized the guard lock as ActionUpdate on the target user, but both OAuth2 grants take it under the end user's own actor, and the member role denies self-update, so token exchange and refresh fail for every non-admin. Seven reviewers converged. One correction to Netero's framing, established empirically by Bisky: this is NOT silent, an existing test (coderd/oauth2_test.go TestOAuth2ProviderTokenExchange/OK) runs as a member and fails, so CI catches it (the failing coderd job is likely among the pending checks). The deeper defect (CRF-35, three reviewers) is that the ActionUpdate gate is role-dependent and protects nothing the guard trigger doesn't already hand out for free; the fix is to model the lock as a system primitive (ResourceSystem), not to escalate two call sites and leave the trap armed, and specifically NOT to grant members ResourceUser:update (that action also gates suspend/activate and DeleteUserAIBudgetOverride). In Hisoka's words: "I came for the guards. The guards held. The thing that broke was the lock you swapped in to avoid a deadlock."

Law's mandatory split (CRF-33: extract the read-side GetAuthorizationUserRoles filter) is validated by a concrete bug the panel found (CRF-34, four reviewers): that query is :one, so the new AND users.deleted = false returns sql.ErrNoRows for a soft-deleted owner, and its two non-authentication callers (provisionerdserver.go:855, dynamicparameters/render.go:319) treat any error as fatal. Every build for a deleted owner then fails, including the delete build, leaving the workspace and its cloud resources unrecoverable without DB surgery. This is exactly why the read-side change deserves its own PR and reviewers: it entered at round 3 as CRF-11 remediation and its blast radius was never reviewed as a primary change.

Strategic point, converged independently by Law, Pariston, Ryosuke, and Meruem (echoing Knov in round 3): the write-side lock-ordering contract exported from the database to four handler transactions has now produced five distinct failures on one mechanism across four rounds (CRF-6, CRF-7, CRF-14, CRF-16, CRF-27), and each was caught by a reviewer, never by a check. Meanwhile the authentication harm the mechanism is priced against is fully neutralized by the read-side filter alone (a resurrected key yields no principal). The recommendation is not to abandon the guards, but to land the read-side half first (Law's split) and then reconsider the write-side lock with the security pressure removed, via either a DB-level BEFORE DELETE/UPDATE trigger that takes the users lock before the child row (making the inversion unrepresentable) or an idempotent reaper that moves child cleanup out of the soft-delete transaction. Zoro's useful caveat: simply swapping the guard's lock to an advisory lock does not help, the ordering contract is inherent to any lock the guard takes after the child rows.

The P1 (CRF-28) is why the contract keeps breaking silently in the reverse direction: TestSoftDeleteGuardLockOrderPaths replays hand-written SQL that mirrors each Go transaction, so removing the lock call, weakening the query's FOR NO KEY UPDATE to FOR KEY SHARE, or reverting it from :one all keep the suite green (Bisky, Takumi, and Razor each demonstrated one). Drive at least one path through its real Go entry point.

Process/nits folded here rather than inline: the two advisory-lock keys are not registered in coderd/database/lock.go (CRF-47); the migrate_test.go loop variable guardedTables should be backfilledTables since it includes the two unguarded tables (CRF-49); require.Less on two literals reads as constant-true but is load-bearing via the adjacent pg_trigger count check, so it is a tidy-up not a defect (CRF-32); and a scattering of pure style nits (message punctuation, the stmt type name colliding with database/sql, id.String()[:13] truncation, the 200 KiB literal, raw INSERT seeds where dbgen helpers exist). The AcquireUserSoftDeleteGuardLock doc comment and the in-body guard rationale that reaches dump.sql are, per Leorio, the best writing in the diff; fix the two overclaims in CRF-44 and they do their job.

coderd/provisionerdserver/provisionerdserver.go:855

P2 [CRF-34] The read-side users.deleted filter turns every provisioner job for a soft-deleted owner into a permanent failure, including the delete build, so the workspace and its cloud resources can never be reclaimed. (Knuckle, Hisoka P2; Ryosuke, Komugi P3)

GetAuthorizationUserRoles is :one; with AND users.deleted = false it returns sql.ErrNoRows for a deleted owner, and both provisioning consumers treat any error as fatal (failJob("get owner authorization roles: %s") here, xerrors.Errorf("user roles: %w") at dynamicparameters/render.go:319). The owner is fetched with GetUserByID (no deleted filter) so the job reaches this line and blows up. Reachability is narrow (needs the deleteUser TOCTOU: GetWorkspaces then UpdateUserDeletedByID unserialized, with a workspace landing in the window) but the failure is permanent and --orphan does not help because it still needs a provisioner job; recovery is manual DB surgery. Fix: treat sql.ErrNoRows as an empty role set in these two callers (like GetGitSSHKey handling 150 lines up), or put the predicate in UserRBACSubject/ValidateAPIKey on the auth path only (which also resolves CRF-29). This is the concrete blast radius behind Law's CRF-33: if C6 is extracted, it ships with this bug unless the consumers are fixed too.

🤖

coderd/httpmw/apikey.go:901

P3 [CRF-29] An orphaned API key of a soft-deleted user now produces HTTP 500, not 401. (Netero)

GetAuthorizationUserRoles is :one, so the new filter returns sql.ErrNoRows for a deleted user; UserRBACSubject wraps it and apiKeyFromRequestValidate maps any error to StatusInternalServerError with Hard: true. The credential is correctly rejected, so this is presentation only, but the deployment sees 500s for a denied credential. Distinguish sql.ErrNoRows and return 401. Fixing this in UserRBACSubject/ValidateAPIKey (rather than in the shared query) also resolves CRF-34 by leaving the two provisioning callers untouched.

🤖

🤖 This review was automatically generated with Coder Agents.

Round 5, focused panel (11) plus Netero and Law. Process note: Law returned another mandatory split (extract the cap changes), which normally skips the panel, but the tooling requires panel reviewers on a post-panel round, so the panel ran and Law/Netero are folded in.

The good news first, and it is substantial: there are no P0 or P1 findings this round. Every round-4 fix was verified holding, most by mutation against live PostgreSQL: the system-primitive authz fix (CRF-27/35), the read-side split honored to #28634 with no dangling consumer (CRF-33/34), the lock-order Go test (CRF-28), the transitive backfill coverage (CRF-39), the shared SoftDeleteUserKeepingRows helper (CRF-31/48), and the isolation gate breaks no current writer (four reviewers checked every RR/Serializable transaction in the tree). As Knuckle put it, "the schema side of this change is now clean. Five rounds of rework to get there, and the temptation at round four was to bolt a fallback onto each table and move on. It was not taken."

What remains is concentrated on the mechanisms added this round, which is exactly what Law's split predicts. Severity count: 5 P2, 9 P3, 5 nits.

Law's verdict (CRF-51) is a mandatory split: extract the cap changes (the advisory-lock rewrite, require_read_committed, and the skills-cap UPDATE leg) as their own PR, landing first. They are independently justified (the pre-existing FOR UPDATE hazard), must precede the guard topologically, and require_read_committed and the skills UPDATE leg are both new this round and change when ordinary writes are rejected. The panel's own findings on those two pieces are the argument for the split.

require_read_committed is over-engineered in two opposite directions at once. Pariston (CRF-52): it rejects every secret and skill write outside READ COMMITTED unconditionally, whether or not any concurrency exists (the author's own Rejected test inserts a single row into an empty table and it fails), so a deployment that sets default_transaction_isolation on the server, database, role, or pooler, or a future caller reaching for database.ReadModifyUpdate, gets a total feature outage, all to prevent a soft cap slipping by one. Komugi (CRF-53): meanwhile the same-owner exemption reopens the exact overshoot the gate was added to close (reproduced, 300k against a 204800 cap, reachable via dbcrypt rotation racing a user's secret edit). The two together say the mechanism is both too broad and too narrow; the advisory lock already makes the caps correct under READ COMMITTED, which is what every caller uses. Removing the gate and enforcing "these writes run at READ COMMITTED" with a CI test is the simpler shape.

Knuckle (CRF-54): the migration DROP TRIGGERs take ACCESS EXCLUSIVE on api_keys and three other tables and hold them, together with eight unbounded backfill DELETEs, until the whole single-transaction migration commits, blocking authenticated reads fleet-wide during the upgrade. CREATE OR REPLACE FUNCTION takes no table lock and keeps the one-shared-function win; ALTER TRIGGER ... RENAME avoids the drop for the zz_ renames; and the backfill belongs in an idempotent reaper.

Hisoka (CRF-55): the new Go lock-order test pins that the lock is taken, not that it is taken first. Moving AcquireUserSoftDeleteGuardLock to after the api_keys DELETE, the precise CRF-4 inversion, keeps the entire suite green; the deadlock class that produced CRF-4, CRF-6, and CRF-7 is still reintroducible at all four call sites. Hisoka wrote and verified the six-line fix (assert the blocked backend holds no RowExclusiveLock on the guarded tables before release). "Two tests, same transaction, one of them notices. Shall I show you which one you shipped?"

The P3s are mostly correctness-of-the-new-surface and comment/test accuracy: the guard fires on INSERT but not on UPDATE ... SET user_id, so a re-parent onto a deleted user (including api_keys) is unguarded (CRF-56, reproduced, no caller today); "safe under any isolation level" is false for the guard's own INSERT path under REPEATABLE READ (CRF-57); the TG_ARGV[2] fail-closed branch is dead and its comment describes a capability no table has, flagged by all ten panel-plus-Netero reviewers (CRF-58); the new seventh guard has no 409 mapping so it 500s with raw pq (CRF-59) and the 409 class also never reached api_keys, the headline table, whose token/key endpoints still 500 (CRF-60); group_members is left unguarded on a justification that is false, since GetAuthorizationUserRoles reads it unfiltered into the RBAC subject and its inertness actually depends on #28634 (CRF-61); and three test/comment accuracy items (CRF-62 a cited test that does not exist, CRF-63 the trigger-order test pins names not firing order, CRF-64 a dbcrypt coverage overclaim).

Nits folded here: the LockPrefix* constants have no reader and nothing ties them to the SQL (CRF-65); the lock.go "different derivation space, so cannot collide" reasoning is false in two places (CRF-66); four cap constraint names are still raw literals so "all declared once" is inaccurate (CRF-67); the guard message says "Cannot create" on the UPDATE path (CRF-68); and the shared lock-race harness file claims two consumers but has one (CRF-69).

Strategic, and a decision for you rather than a defect (Pariston, Law): after the #28634 split, this PR is defense-in-depth plus metric hygiene, and the security consequence the description leads with now lives in #28634. If #28546 lands alone it closes the race but not orphaned rows that predate the backfill or arrive through an unenumerated path, the class that produced CRF-6, CRF-7, CRF-17, and CRF-38 across four rounds; landing #28634 first makes the protection source-agnostic. Worth stating the intended merge order in one of the two descriptions.

enterprise/coderd/aibridge.go:885

P3 [CRF-59] The seventh guard, added this round on user_ai_budget_overrides, has no handler mapping, so a violation returns a 500 with raw pq text. (Kurapika, Bisky, Hisoka, Mafuuu)

upsertUserAIBudgetOverride maps the membership-trigger violation to a 400 and hands everything else to httpapi.InternalServerError(rw, err), which writes err.Error() into Detail, so a user_ai_budget_override_user_deleted violation returns pq: Cannot create user_ai_budget_override for deleted user. CRF-40/41 fixed exactly this class for the other two guards added earlier (409 + endpoint test); the seventh, added in the same commit, was left out. Reachable only in the race (the membership trigger sorts first and rejects a plainly-deleted user with the 400), which is why it is P3, but that race is what the guard exists for. Fix: one IsCheckViolation(err, database.CheckUserAIBudgetOverrideUserDeleted) branch returning 409, matching the other two.

🤖

coderd/apikey.go:179

P3 [CRF-60] The 409 mapping stops short of api_keys, the guard's headline table, which still returns a 500 with raw pq for a reachable request. (Mafu-san, verified end-to-end)

POST /users/{id}/keys/tokens and POST /users/{id}/keys resolve the target user by UUID with no deleted filter (GetUserByID), so a stale deleted-user id reaches InsertAPIKey and returns status=500 "Failed to create API key." detail="insert API key: pq: Cannot create API key for deleted user". The PR description says "Reachable guard violations map to 409 Conflict"; that is false, two of the three reachable ones map and the third, the one the PR's own summary calls the point of the change ("for api_keys that resurrects a live session token"), does not. The 500 is not a regression, but this PR establishes the error contract and leaves the primary table outside it. Map database.CheckAPIKeyUserDeleted at both handlers, or say in the description that api_keys key creation for a deleted user is deliberately unmapped.

🤖

enterprise/cli/server_dbcrypt_test.go:263

P3 [CRF-64] A dbcrypt test comment is false and hides that dbcrypt rotate breaks on the orphaned state this migration cleans up; the description overclaims the coverage. (Mafuuu, Bisky)

The comment "Deleted users cannot have user_links or user_secrets" contradicts the same loop, which reconstructs exactly that state with SoftDeleteUserKeepingRows twelve lines up. The reason those two tables are skipped is behavioral: Rotate iterates AllUserIDs (which includes soft-deleted users) and calls UpdateUserLink / UpdateUserSecretByUserIDAndName, and the guard's UPDATE branch raises on both for a deleted user, so a single orphaned row aborts the whole coder server dbcrypt rotate run. The behavior predates this PR, and post-backfill the state should not exist, but the description claims "dbcrypt rotation/decryption over legacy orphaned rows" as tested coverage when the tested coverage is user_ai_provider_keys only (the one INSERT-only guard). Reword the comment to say why the two tables are excluded, and narrow the description's coverage claim.

🤖

🤖 This review was automatically generated with Coder Agents.

Round 6, churn-guard pass only. This is a blocking status check, not a full review: the panel does not run this round because two findings from round 5 are unaddressed with no response, and the review process holds until they are closed one way or the other.

First, real credit: this round resolved 17 of the 19 open findings and honored Law's mandatory split. The per-user caps (advisory locks, isolation contract, and the round-5 P2s CRF-52/53) are gone from this PR, split to #28870, which this branch is now stacked on. The migration was reworked exactly as asked: no DROP TRIGGER (so no ACCESS EXCLUSIVE on api_keys during upgrade, CRF-54), the backfill DELETEs replaced by the idempotent PurgeSoftDeletedUserResources reaper, the guard now locks on ownership change as well as INSERT (CRF-56), group_members guarded on the corrected justification (CRF-61), the dead TG_ARGV[2] branch removed (CRF-58), the isolation comment corrected (CRF-57), the lock-order test now pins the lock's position (CRF-55), and the trigger-order test now checks BEFORE ROW timing (CRF-63). That is a large, clean round.

What blocks the next review are two round-5 findings that received no fix and no response, and were omitted from the round-6 summary comment. Both were posted in the round-5 review body (their files are outside this PR's diff), so there was no inline thread to reply to, which is likely why they slipped:

• CRF-60 (P3): POST /users/{user}/keys/tokens and POST /users/{user}/keys still return HTTP 500 with raw pq: Cannot create API key for deleted user in the Detail for a stale deleted-user id. coderd/apikey.go maps only the unique-violation to 409 and falls through to InternalServerError. api_keys is the guard's headline table, and the round-6 description dropped the earlier false "reachable guard violations map to 409" sentence but does not say api_keys is deliberately left unmapped. Either map database.CheckAPIKeyUserDeleted to 409 at both handlers (six lines, like members.go), or state in the description that api_keys key creation for a deleted user is intentionally a 500.

• CRF-59 (P3): the seventh guard, user_ai_budget_overrides, added to close CRF-38, has no handler mapping. upsertUserAIBudgetOverride (enterprise/coderd/aibridge.go) maps only the membership-trigger violation and falls through to InternalServerError, so the guard violation surfaces as a 500 with raw pq text in exactly the race it was added to catch. Add the IsCheckViolation(err, database.CheckUserAIBudgetOverrideUserDeleted) branch returning 409, matching the two guards that already do.

Neither needs to be a fix if there is a reason not to: a one-line reply ("api_keys creation for a deleted user intentionally returns 500 because ...") closes each. A bare deferral without a ticket does not; if either is out of scope, say why or file an issue.

One related loose end, not itself blocking (CRF-64 was otherwise addressed): the comment at enterprise/cli/server_dbcrypt_test.go:263, "Deleted users cannot have user_links or user_secrets.", is still there and still false (the same test reconstructs that state with SoftDeleteUserKeepingRows); reword it while you are in the file.

The full panel will run once CRF-59 and CRF-60 are addressed or explicitly declined.

🤖 This review was automatically generated with Coder Agents.

Round 7, full panel (13) plus Netero, first panel on the guard-only PR and the new dbpurge reaper. The round-6 blockers are resolved (churn guard PROCEED): CRF-59 and CRF-60 now map their guard violations to 409 with real, race-driven handler tests.

This is in good shape and worth saying so. There are no P0 or P1 findings in the code. The guard core is solid, the authorization is correctly scoped (Kurapika found no security issues: the lock is a ResourceSystem primitive, the reaper runs as subjectDBPurge with exactly ResourceSystem:delete, the 409 mappings match specific constraints with no info disclosure), and the tests are genuine and mutation-verified (Bisky: strip the guard lock and the race tests fail red; the CRF-55 lock-position test, the CRF-63 firing-order test, and the CRF-59/60/64 fixes all hold). Moving the backfill out of the migration into a reaper (CRF-54) was the hard, correct call. Bisky: "I came in expecting costume jewelry. I stayed to appraise real stones."

The findings cluster on the reaper (never panel-reviewed until now) and one missed handler mapping. Severity: 2 P2, 4 P3, 3 nits, plus a merge blocker and a description issue below.

The reaper needs two reworks before it ships. CRF-70 (P2): it is the only delete in purgeTick with no batch bound, and it shares the single purge transaction with every retention delete. A large first pass on an old deployment, or one lost deadlock, rolls back the whole tick, so audit-log and connection-log purges make zero progress while the reaper keeps failing on the same oversized statement. Pariston put it well: a fix whose failure mode (all retention cleanup stalls, disk fills) is worse than the problem it solves. Batch it by user per tick like its siblings, or give it its own transaction. CRF-71 (P2): after the backlog clears the reaper does zero work forever but still runs every 10 minutes, and it runs unindexed: SELECT id FROM users WHERE deleted seq-scans users (every partial index is WHERE deleted = false), and group_members has no index at all, so that table is full-scanned end to end each tick, cost growing with the soft-deleted-user count. Its siblings latch after a clean pass (identifiedModuleCachePurged); this one does not. Add the indexes and latch, or gate the sweep. Also CRF-72 (P3): the reaper is the one operator-visible security cleanup with no rowcount, no metric, and no log field, so nobody can confirm the resurrectable-api_keys cleanup ran or measure it; and CRF-74 (P3, six reviewers): its "deletes in the same table order to minimize deadlock exposure" comment is false twice over (Postgres does not order data-modifying CTEs, and the reaper only touches already-committed-deleted users so it never races the cleanup).

CRF-73 (P3): the class fix for guard-violation-to-409 is one instance short again. The eighth guard, group_members, has no handler mapping, so patchGroup AddUsers for a deleted user still returns a raw 500 with pq text. Hisoka: "You built eight guards and mapped seven of them. One table still bleeds a raw 500." CheckGroupMemberUserDeleted is already declared; add the IsCheckViolation -> 409 branch. CRF-75 (P3): the reaper's test starts a global delete that can reap fixtures other tests in this PR build, a flake under a shared dev database (not CI). Nits CRF-77/78/79 are small message-consistency items.

Two things for you, not code findings:

1. CI is red across 20 required jobs (gen, fmt, lint, sqlc-vet, build, test-go, test-js, storybook, offlinedocs). This is a merge blocker and Mafu-san rates it P1: a change is not done with required red and no diagnosis in the PR. I ran the checks this diff is responsible for and they are clean: the migrations are contiguous (000590/591/592, no collision with the base's cap migration), the generated layers are in sync (AcquireUserSoftDeleteGuardLock and PurgeSoftDeletedUserResources in querier/queries.sql.go/dbmock/dbmetrics), gofmt -l is empty, dump.sql carries check_user_not_deleted, and go build passes. The breadth (frontend jobs red on a backend-only diff) points to a base/rebase/stacking problem rather than this content, but I cannot confirm the cause (gh is 401 in the review sandbox). Please diagnose and get CI green, or state what is inherited from the base; do not treat "the diff builds locally" as clearance while sqlc-vet/gen are red.

2. The description's Handlers line still says guard violations "map to 409s (organization_members, user_ai_provider_keys)", but the code now maps five tables including api_keys (both token and key endpoints) and user_ai_budget_overrides, which are the exact CRF-59/60 fixes that unblocked this round. Mafu-san rates this P2 as the third recurrence of description-vs-code drift; leaving the two headline fixes out of the one summarizing sentence undersells them. Name all the mapped tables or say "every guarded handler maps its violation to 409." (Minor, related: commit 86668c8's body cites "Migration 000591" which is now 000592; harmless if squash-merged with a corrected body.)

Finally, the strategic reminder the panel has carried since round 4: this guard-only PR stops orphan rows from being created and reaps existing ones, but it does not stop an orphan api_keys row from authenticating; that read-side filter is #28634. Until #28634 lands, an orphan key between soft-delete and the reaper's next tick still authenticates. The split is deliberate; worth confirming the intended merge order in the description.

enterprise/coderd/groups.go:302

P3 [CRF-73] The eighth guard, group_members, has no 409 handler mapping, so patchGroup AddUsers for a deleted user returns a raw 500 with pq text. (Netero, Hisoka)

Migration 000592 installs trigger_insert_group_members raising group_member_user_deleted, and CheckGroupMemberUserDeleted is declared, but referenced only by tests. patchGroup inserts each req.AddUsers id via tx.InsertGroupMember after only a uuid.Parse (no deleted filter), and the error ladder checks IsUniqueViolation/IsUnauthorizedError/Is404Error then falls through to httpapi.InternalServerError, leaking pq: Cannot create group_member for deleted user. This is the exact class fixed for api_keys (CRF-60) and user_ai_budget_overrides (CRF-59); the remediation was applied instance-by-instance and missed the fifth guarded insert path. Add if database.IsCheckViolation(err, database.CheckGroupMemberUserDeleted) returning 409 before the fallthrough. (The guard firing is correct: pre-PR this silently created an orphan group_members row that GetAuthorizationUserRoles reads into rbac.Subject.Groups.)

🤖

🤖 This review was automatically generated with Coder Agents.