fix(coderd): serialize per-user caps on advisory locks instead of the users row - #28870
fix(coderd): serialize per-user caps on advisory locks instead of the users row#28870ThomasK33 wants to merge 3 commits into
Conversation
b175f44 to
e968694
Compare
|
/coder-agents-review |
|
Chat: Review in progress (18/18 reviewers complete) | View chat Review history
deep-review v0.9.0 | Round 2 | Last posted: Round 2, 14 findings (1 P1, 2 P2, 4 P3, 3 Nit, 4 Note), COMMENT. Review Finding inventoryFinding inventory - PR #28870Findings
Round logRound 1Netero-only first-pass gate. P0-P2 present (CRF-1 P1, CRF-2 P2), so panel deferred until the mechanical floor is clean. 1 P1, 1 P2, 2 P3, 2 Nit, 2 Note. Reviewed against b5c49e1..e968694. Round 2Churn guard: PROCEED, all 8 R1 findings author-fixed (claims verified when reviewers encounter code). Branch rebased onto main, storybook carry dropped, migration renumbered 000590->000591. First panel round. Reviewed against 90c7533..1d5631f. Round 2 panelNetero clean (no findings). 16-reviewer panel. New: 1 P2 (CRF-9), 2 P3 (CRF-10, CRF-12), 1 Nit (CRF-14), 2 Note (CRF-11, CRF-13). CRF-9 verified empirically by orchestrator (ALTER TRIGGER RENAME -> AccessExclusiveLock on the test Postgres); it contradicts the PR body, the migration comment, and three panel reviewers (Mafuuu, Meruem, Mafu-san) who repeated the false claim. Gon rated CRF-12 P2; downgraded to P3 (test-file comment, no behavioral impact, reusability rationale independently valid; consequence is a mildly stale comment). Ryosuke's/Pariston's constraint-no-handler notes reinforce closed CRF-7 and were not re-raised. Reviewed against 90c7533..1d5631f. Law analysisNot run. R1 effective additions 844, R2 700, both < 1000 threshold. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
First-pass review only. These are mechanical findings from a single automated reviewer; the full review panel has not yet looked at this PR and will do so once the floor below is clean. These are defects worth addressing before the panel spends parallel review time.
The core change is well-scoped and unusually well-documented: the migration swaps FOR UPDATE users-row locks for transaction-scoped advisory locks, the owner-reassignment leg is real defense in depth, and the test suite is genuinely load-bearing where it claims to be (three of six new tests fail with migration 000590 removed, verified by moving the migration out and rerunning). Test density is 61%.
Severity count: 1 P1, 1 P2, 2 P3, 2 Nit, 2 Note.
The P1 is why CI is red: the title check requires the scope to be a real path containing every changed file, and the carried site/ storybook commits fall outside coderd/. The PR body says those commits drop on rebase once #28874 merges, but until then the check stays red and the PR is unmergeable. The P2 is that retry: 2 is justified for local make pre-push load but lives in the shared storybook config, so CI silently retries too and stops reporting the non-deterministic failures it absorbs.
Two test doc comments overstate what a green run proves: they claim to demonstrate the advisory lock is load-bearing, but pass unchanged when the old FOR UPDATE lock is restored. The tests are worth keeping; the comments should say what they actually cover so the next person deleting the advisory lock does not trust a false signal.
As Netero put it: "an unused parameter with an unexercised body is untested machinery that the next reader has to reason about."
🤖 This review was automatically generated with Coder Agents.
e968694 to
8f2226f
Compare
Round 2: first-pass findings addressed, storybook carry removedPushed as 8f2226f, rebased directly onto
Stack: this PR → #28546 → #28423, all rebased onto the same trunk head. #28634 is not part of this stack. Validation (PostgreSQL-backed): Generated with |
|
/coder-agents-review |
8f2226f to
c5aeace
Compare
|
/coder-agents-review |
… users row Swap the user_secrets and user_skills cap trigger functions from a users-row FOR UPDATE onto transaction-scoped per-user advisory locks (CREATE OR REPLACE, no trigger drops), rename the triggers with a zz_ prefix via ALTER TRIGGER RENAME to reserve guard-before-cap firing order, and add an owner-reassignment UPDATE leg to the skills count cap. Extracted from the soft-delete guard change per review; deliberately no runtime isolation gate.
…review Review round 1: drop the unexercised beforeCommit parameter from runLockRace (the stacked guard branch re-adds it with consumers), reword the two concurrency-test doc comments that overstated what a green run proves, attach the trigger-lock registry to real key-prefix constants quoting the exact SQL expression, say heavyweight lock where the harness said row lock, and note that the skills UPDATE-leg trigger has no production caller today.
Trunk took 000590 for workspace_agent_session_counts (#28126), so the per-user cap migration moves to 000591; the two Go comments that cite the migration number follow. No SQL changes; dump.sql regenerates identically. --- _Generated with [`mux`](https://github.com/coder/mux) • Model: `anthropic:claude-fable-5-1` • Thinking: `xhigh`_
c5aeace to
1d5631f
Compare
|
Rebased onto Generated with |
|
/coder-agents-review |
There was a problem hiding this comment.
Round 2 (first panel round). All eight round-1 findings are addressed: the branch was rebased onto main and dropped the carried storybook commits (fixing the red title check and the retry: 2 config), the two concurrency-test doc comments now say what a green run actually proves, the lock registry attaches to real constants, and the isolation contract is scoped honestly in both the test and the PR body. The panel confirmed the core design is right: moving cap serialization off the users row onto per-user advisory locks makes the FK-conflict and child-then-parent deadlock classes unrepresentable, the READ COMMITTED recount is genuinely race-free because each plpgsql statement re-snapshots after the lock wait, and the owner-reassignment leg closes a real bypass. The lock-race harness is deterministic by construction, verified under squeezed scheduling and the race detector.
Severity count: 1 P2, 2 P3, 1 Nit, 2 Note.
The P2 is the one to fix before merge, and it is instructive: the migration comment and the PR body both state that ALTER TRIGGER ... RENAME takes no ACCESS EXCLUSIVE lock, which is the load-bearing safety claim of the whole upgrade. It is false. renametrig() opens the relation with AccessExclusiveLock and holds it to end of transaction. Three reviewers caught it independently (one measured it on the project's Postgres 13, two cite the source), and I confirmed it directly on the test Postgres: ALTER TRIGGER ... RENAME reports AccessExclusiveLock on the table. Worth flagging that three other panel reviewers repeated the comment's claim as fact, which is exactly why a wrong safety comment is worse than no comment: it launders a false invariant that the stacked #28546/#28423 PRs are told to build on. The fix is to correct the claim in both places; the functional swap itself (CREATE OR REPLACE FUNCTION) is genuinely lock-free.
The rest are documentation-accuracy and latent-hazard notes on the same migration, plus one test-file comment that names PR numbers destined to drift. Nothing else blocks.
As Bisky put it: "a test suite about advisory locks and cap triggers... usually hide a fake behind a green checkmark. This one doesn't. I looked for the paste, and the stones are real."
🤖 This review was automatically generated with Coder Agents.
| -- * is a stronger lock than counting requires. | ||
| -- | ||
| -- Lock hygiene: the bodies are swapped with CREATE OR REPLACE FUNCTION and | ||
| -- the triggers renamed with ALTER TRIGGER ... RENAME, neither of which |
There was a problem hiding this comment.
P2 [CRF-9] The "lock hygiene" note and the PR body both claim ALTER TRIGGER ... RENAME takes no ACCESS EXCLUSIVE lock. It takes exactly that, on both user_secrets and user_skills. (Knuckle P2, Pariston P2, Leorio P2)
the triggers renamed with ALTER TRIGGER ... RENAME, neither of which takes ACCESS EXCLUSIVE on the tables
renametrig() opens the target relation with AccessExclusiveLock and holds it to end of transaction (src/backend/commands/trigger.c: RangeVarGetRelidExtended(stmt->relation, AccessExclusiveLock, ...)). Pariston measured it on the project's Postgres 13 (ALTER TRIGGER ... RENAME -> AccessExclusiveLock, CREATE OR REPLACE FUNCTION -> no table lock, CREATE TRIGGER -> ShareRowExclusiveLock), and I reproduced the same result directly. Avoiding DROP TRIGGER buys nothing, since DROP TRIGGER and ALTER TRIGGER RENAME take the same lock level.
Consequence: pgTxnDriver runs the whole pending batch in one transaction, so the rename's AccessExclusiveLock blocks all reads and writes on both tables until the batch commits, and queues behind any in-flight reader of those tables. On these small tables the window is short, hence P2, but the comment hands the operator a false guarantee ("blocks writes briefly, never reads") and this is the invariant the stacked #28546/#28423 PRs are told to inherit. Correct the claim in the comment and the PR body: the rename briefly takes ACCESS EXCLUSIVE, blocking reads and writes for the (fast, catalog-only) transaction. The functional fix does not need it; the ACCESS EXCLUSIVE is imported solely by the zz_ rename, whose only consumer is the downstream firing-order contract.
🤖
| -- Serialize cap checks per user so concurrent inserts or updates cannot | ||
| -- all observe the same pre-statement aggregates and exceed the caps. | ||
| -- The advisory lock avoids the users row entirely: no writer of other | ||
| -- tables referencing users is affected, and no lock cycle through the |
There was a problem hiding this comment.
P3 [CRF-10] The trigger comment asserts an absolute invariant, "no lock cycle through the users row is possible," that is false. (Hisoka)
no writer of other tables referencing users is affected, and no lock cycle through the users row is possible
Swapping FOR UPDATE for an advisory lock adds a second orderable resource while the insert still touches the users row via FK FOR KEY SHARE. Hisoka reproduced an ABBA deadlock at HEAD: tx A takes users FOR UPDATE, tx B holds advisory:X and waits on users KEY SHARE, tx A then waits on advisory:X -> 40P01 deadlock detected. It is unreachable today (no current writer takes users FOR UPDATE before a capped-child write; UPDATE users SET last_seen_at is FOR NO KEY UPDATE and does not conflict), and 40P01 is not auto-retried (only 40001 is). The defect is the comment, not the current code path. State the real contract: the cap takes no users-row lock itself, and no current writer locks the users row before a capped-child write, so no cycle exists among current callers, rather than asserting impossibility in a migration later PRs are told to rely on.
🤖
| -- pre-statement count and exceed the hard limit. See | ||
| -- enforce_user_secrets_per_user_limits for why this is an advisory | ||
| -- lock; the key registry is coderd/database/lock.go. | ||
| PERFORM pg_advisory_xact_lock(hashtextextended('user_skills_cap:' || NEW.user_id::text, 0)); |
There was a problem hiding this comment.
Note [CRF-11] The two per-user advisory keys (user_secrets_cap:<uid> and user_skills_cap:<uid>) create a cross-table lock-order axis the single users-row lock did not have. (Ryosuke, Razor, Hisoka)
A transaction writing both tables for one user acquires A then B; a concurrent one writing them in the opposite order acquires B then A, and the two deadlock. Three reviewers verified this is unreachable today: no handler writes both user_secrets and user_skills in one transaction, the cap triggers are BEFORE INSERT/UPDATE only so delete_deleted_user_resources's DELETEs never take these locks, and no owner-reassignment query exists. Worth recording because the PR names #28546/#28423 as new consumers of this migration's invariants; the honest framing is that the advisory swap removes the deadlock class for the current caller set, not universally. Any future code that writes both capped tables in one transaction must fix a lock order.
🤖
| ) | ||
|
|
||
| // This file owns the deterministic lock-race harness. On this branch its | ||
| // consumer is the per-user cap tests in user_caps_test.go; the stacked |
There was a problem hiding this comment.
P3 [CRF-12] The harness's file-split rationale is justified by naming two unmerged PR numbers (#28546, #28423) that drift. (Gon, rated P2)
the stacked soft-delete-guard (#28546) and agent-memory (#28423) changes add more consumers, which is why it lives in its own file
On this branch the harness has exactly one consumer (user_caps_test.go), so the "why it lives in its own file" claim rests entirely on work outside this repo. PR numbers drift, this PR's own migration was renumbered 000590 -> 000591 when trunk took 000590, so if either stacked PR is renumbered, abandoned, or squashed, the comment misleads. Keep the reusability rationale, drop the numbers: "Split into its own file so the cap tests here and future lock-race tests can share it." Recorded at P3 rather than Gon's P2: it is a test-file comment with no behavioral impact and the reusability rationale is independently valid, but the naming of volatile external identifiers is a real staleness hazard.
🤖
| -- deliberately no isolation-level trigger gate: a runtime gate would turn | ||
| -- a deployment-level default_transaction_isolation setting into a total | ||
| -- outage of secret and skill writes to prevent a bounded cap slip. | ||
|
|
There was a problem hiding this comment.
Note [CRF-13] The numeric caps (50, 204800, 24576, 100) are re-declared in the trigger bodies while the same values live in codersdk (MaxUserSecretsPerUserCount, MaxUserSecretsTotalValueBytes, ...), with nothing pinning the two in sync. (Meruem)
The SQL trigger is the enforcement authority; the codersdk constants supply only the numbers rendered in the 400 responses. They match today, so this is informational. But the coupling is manual: bumping the codersdk constant without editing this migration (or vice versa) would make the error report a limit the database does not enforce, and no test fails on the drift. The cap-name constants got the mechanical-drift treatment (declared once, pinned by a failing write); the cap values did not. A test asserting the SQL value equals the codersdk constant would make the drift visible.
🤖
| // check_constraint.go. | ||
| userSkillsPerUserLimitConstraint database.CheckConstraint = "user_skills_per_user_limit" | ||
| userSkillUserDeletedConstraint database.CheckConstraint = "user_skill_user_deleted" | ||
| // Raised by the insert_user_skill_fail_if_user_deleted trigger with |
There was a problem hiding this comment.
Nit [CRF-14] The comment calls insert_user_skill_fail_if_user_deleted a "trigger"; that is the function name, not the trigger. (Leorio)
The user_skill_user_deleted constraint is raised inside function insert_user_skill_fail_if_user_deleted (dump.sql:1266-1279), executed by trigger trigger_upsert_user_skills (dump.sql:5248). A reader searching pg_trigger for insert_user_skill_fail_if_user_deleted finds nothing. Say "the insert_user_skill_fail_if_user_deleted trigger function," matching the phrasing used for the cap functions elsewhere in this PR.
🤖
Moves the per-user cap triggers on
user_secretsanduser_skillsoff the users row onto transaction-scoped per-user advisory locks, and closes the owner-reassignment bypass of the skills count cap.Extracted from #28546 (review round 5, CRF-51): the cap changes are an independently justified fix for the pre-existing
FOR UPDATEhazard and must land before the soft-delete guards, whoseFOR NO KEY UPDATEusers-row lock cannot coexist with the caps'FOR UPDATE.Why
The cap trigger functions serialized per-user cap checks by locking the users row with
FOR UPDATE. That lock:FOR KEY SHARElocks foreign-key validation takes on the users row for every table referencingusers,delete_deleted_user_resourcesdeletes child rows inside the statement that updatedusers), andMigration
000591swaps the function bodies topg_advisory_xact_lock(hashtextextended('<table>_cap:' || user_id, 0))withCREATE OR REPLACE FUNCTIONand renames the triggers withALTER TRIGGER ... RENAME— noDROP TRIGGER, so noACCESS EXCLUSIVElock is taken during the upgrade. The oneCREATE TRIGGERtakesSHARE ROW EXCLUSIVEonuser_skillsonly.Owner-reassignment leg
trigger_zz_user_skills_per_user_limit_updatefiresBEFORE UPDATE ... WHEN (NEW.user_id IS DISTINCT FROM OLD.user_id), soUPDATE user_skills SET user_idrecounts against the target owner instead of bypassing the cap; same-owner updates never enter plpgsql.user_secretsneeds no new leg: its trigger already fires on every INSERT/UPDATE because same-owner updates can change the byte aggregates.Isolation contract (stated, not enforced)
The caps count committed sibling rows under the advisory lock, which is race-free at READ COMMITTED — the level every production writer uses. A deliberate decision, responding to review findings CRF-52/CRF-53 on #28546: there is no runtime isolation gate. A trigger that rejects writes outside READ COMMITTED would turn a deployment-level
default_transaction_isolationsetting into a permanent outage of secret and skill writes to prevent a bounded cap slip. Maintenance transactions running at REPEATABLE READ (dbcrypt rotation rewritesuser_secretsvalues under RR and relies on 40001 retries) still serialize on the advisory lock but count from their own snapshot, so byte caps are best-effort for such writers — exactly as they were under the old users-row lock.TestUserCapsIsolationContractpins the accepted-not-rejected half of that contract (a REPEATABLE READ secret rewrite succeeds; it passes with or without this migration). The best-effort-for-snapshot-isolated-writers half is stated here and in the migration, not exercised by a test.TestUserCapAdvisoryLockspins the advisory-lock key derivations, andTestUserSkillsCapConcurrentReassignmentis the test that fails with migration 000591 removed.Also here
coderd/database/usercaps.goso they are declared once and matched by both handlers and tests.coderd/database/lock.godocuments the SQL-side advisory key derivations (no exported Go constants: nothing in Go takes these locks) and corrects the "different derivation space, cannot collide" claim — all advisory IDs share one flat bigint keyspace; collision is vanishingly unlikely, not impossible.lockrace_test.goadds the deterministic lock-race harness (blocking/racing transactions ordered viapg_stat_activity), consumed here by the cap tests and by the stacked fix: lock parent user row in user soft-delete guards #28546/feat: add agent memory database foundation #28423 test suites.Merge order
This PR lands first, directly on
main. #28546 (soft-delete guards) stacks on the invariants established here: its guard triggers rely on the caps no longer touching the users row, and itszz_firing-order contract on the trigger names this migration establishes. #28423 (agent memory) stacks on #28546.Validation
make gen/db;go test ./coderd/database(TestUserCapAdvisoryLocks,TestUserSecretsCapConcurrentUpdates,TestUserSkillsCapOwnerReassignment,TestUserSkillsCapConcurrentInserts,TestUserSkillsCapConcurrentReassignment,TestUserCapsIsolationContract,TestUserSkillSchemaConstants);go test ./coderd/database/migrations;go test ./coderd -run 'TestUserSecrets|TestUserSkills';go test ./enterprise/dbcrypt/...(rotation under REPEATABLE READ unchanged).Generated with
mux• Model:anthropic:claude-fable-5-1• Thinking:xhigh