test(coderd): guard chat history writes in test databases - #29243
test(coderd): guard chat history writes in test databases#29243mafredri wants to merge 6 commits into
Conversation
Every chat_messages and chat_queued_messages write must run in the transaction that allocated a snapshot for the chat, which production does through chatstate.CreateChat and ChatMachine.Update. Nothing enforced this for tests, so fixtures wrote history rows the runner could not attribute to a snapshot. dbtestutil.NewDB now wraps the store in a guard that rejects the thirteen writer methods unless InsertChat or LockChatAndBumpSnapshotVersion ran earlier in the same transaction, and fails the test at cleanup for every rejection even when the caller dropped the error. dbgen.ChatMessage and the new dbgen.ChatQueuedMessage allocate inside their own transaction, and the direct seeds in chatd, chatstate, coderd, database and telemetry tests move to them or to an allocating InTx around the writer under test. A completeness test parses the generated queries so a new writer cannot be added without the guard learning about it. Raw-SQL fixtures are unchanged and unguarded by design.
The seed helpers stopped using their context once they moved to dbgen, so the parameter now misstates what they do; drop it and its callers' argument. TestUpdateChatLastTurnSummary and TestUpdateChatSummary build their store without NewDB and were still bumping the snapshot and inserting the message in two autocommit statements, the shape the guard exists to reject; they seed through dbgen.ChatMessage like every other fixture. The canary no longer re-proves the allocating path that every dbgen seed in the suite already exercises.
A database.Store handle, including a transaction handle, may be used from more than one goroutine, so the per-transaction allocation set needs a mutex like the recorder already has. Two more chatd seed helpers had kept a context parameter they no longer use; it goes the same way as the others.
The cleanup report is what fails a test whose caller dropped the rejection error, and no test exercised it: deleting the block left the guard tests green. A fake testing.TB captures NewDB's cleanups and Errorf calls so the report can be asserted directly. The soft-delete pass-through comment now names the real mechanism (GetChatMessageByID excludes deleted rows; the trigger treats an identical rewrite as a no-op), and the completeness failure message describes both directions of a mismatch.
|
/coder-agents-review
|
|
Chat: Review in progress (17/17 reviewers complete) | View chat deep-review v0.9.0 | Round 1 | Last posted: Round 1, 13 findings (4 P3, 1 P4, 5 Nit, 3 Note), COMMENT. Review Finding inventoryFinding inventory, PR #29243Findings
Round logRound 1Netero-only first pass: no findings, mechanical floor clean. Panel of 16 (Bisky, Hisoka, Mafu-san, Mafuuu, Pariston, Ging-go, Gon, Leorio, Komugi, Knuckle, Ryosuke, Takumi, Meruem, Robin + wildcards Knov, Razor). No P0/P1/P2. 4 P3, 1 P4, 3 Notes, 5 Nits posted. Reviewed against 5c73fa6..e784d9b. Dropped with reason:
About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
Test-only change, and a well-built one. The guard sits at the single layer where it sees every database.Store write a test issues (dbtestutil.NewDB wraps database.New directly, below any dbauthz), the guarded method set is pinned to the generated SQL by a completeness test, the recorder-plus-cleanup design makes a dropped error still fail the test, and the nested-transaction pointer-identity logic holds under dbauthz re-wrapping. Six reviewers independently verified the SoftDeleteChatMessageByID bypass is a genuine trigger no-op, and the disclosed per-row snapshot_version bump (one per seeded message, not per turn) is design-intended and did not break an existing assertion. No production code changes.
No P0/P1/P2. Findings: 4 P3, 1 P4, 3 Notes, 5 Nits.
The P3 cluster is about the safety net having gaps the PR's own framing does not admit. The completeness test proves an override method exists per writer query but never that it guards (11 of 13 overrides are never driven to a rejection by any test), so a future copy-paste override that drops require reopens the exact CODAGT-749 class silently. The guard's own suite asserts only rejections, never that an allocated write is permitted or that a nested transition shares the set, leaving the one load-bearing branch untested. The dbgen seed helpers now take a FOR UPDATE lock through a deadline-less context, so a future nested-seed caller holding that chat's lock hangs until the test timeout rather than failing loud. Meruem's table-driven-test fix closes the first two at once.
One deferral needs a human decision, not silent acceptance: raw-SQL fixtures write both tables with no snapshot and bypass the guard entirely (CRF-5). The PR body lists the files "for a separate cleanup" but there is no ticket, so under a no-follow-up assumption the gap is permanent. File a ticket or state explicitly that the raw-SQL boundary is accepted as-is.
Hisoka, who tried hardest to break it: "I came to fight the guard. It fought back well."
🤖 This review was automatically generated with Coder Agents.
| }) | ||
| var msgs []database.InsertChatMessagesRow | ||
| err := db.InTx(func(tx database.Store) error { | ||
| if _, err := tx.LockChatAndBumpSnapshotVersion(genCtx, seed.ChatID); err != nil { |
There was a problem hiding this comment.
P3 [CRF-3] dbgen.ChatMessage now opens its own transaction and takes FOR UPDATE on the chat row through a deadline-less context; called while that chat's row lock is already held, the seed blocks until the test times out instead of returning. (Komugi)
Before this PR
ChatMessageissued a singleInsertChatMessageswith no row lock, so it never blocked on the chat row. Now it wrapsLockChatAndBumpSnapshotVersion(... FOR UPDATE) plus the insert in a freshInTx. When a caller holds that chat'sFOR UPDATElock in an outer transaction and callsdbgen.ChatMessageon the root store, the inner transaction runs on a separate session and waits for the lock. PostgreSQL does not report this as a deadlock because the two sessions differ, andgenCtx(derived fromcontext.Background()) never cancels the waiting query. A CI timeout reads as a flake, not as a clear seed error.
No current caller triggers it (every converted seed runs after its transition closes), but this helper is now the canonical seed path across ~24 sites, so the natural next nested-seed caller lands on it. Sibling, same root cause and fix: dbgen.ChatQueuedMessage (dbgen.go:169) locks the same row the same way. Pin it by giving genCtx a bounded deadline here so a contended seed fails loud, or by reusing the caller's transaction store when one is passed so the lock is re-entrant within one session.
🤖
There was a problem hiding this comment.
Not changed. The observation holds (genCtx derives from context.Background() and both fixtures take FOR UPDATE in their own transaction), but it is a property of the whole dbgen package: every helper uses the deadline-less genCtx, including the three others that already open InTx (dbgen.go:976, 1041, 1630). A timeout in these two helpers alone would be inconsistent with the package, and reusing the caller's transaction is not possible when the caller passes the root store. No current caller holds the chat lock while seeding (full suite passes). A dbgen-wide deadline convention is a separate change; leaving this thread open for the human reviewer.
🤖 Posted using
/amend-reviewskill via Coder Agents.
| db = newChatWriteGuard(db, rejections) | ||
| t.Cleanup(func() { | ||
| for _, r := range rejections.list() { | ||
| t.Errorf("chat write guard rejected %s for chat %s outside a chat state transition; "+ |
There was a problem hiding this comment.
P3 [CRF-4] The cleanup failure message names the problem and where to look but drops the fix, and this is the one message the developer actually reads. (Leorio P3, Gon Note)
The cleanup path exists precisely for the case where the caller dropped the returned error. In that case the developer never sees the
addmessage; the cleanup line is all they get. So the remedy is missing exactly when it is the only message on screen.
The returned error from add prescribes the treatment ("write history through chatstate.ChatMachine.Update or CreateChat, or dbgen in tests"); this cleanup message says "the call site is the assertion that received this error, otherwise search for callers of ..." and stops. That first clause is also wrong for this path: if an assertion had received the error, the test would have failed there and this line would not fire. When it fires, no assertion received the error, so the clause sends the reader looking for something that does not exist. Give the cleanup message the same content the returned error carries: name the method, the chat, and dbgen.ChatMessage/dbgen.ChatQueuedMessage (or chatstate.ChatMachine.Update/CreateChat) as the fix. Gon separately noted the two messages are maintained independently and will drift.
🤖
There was a problem hiding this comment.
Keeping the text. The cleanup sentence was proposed during implementation and accepted by the orchestrator together with the decision not to capture call stacks. Its second clause, "otherwise search for callers of ", is the instruction for the dropped-error path; the first clause covers the case where the error was asserted on and the cleanup still runs (the assertion failed the test first, and the cleanup line points back at it). The fix text lives in the returned error so the two messages carry different information rather than duplicate it. Leaving open for the human reviewer.
🤖 Posted using
/amend-reviewskill via Coder Agents.
| r.rejections = nil | ||
| } | ||
|
|
||
| // chatWriteGuard is a database.Store that rejects writes to chat_messages |
There was a problem hiding this comment.
P4 [CRF-5] The guard closes the store-method instance of the bug but not the class: raw-SQL fixtures write the same tables with no snapshot and bypass it entirely, with no ticket for the promised cleanup. (Pariston)
coderd/x/chatd/chatstate/trigger_test.go,coderd/x/chatd/auto_archive_internal_test.go,coderd/database/dbpurge/dbpurge_test.go,enterprise/coderd/usage/generator_test.goand the two others named in the PR body executeINSERT INTO chat_messages/INSERT INTO chat_queued_messagesagainst a raw*sql.DB. These are exactly the snapshot-less history rows CODAGT-749 was about, and the guard is blind to them by construction.
This is operator decision 7 (raw-SQL fixtures not catered for), so it is a deliberate boundary, not an oversight; some of trigger_test.go's raw inserts legitimately exercise the DB triggers and should stay raw. But the "separate cleanup" has no ticket, so under a no-follow-up assumption the gap is permanent: a future author copying an existing raw-SQL seed in one of the non-trigger files reintroduces invisible-row history and nothing flags it. This needs a human decision: file a ticket for the fixture-seed cleanup, or state explicitly that the raw-SQL boundary is accepted as permanent.
🤖
There was a problem hiding this comment.
Operator decision (plan decision 7): raw-SQL fixtures are not catered for and stay unchanged, and this unit files no tickets. The PR body names the six files and states the boundary. Whether to file a cleanup ticket or accept the boundary as permanent is the human reviewer's call; leaving open.
🤖 Posted using
/amend-reviewskill via Coder Agents.
| @@ -13072,24 +13072,30 @@ func TestInsertChatMessages(t *testing.T) { | |||
|
|
|||
| insertMessage := func(t *testing.T, store database.Store, ctx context.Context, chatID, userID, modelConfigID uuid.UUID, content string) { | |||
There was a problem hiding this comment.
Nit [CRF-14] The insertMessage seed helper reimplements the allocate-then-insert body that this PR just gave dbgen.ChatMessage. (Robin)
This helper opens InTx, calls LockChatAndBumpSnapshotVersion, then InsertChatMessages with the full params slice, which is exactly the shape of dbgen.ChatMessage that insertSystemTextMessage, insertOrphanProviderToolCall, and the telemetry seeds all switched to. It only seeds one message to exercise the model-switch trigger, so it is a seed, not a writer-under-test, and dbgen.ChatMessage(t, store, ...) produces the same insert and fires the same trigger. Leaving the hand-rolled copy beside the new helper is the duplication the guard was meant to retire, and the allocation logic now lives in two places that can drift.
🤖
There was a problem hiding this comment.
Not changed. The enclosing test is TestInsertChatMessages, and the plan names this site (querier_test.go:13093) as one of four tests of a writer that call it directly inside an allocating InTx. The helper's InsertChatMessages call is the subject under test, including the model-switch trigger it fires; routing it through dbgen.ChatMessage would test the fixture rather than the writer. The duplication with dbgen.ChatMessage is real but intentional at this site. Leaving open for the human reviewer.
🤖 Posted using
/amend-reviewskill via Coder Agents.
…ection The completeness test proved an override exists per writer query but not that it rejects; only two of the thirteen were ever driven to the guard error. A table now calls each writer on the root handle and the completeness test requires the table to name every override, so a future override that drops the check fails a test instead of passing silently. A nested InTx case covers the one branch that shares the outer allocation set. Also widen the writer query pattern to ONLY and MERGE INTO forms, and rename require, mark, covered and rejections to say what they hold.
… own test The rejection test also proved the positive path, so a failure there would report under a name that says "rejects". The nested allocation case now has a test named for what it asserts. The table comment also claimed only the chat id is set, which the SoftDeleteChatMessageByID entry contradicts; it now describes the id the guard reads.
|
/coder-agents-review
|
Tests could write
chat_messagesandchat_queued_messagesrows through the store without allocating a chat snapshot, and such rows are invisible to consumers that gate onsnapshot_version(CODAGT-749 was one such case). Production only writes these tables through chatstate transitions, which allocate first, so the gap existed in test fixtures alone.dbtestutil.NewDBnow wraps the store in a guard that rejects the thirteen writer methods on either table unlessInsertChatorLockChatAndBumpSnapshotVersionran earlier in the same transaction, and fails the test at cleanup for every rejection even when the caller dropped the error.dbgen.ChatMessageallocates inside its own transaction and the newdbgen.ChatQueuedMessagedoes the same; the direct seeds in chatd, chatstate, coderd, coderd/database and telemetry tests use them or an allocatingInTxaround the writer under test.TestGetChatsFilter.makeUnreadseededContentVersion0 and now seeds the current version throughdbgen.ChatMessage; thehas_unreadfilter it tests does not readcontent_version. Three tests indbtestutilcover the guard: every guarded writer is rejected on the root handle andInsertChatMessagesinside an unallocated transaction, with no rows landing, while a nestedInTxwrite after the outer transaction allocated lands; every generated query that writes either table (INSERT INTO,MERGE INTO,UPDATE,DELETE FROM, with or withoutONLY) is guarded or listed assearch_tsvmaintenance and every guard override has a rejection case; and the cleanup report fires when the error is dropped.No production code changes. Raw-SQL fixtures are unchanged and unguarded by design; they live in
coderd/x/chatd/chatstate/trigger_test.go,coderd/x/chatd/auto_archive_internal_test.go,coderd/database/dbpurge/dbpurge_test.go,coderd/database/querier_test.go,coderd/database/migrations/migrate_test.goandenterprise/coderd/usage/generator_test.go, for a separate cleanup.TestUpdateChatLastTurnSummaryandTestUpdateChatSummarybuild their store withdatabase.New(testSQLDB(t))and stay outside the guard, but they seed throughdbgen.ChatMessagenow.Verification: full
make test(31272 tests, 0 failures),make lint,make fmt,make gen(no drift) andmake build. With the guard installed and the seeds still unconverted, every rejection traced to a test seed; none came from a production path.Sibling finding, not addressed here:
InsertChatQueuedMessage,DeleteChatQueuedMessage,DeleteAllChatQueuedMessages,PopNextQueuedMessage,ReorderChatQueuedMessageToFrontandSoftDeleteContextFileMessageshave no callers outside the generated wrappers.Refs CODAGT-1019
Plan
CODAGT-1019: test-time Store guard for chat history and queue writes
Direction
Outcome
Every Go test that opens a database through
dbtestutil.NewDBfails when any code, production or test, calls adatabase.Storemethod that inserts or changeschat_messagesorchat_queued_messagesrows for a chat without having allocated a snapshot for that chat earlier in the same transaction. Production satisfies this by construction:chatstate.CreateChatinserts the chat row,chatstate.ChatMachine.UpdatecallsLockChatAndBumpSnapshotVersion, both before any write. Nothing in production changes.Observable end state
dbtestutil.NewDBreturns the store wrapped in a guard: adatabase.Storeimplementation embedding the real store, overridingInTx, the two snapshot allocators and the thirteen writer methods.dbgen.ChatMessageand a newdbgen.ChatQueuedMessageallocate a snapshot inside their own transaction. Tests that seed rows use them. Tests that exercise a writer method itself call it insideInTxafterLockChatAndBumpSnapshotVersion. No opt-out exists.coderd/database/dbtestutil/db_internal_test.goasserts that every generated query whose SQL writes either table is guarded or explicitly listed as a non-history writer, so a new writer query cannot be added without the guard learning about it.Recommended direction and reason
A
database.Storewrapper, installed only bydbtestutil.NewDB. Reason: production writes these tables only throughdatabase.Storemethods (verified: no raw SQL on either table in non-generated Go), so the wrapper sees every production write; it enforces the same rule the database triggers would with none of their machinery (no SQL, no duplicated column rules, no PostgreSQL version sensitivity, no sequence); it reports the Go call site of a rejection; and it could run in production later by wrapping the store incoderd.go, with no schema change. Precedent:enterprise/dbcryptanddbmetricsare hand-written wrappers embeddingdatabase.Store. Measured on maine9429903d4with the prototype installed in everyNewDB: zero production-path rejections across chatd (647 pass), chatstate, chattool, chathooks, chatdebug, chatprompt, dbpurge, telemetry, toolsdk, the coderd chat tests and the coderd/database chat tests; 38 failing tests, all at 28 direct test seeds; nestedUpdateinsideUpdateand dbauthz above the guard (coderdtest) worked without false rejections.High-level shape
NewDB, with the rejection recorder read at cleanup.dbgen.ChatMessageallocates a snapshot;dbgen.ChatQueuedMessageis added with the same shape.dbgencalls, 4 call the writer inside an allocatingInTx.db_internal_test.go.Ruled out (operator decisions and evidence)
chats.xmin, Go callingset_config, a bespoke analyzer, ruleguard: rejected earlier; reasons in the vault and~/my-agent/artifacts/coder/codagt-1019-ideation/BRIEF.md.xmingrouping: rejected on evidence in the four lens reports.chatstate-based fixture path fordbgen: transitions cannot produce every seeded shape.database.Store): not ruled out; it is adatabase.StoreAPI change that closes one class the wrapper cannot see (a writer called on the callback store insideUpdate). Its timing is a separate decision with Hugo Dutka and the database owners.Decisions already made
database.Storewrapper in tests. Layers compose; the capability handle stays opendbtestutil.NewDBdbgen; no opt-outchat_messagesandchat_queued_messagesare guardedInsertChatMessages,SoftDeleteChatMessageByID,SoftDeleteChatMessagesAfterID,SoftDeleteContextFileMessages,InsertChatQueuedMessage,InsertChatQueuedMessageWithCreator,DeleteChatQueuedMessage,DeleteChatQueuedMessageReturningCount,DeleteAllChatQueuedMessages,DeleteAllChatQueuedMessagesReturningCount,PopNextQueuedMessage,ReorderChatQueuedMessageToFront,ReorderChatQueuedMessageToHead. Allocators:InsertChat,LockChatAndBumpSnapshotVersion. Explicitly unguarded writers of the tables:BackfillChatMessagesSearchTsv,ReindexStaleChatMessagesSearchTsvqueries/chats.sqlthat writes either table; the two search queries change onlysearch_tsvcolumns, which the shipped triggers excludeSoftDeleteChatMessageByIDtakes a message id; the guard reads the message's chat withGetChatMessageByIDbefore checkingInTxpasses the same guard instance to nested callbacks when the inner transaction store is the same value as the outer (tx == g.Store), so nestedUpdateshares the allocation set; otherwise it wraps the transaction store with a fresh set. The root handle has a nil set and rejects every guarded writesqlQuerier.InTxreuses itself when already in a transaction (db.go:186-196); prototype run showed no false rejection on nested transitionsNewDB; cleanup callst.Errorfper rejection naming the method and chat and stating that the call site is the assertion that received the error, otherwise the callers of that method; the method also returns the error. No call stack capturecoderd/database/dbtestutil(one file), unexported, wired inNewDBarounddatabase.Newand below dbauthz. Moving it to its own package is deferred until a production use existsdbcryptshows the wrapper shape needs no shared packagecoderd/database/queries.sql.gofor query constants whose SQL matches `INSERT INTOWrappers()appends the guard's name so double wrapping is detectabledatabase.Store.Wrappersexists for this purpose (db.go:38, dbauthz uses it)db_internal_test.goARCHITECTURE.mdis not editedAssumptions, constraints, tradeoffs
e9429903d4: production writes both tables only through the methods in decision 8, all called fromcoderd/x/chatd/chatstate/transitions.go;InsertChatandLockChatAndBumpSnapshotVersioneach have one production caller.database.Storecalls. Raw SQL by tests is invisible to it by design.Updatecallback and a writer that allocates a snapshot itself both pass. The first is what the capability handle would close.database.New(sqlDB)bypassNewDBand the guard; none were found in the run packages.-- name:headers and a table-name pattern.Risks that affect the direction
enterprise/coderd/usage/generator_test.gohas onedbgensite, fixed by thedbgenchange, and one raw SQL site, unaffected). Trigger: CI failure with the guard message naming the method and chat. Response: replace the seed with thedbgencall.snapshot_versionafter seeding. Trigger: assertion failure. Response: recompute; the shift is one per seeded row.NewDB.