CODAGT-722: dbpurge backfill to populate and maintain chat_messages.search_tsv

Issue: https://linear.app/codercom/issue/CODAGT-722
Depends on: CODAGT-721 schema (PR #26968, migration 000541_chat_search_schema)
Branch: cian/codagt-722-dbpurge-sweep-to-backfill-and-maintain, stacked on the 721 branch until #26968 merges. Deliverable: one draft PR.

Context

Migration 000541 added chat_messages.search_tsv (all NULL), the search GIN index, and the pending-queue partial index (idx_chat_messages_search_tsv_pending: search_tsv IS NULL AND deleted = false AND visibility IN ('user','both') AND role IN ('user','assistant'), btree on id DESC). This task drains that queue: it computes tsvectors for pending rows in batches, newest first, inside the existing dbpurge tick.

Design

Query (sqlc, in coderd/database/queries/chats.sql)

One statement, select+update combined (mirrors the benchmarked form):

-- name: BackfillChatMessagesSearchTsv :execrows
WITH batch AS (
    SELECT id FROM chat_messages
    WHERE search_tsv IS NULL
      AND deleted = false
      AND visibility IN ('user', 'both')
      AND role IN ('user', 'assistant')
    ORDER BY id DESC
    LIMIT @batch_size::int
)
UPDATE chat_messages cm
SET search_tsv = COALESCE(
    to_tsvector('simple', chat_message_search_text(cm.content)),
    ''::tsvector)
FROM batch WHERE cm.id = batch.id;

• WHERE clause matches the pending index predicate exactly (required for the index to serve it); add a comment stating this coupling and referencing the index name.
• COALESCE(..., '') sentinel: distinguishes "backfilled, no text" from "pending". Comment this.
• :execrows returns affected rows; the backfill loop stops when < batch size.
• dbauthz: follow the pattern of existing dbpurge queries (e.g. DeleteOldWorkspaceAgentLogs) for the generated authz wrapper; context is already dbauthz.AsDBPurge.

Backfill loop (in coderd/database/dbpurge/dbpurge.go)

• New step inside the existing purgeTick transaction, after the current purge work (lock already held via TryAcquireLock(LockIDDBPurge)).
• Constants: chatSearchBackfillBatchSize = 10_000 (benchmark sweet spot, ~800ms/batch), chatSearchBackfillMaxBatches = 5 per tick.
• Loop: call BackfillChatMessagesSearchTsv up to maxBatches times; stop early when rows affected < batch size. Caps per-tick transaction growth at ~4-5s even on a cold 1M-row backlog.
• Record the correctness rationale as a comment (queue membership is per-row, content immutable post-insert, soft-deletes handled by index maintenance).
• Metrics: increment the existing coderd_dbpurge_records_purged_total pattern is wrong semantically ("purged"); add coderd_dbpurge_chat_search_rows_backfilled_total counter (no labels), registered alongside the existing ones.

Drain-rate trade-off (flagged decision)

5 batches x 10k per 10-minute tick = 50k rows/tick. A dev.coder.com-scale backlog (~500k eligible rows) drains in ~100 minutes; typical customer deployments in one or two ticks. Alternative: loop until empty on the first tick (simple, but a single multi-minute transaction holding the dbpurge lock delays all other purge work and bloats the tx). Recommendation: ship the cap; revisit only if field feedback demands faster initial drain.

Red: tests first

Postgres-backed, in coderd/database/dbpurge/dbpurge_test.go, following the existing quartz + awaitDoTick patterns (non-parallel, shared lock ID):

1. Drain converges: seed eligible + ineligible messages (mix of no-text, tool-role, model-only, deleted, child-chat irrelevant here); run ticks; eventually zero pending rows; all eligible rows have non-NULL search_tsv; ineligible rows remain NULL.
2. Newest first: seed > 1 batch of eligible rows with a small batch size (override via test hook or use the constant with small seed); after one batch, the backfilled rows are the highest ids.
3. Sentinel: no-text eligible rows end up with ''::tsvector (not NULL) and do not reappear in the pending query.
4. Per-tick bound: seed > maxBatches x batchSize rows; one tick backfills at most maxBatches x batchSize; next tick continues.
5. Deleted rows: soft-delete a not-yet-backfilled eligible row; it is never backfilled (search_tsv stays NULL) and does not appear in pending results.
6. Post-drain freshness: after drain, insert a new eligible message; next tick backfills it.
7. Steady state: tick with empty queue performs no updates (rows affected 0 on first batch; loop exits).
8. Metrics: counter increments by rows backfilled.
9. Lock safety: reuse the existing mock-store contention test pattern to assert the backfill is skipped when the lock is held.

Batch size/max batches need to be overridable for tests (package-level vars or an Option, following how dbpurge handles clock injection; prefer an Option to keep tests race-safe).

Green: implementation

1. Add the sqlc query; make gen (querier, dbauthz, dbmetrics, dbmock all regenerate; no audit-table entry, internal operation).
2. Add the backfill step + metrics + options to dbpurge.
3. Make tests pass; make fmt && make lint; run make test RUN=TestPurge (or the relevant test names) and make test-race for the new tests.

Refactor

• Re-check the backfill loop against the rest of purgeTick for structural consistency (error wrapping style, logging fields).
• Confirm comments capture: predicate coupling, sentinel, immutability rationale, and the drain-rate cap; nothing restating the code.

Verification

• make gen no stray diff; make test, make test-race, make lint pass.
• Manual EXPLAIN smoke check on a seeded DB: pending SELECT uses idx_chat_messages_search_tsv_pending.
• Draft PR stacked on DNM: feat(coderd): add chat search schema #26968 with plan in collapsible section and Coder Agents disclosure.