fix: stem chat search terms with the english text search config - #28319
Conversation
Chat full-text search used the 'simple' text search config, which folds case but does no stemming, so searching "refactor" did not match messages containing "refactoring". Switch every chat search site (chat titles, PR titles, and message bodies) to the 'english' config so inflected word forms match. The config baked into a tsvector must match the config used by the tsquery, so a migration rebuilds the two expression indexes and resets chat_messages.search_tsv to NULL; the existing dbpurge sweep repopulates the column with 'english' lexemes in bounded batches, newest first. A side effect of 'english' is stopword removal: queries consisting only of stopwords (e.g. search:"or") now tokenize to nothing and return an empty list, matching the existing empty-after-tokenization contract.
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
|
@codex review |
|
@f0ssel run the codex review loop please, and let's make sure we understand what exactly happens with the old |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 16a4f03a4a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Address review feedback on the english text search config switch: The migration no longer rewrites chat_messages. A full-table UPDATE of search_tsv would rewrite every indexed row inside the migration transaction, churn both partial indexes, and hold row locks and WAL until the batch commits. Instead a new search_tsv_config column records which config produced each stored vector. The pending-queue index and backfill sweep treat any row whose config is not 'english' as pending, so the existing bounded dbpurge sweep re-vectorizes the backlog incrementally, newest first. Old binaries in a rolling upgrade cannot set the column, so any 'simple' vectors they write stay pending and are rewritten by an upgraded replica's sweep instead of being stranded with mismatched lexemes. Both chatd trigger functions now exclude search_tsv_config alongside search_tsv so backfill batches do not advance message revisions or chat history_version.
…ckend-search-improvements-to-handle-partial-matches
Migration 000574 was taken on main by 000574_mcp_server_configs_organization_id after this branch was cut, which made the merge commit fail to migrate with a duplicate migration file error.
|
@codex review |
|
To use Codex here, create a Codex account and connect to github. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4290facbd5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…config During the post-migration drain window, message vectors not yet rewritten by the sweep still hold 'simple' lexemes, and an old replica in a rolling upgrade can add more. Querying those rows with an 'english' tsquery misses them even for the exact word form users typed before the config switch (stored lexeme 'refactoring' never matches the stemmed query 'refactor'). GetChats now matches each vector with the config that produced it: rows stamped 'english' use the stemmed query, rows with a NULL search_tsv_config use the 'simple' query, preserving their pre-migration matching until the sweep rewrites them. Both arms use constant tsqueries, verified via EXPLAIN to plan as a BitmapOr of two scans over idx_chat_messages_search_tsv; once the backlog drains the 'simple' arm matches no rows.
|
@codex review |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 930e3df3a3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ent sweep On rollback, rows the upgraded sweep re-vectorized held 'english' lexemes while the parent binary queries with 'simple' and its backfill only selects search_tsv IS NULL rows, so those rows were never repaired. The down migration now resets search_tsv to NULL for rows stamped search_tsv_config = 'english' before dropping the column, handing exactly the affected set back to the parent's bounded dbpurge sweep. Rows never re-vectorized keep their valid 'simple' vectors. The reset runs while the replacement trigger functions are still installed, so it does not advance message revisions or chat history_version.
…ckend-search-improvements-to-handle-partial-matches
Migration 000580 was taken on main after this branch was last synced, which made the merge commit fail to migrate with a duplicate migration file error.
…dex rebuild The migration previously dropped and recreated idx_chat_messages_search_tsv_pending with a wider predicate covering stale-config rows. A non-concurrent CREATE INDEX takes a SHARE lock on chat_messages and initially covers every eligible row, so message writes from still-running replicas would block for the whole build during a rolling upgrade. The migration now leaves chat_messages indexes untouched. The pending queue keeps its original search_tsv IS NULL predicate for the permanent workload (new messages), and a new deliberately unindexed sweep query, ReindexStaleChatMessagesSearchTsv, drains the one-time stale backlog: ORDER BY id DESC with LIMIT walks the primary key backwards and terminates early while stale rows are dense, which is the entire drain. Once a pass returns fewer rows than the batch size the scan provably reached the end of the table and dbpurge latches the stale pass off for the process lifetime; upgraded binaries always stamp search_tsv_config, so the backlog can never refill. A stale row written after the latch (old replica racing the tail of a rolling upgrade) stays correctly searchable via its recorded config and is repaired by the single stale pass after the next process restart.
…etrics tests TestMetrics/FailedChatRetentionRead and FailedChatDebugRetentionRead drive doTick against a MockStore, which fails on any call without an expectation. Register ReindexStaleChatMessagesSearchTsv alongside the existing BackfillChatMessagesSearchTsv expectations, and assert it is never called when the purge lock is held elsewhere.
Documentation CheckUpdates Needed
Automated review via Coder Agents |
|
Chat messages now have a search config, I've also intentionally not touched specific filters like |
|
@mafredri I cut out a lot of fluff, thanks for the call out |
| -- The Postgres text search configs this system has produced vectors | ||
| -- with. Extend with ALTER TYPE ... ADD VALUE if the config changes | ||
| -- again. | ||
| CREATE TYPE chat_message_search_tsv_config AS ENUM ('simple', 'english'); |
There was a problem hiding this comment.
I didn't realize when proposing enum, but we don't really use simple at all, but rather represent it currently via NULL.
I think this is fine though, and maybe we even make it an option for admins to change between English and simple at some point.
| -- again. | ||
| CREATE TYPE chat_message_search_tsv_config AS ENUM ('simple', 'english'); | ||
|
|
||
| ALTER TABLE chat_messages ADD COLUMN search_tsv_config chat_message_search_tsv_config; |
There was a problem hiding this comment.
If we want to be really thorough in representing the enum correctly, we could UPDATE chat_messages SET search_tsv_config = 'simple' WHERE cm.search_tsv; at which point the search query wouldn't use NULL and infer simple, but rather check the value explicitly.
Chat message search used the
simpletext search config, which folds case but does no stemming, sorefactordid not match messages containingrefactoring. Message bodies now index and query with theenglishconfig, which stems both sides. Titles, PR titles, and all other filters are unchanged.Migration
000585adds achat_messages.search_tsv_configenum column recording which config produced each stored vector and stamps existing vectors'simple'(a column-only update; vectors and indexes are untouched, since rewritingsearch_tsvwould block message writes on large tables). The dbpurge sweep gains a boundedReindexStaleChatMessagesSearchTsvpass that rewrites stale vectors newest first and stops for the process lifetime once drained. Until a row is rewritten,GetChatsmatches it with the config that produced it, so pre-migration exact-form searches keep working during the drain. Vectors written by old binaries mid rolling upgrade cannot stamp the config, stay pending, and self-heal on the next upgraded sweep. Steady-state DB load is unchanged.The down migration resets
englishvectors toNULLso the parent version's existing sweep rewrites them withsimple.Closes CODAGT-867
Decision log
englishstemming over prefix matching. On a 300k-row corpus a stemmed lookup is a single GIN entry-tree probe, whileterm:*prefix matching unions posting lists of every lexeme sharing the prefix (up to a full scan for short prefixes), is asymmetric, and loses thewebsearch_to_tsqueryphrase/OR/negation UX.pg_trgmrejected as a heavier index with different semantics.simpleFTS expressions and indexes, identical to main.UPDATE ... SET search_tsv = NULLrewrites every indexed row in the migration transaction, and an old replica could backfill reset rows withsimplevectors that permanently leave thesearch_tsv IS NULLqueue. The config column makes staleness explicit and both problems disappear.text(review) for the config column.🤖 This PR was generated by Coder Agents on behalf of @f0ssel.