Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix: stem chat search terms with the english text search config - #28319

Merged
f0ssel merged 18 commits into
mainfrom
garrett/codagt-867-backend-search-improvements-to-handle-partial-matches
Aug 25, 2026
Merged

fix: stem chat search terms with the english text search config#28319
f0ssel merged 18 commits into
mainfrom
garrett/codagt-867-backend-search-improvements-to-handle-partial-matches

Conversation

@f0ssel

@f0ssel f0ssel commented Aug 19, 2026

Copy link
Copy Markdown
Member

Chat message search used the simple text search config, which folds case but does no stemming, so refactor did not match messages containing refactoring. Message bodies now index and query with the english config, which stems both sides. Titles, PR titles, and all other filters are unchanged.

Migration 000585 adds a chat_messages.search_tsv_config enum column recording which config produced each stored vector and stamps existing vectors 'simple' (a column-only update; vectors and indexes are untouched, since rewriting search_tsv would block message writes on large tables). The dbpurge sweep gains a bounded ReindexStaleChatMessagesSearchTsv pass that rewrites stale vectors newest first and stops for the process lifetime once drained. Until a row is rewritten, GetChats matches it with the config that produced it, so pre-migration exact-form searches keep working during the drain. Vectors written by old binaries mid rolling upgrade cannot stamp the config, stay pending, and self-heal on the next upgraded sweep. Steady-state DB load is unchanged.

The down migration resets english vectors to NULL so the parent version's existing sweep rewrites them with simple.

Closes CODAGT-867

Decision log
  • english stemming over prefix matching. On a 300k-row corpus a stemmed lookup is a single GIN entry-tree probe, while term:* prefix matching unions posting lists of every lexeme sharing the prefix (up to a full scan for short prefixes), is asymmetric, and loses the websearch_to_tsquery phrase/OR/negation UX. pg_trgm rejected as a heavier index with different semantics.
  • Scope to message bodies only (review). Titles and PR titles keep their simple FTS expressions and indexes, identical to main.
  • Config-versioned queue over migration-time reset (review). An unqualified UPDATE ... SET search_tsv = NULL rewrites every indexed row in the migration transaction, and an old replica could backfill reset rows with simple vectors that permanently leave the search_tsv IS NULL queue. The config column makes staleness explicit and both problems disappear.
  • Enum over text (review) for the config column.
  • Trade-offs. English stopwords are dropped from message search (stopword-only queries match no messages); stemming is English-specific; titles do not stem; prefix matching remains out of scope.

🤖 This PR was generated by Coder Agents on behalf of @f0ssel.

Chat full-text search used the 'simple' text search config, which folds
case but does no stemming, so searching "refactor" did not match
messages containing "refactoring". Switch every chat search site
(chat titles, PR titles, and message bodies) to the 'english' config so
inflected word forms match.

The config baked into a tsvector must match the config used by the
tsquery, so a migration rebuilds the two expression indexes and resets
chat_messages.search_tsv to NULL; the existing dbpurge sweep repopulates
the column with 'english' lexemes in bounded batches, newest first.

A side effect of 'english' is stopword removal: queries consisting only
of stopwords (e.g. search:"or") now tokenize to nothing and return an
empty list, matching the existing empty-after-tokenization contract.
@linear-code

linear-code Bot commented Aug 19, 2026

Copy link
Copy Markdown

CODAGT-867

@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@f0ssel
f0ssel requested a review from mafredri August 19, 2026 18:40
@f0ssel
f0ssel marked this pull request as ready for review August 19, 2026 18:53
@ibetitsmike

Copy link
Copy Markdown
Collaborator

@codex review

@ibetitsmike

Copy link
Copy Markdown
Collaborator

@f0ssel run the codex review loop please, and let's make sure we understand what exactly happens with the old simple vectors

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 16a4f03a4a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/database/migrations/000574_chat_search_english_config.up.sql Outdated
Comment thread coderd/database/queries/chats.sql Outdated
f0ssel added 3 commits August 20, 2026 19:12
Address review feedback on the english text search config switch:

The migration no longer rewrites chat_messages. A full-table UPDATE of
search_tsv would rewrite every indexed row inside the migration
transaction, churn both partial indexes, and hold row locks and WAL
until the batch commits.

Instead a new search_tsv_config column records which config produced
each stored vector. The pending-queue index and backfill sweep treat
any row whose config is not 'english' as pending, so the existing
bounded dbpurge sweep re-vectorizes the backlog incrementally, newest
first. Old binaries in a rolling upgrade cannot set the column, so any
'simple' vectors they write stay pending and are rewritten by an
upgraded replica's sweep instead of being stranded with mismatched
lexemes.

Both chatd trigger functions now exclude search_tsv_config alongside
search_tsv so backfill batches do not advance message revisions or
chat history_version.
…ckend-search-improvements-to-handle-partial-matches
Migration 000574 was taken on main by
000574_mcp_server_configs_organization_id after this branch was cut,
which made the merge commit fail to migrate with a duplicate migration
file error.
@f0ssel

f0ssel commented Aug 20, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@f0ssel

f0ssel commented Aug 20, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4290facbd5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/database/queries/chats.sql Outdated
…config

During the post-migration drain window, message vectors not yet
rewritten by the sweep still hold 'simple' lexemes, and an old replica
in a rolling upgrade can add more. Querying those rows with an
'english' tsquery misses them even for the exact word form users typed
before the config switch (stored lexeme 'refactoring' never matches
the stemmed query 'refactor').

GetChats now matches each vector with the config that produced it:
rows stamped 'english' use the stemmed query, rows with a NULL
search_tsv_config use the 'simple' query, preserving their
pre-migration matching until the sweep rewrites them. Both arms use
constant tsqueries, verified via EXPLAIN to plan as a BitmapOr of two
scans over idx_chat_messages_search_tsv; once the backlog drains the
'simple' arm matches no rows.
@f0ssel

f0ssel commented Aug 24, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@f0ssel

f0ssel commented Aug 24, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 930e3df3a3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/database/migrations/000580_chat_search_english_config.down.sql Outdated
@f0ssel f0ssel changed the title feat: stem chat search terms with the english text search config fix: stem chat search terms with the english text search config Aug 24, 2026
…ent sweep

On rollback, rows the upgraded sweep re-vectorized held 'english'
lexemes while the parent binary queries with 'simple' and its backfill
only selects search_tsv IS NULL rows, so those rows were never
repaired.

The down migration now resets search_tsv to NULL for rows stamped
search_tsv_config = 'english' before dropping the column, handing
exactly the affected set back to the parent's bounded dbpurge sweep.
Rows never re-vectorized keep their valid 'simple' vectors. The reset
runs while the replacement trigger functions are still installed, so
it does not advance message revisions or chat history_version.
f0ssel added 5 commits August 24, 2026 13:41
…ckend-search-improvements-to-handle-partial-matches
Migration 000580 was taken on main after this branch was last synced,
which made the merge commit fail to migrate with a duplicate migration
file error.
…dex rebuild

The migration previously dropped and recreated
idx_chat_messages_search_tsv_pending with a wider predicate covering
stale-config rows. A non-concurrent CREATE INDEX takes a SHARE lock on
chat_messages and initially covers every eligible row, so message
writes from still-running replicas would block for the whole build
during a rolling upgrade.

The migration now leaves chat_messages indexes untouched. The pending
queue keeps its original search_tsv IS NULL predicate for the permanent
workload (new messages), and a new deliberately unindexed sweep query,
ReindexStaleChatMessagesSearchTsv, drains the one-time stale backlog:
ORDER BY id DESC with LIMIT walks the primary key backwards and
terminates early while stale rows are dense, which is the entire drain.
Once a pass returns fewer rows than the batch size the scan provably
reached the end of the table and dbpurge latches the stale pass off for
the process lifetime; upgraded binaries always stamp search_tsv_config,
so the backlog can never refill. A stale row written after the latch
(old replica racing the tail of a rolling upgrade) stays correctly
searchable via its recorded config and is repaired by the single stale
pass after the next process restart.
…etrics tests

TestMetrics/FailedChatRetentionRead and FailedChatDebugRetentionRead
drive doTick against a MockStore, which fails on any call without an
expectation. Register ReindexStaleChatMessagesSearchTsv alongside the
existing BackfillChatMessagesSearchTsv expectations, and assert it is
never called when the purge lock is held elsewhere.
@coderagents

coderagents Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Documentation Check

Updates Needed

  • coderd/exp_chats.go (source annotation; regenerated docs/reference/api/chats.md) - The q parameter's search: description previously claimed stemming applied "across chat titles, PR titles, PR numbers, and message bodies," which was inaccurate. Fixed in 515e6a06: the annotation now states "message bodies match English word stems, e.g. refactor matches refactoring, and ignore English stopwords; titles and PR titles match whole words case-insensitively without stemming," and docs/reference/api/chats.md (plus swagger.json/docs.go) were regenerated. This matches the code (titles use the simple full-text config, message bodies use english).

Automated review via Coder Agents

@f0ssel

f0ssel commented Aug 24, 2026

Copy link
Copy Markdown
Member Author

Chat messages now have a search config, NULL or 'simple' will be picked up by background job and converted to english config async. I intentionally did not create a new index for the ReindexStaleChatMessagesSearchTsv because I did not want to do so synchronously in migrations and I figure the effective lifetime of the query being used is relatively short.

I've also intentionally not touched specific filters like pr_title: in this PR, but can do follow ups to change the behavior there too to english search if we'd like. To me the filters being more specific still makes sense enough to leave alone.

@f0ssel
f0ssel requested review from johnstcn and mafredri and removed request for mafredri August 24, 2026 17:28
Comment thread coderd/database/dbpurge/dbpurge.go Outdated
Comment thread coderd/database/dbpurge/dbpurge.go Outdated
Comment thread coderd/database/migrations/000585_chat_search_english_config.up.sql Outdated
Comment thread coderd/database/queries/chats.sql Outdated
Comment thread coderd/database/queries/chats.sql Outdated
Comment thread coderd/x/chatd/ARCHITECTURE.md Outdated
@f0ssel

f0ssel commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@mafredri I cut out a lot of fluff, thanks for the call out

-- The Postgres text search configs this system has produced vectors
-- with. Extend with ALTER TYPE ... ADD VALUE if the config changes
-- again.
CREATE TYPE chat_message_search_tsv_config AS ENUM ('simple', 'english');

@mafredri mafredri Aug 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't realize when proposing enum, but we don't really use simple at all, but rather represent it currently via NULL.

I think this is fine though, and maybe we even make it an option for admins to change between English and simple at some point.

-- again.
CREATE TYPE chat_message_search_tsv_config AS ENUM ('simple', 'english');

ALTER TABLE chat_messages ADD COLUMN search_tsv_config chat_message_search_tsv_config;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to be really thorough in representing the enum correctly, we could UPDATE chat_messages SET search_tsv_config = 'simple' WHERE cm.search_tsv; at which point the search query wouldn't use NULL and infer simple, but rather check the value explicitly.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this is cleaner

@f0ssel
f0ssel merged commit 378ac91 into main Aug 25, 2026
31 checks passed
@f0ssel
f0ssel deleted the garrett/codagt-867-backend-search-improvements-to-handle-partial-matches branch August 25, 2026 13:08
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 25, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants