feat(coderd/x/chatd): improve title generation repeatability and quality - #26982
Conversation
Pin temperature 0 on title and turn-status-label generation so repeated runs over the same input produce stable output. Extend the title prompt with a same-language rule and four worked examples, which measurably improved titles for vague, non-English, and error-paste inputs across gpt-4o-mini and claude-haiku-4-5.
|
/coder-agents-review |
|
Chat: Review posted | View chat Review historydeep-review v0.9.0 | Round 4 | Last posted: Round 4, 6 findings (1 P2, 2 P3, 1 Nit, 2 Note), APPROVE. Review Finding inventoryFindings
Contested and acknowledgedCRF-2 (Nit, quickgen.go:525) - ptr.Ref vs Go 1.26 new() builtin
CRF-4 (Note, quickgen.go:43) - Few-shot example paraphrases error the instruction says to preserve
CRF-8 (Note, quickgen.go:844) - Status-label prompt lacks same-language rule
Round logRound 1Panel (13 reviewers: Bisky, Hisoka, Mafu-san, Mafuuu, Pariston, Komugi, Ging-Go, Ryosuke, Gon, Leorio, Chopper, Kite, Meruem). Netero first pass: 1 P3 (elevated to P2 by panel convergence). Panel: 1 P2, 1 P3, 1 Nit, 1 Note new. 1 dropped. Reviewed against 121107f..7a6a6c6. Round 2Panel (13 reviewers: Bisky, Hisoka, Mafu-san, Mafuuu, Pariston, Komugi, Ging-Go, Ryosuke, Gon, Leorio, Chopper, Kite, Razor). CRF-1 fixed, CRF-3 fixed, CRF-2 panel-closed, CRF-4 closed. 1 P3, 1 Note new. 1 dropped (CRF-7, comment verbosity disagreement). Reviewed against 121107f..259022e. Round 3BLOCKED. CRF-8 (Note) silent. CRF-6 addressed (7b6ee18). No review. Round 4Panel (13 reviewers: Bisky, Hisoka, Mafu-san, Mafuuu, Pariston, Komugi, Ging-Go, Ryosuke, Gon, Leorio, Chopper, Kite, Zoro). All findings resolved. CRF-6 fix verified. CRF-8 accepted. 0 new findings. APPROVE. Reviewed against 121107f..7b6ee18. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
Clean, focused change. The temperature pin, same-language rule, and few-shot examples each target a specific observed failure mode at the right causal level. The shared quickgenTemperature constant prevents drift between the two call sites, and the PR description documents rejected alternatives with concrete failure modes.
1 P2, 1 P3, 1 Nit, 1 Note.
Pariston on this PR: "I tried to build a case against this change and couldn't."
coderd/x/chatd/quickgen_internal_test.go:791
P2 [CRF-1] Both generateStructuredTitleWithUsage (line 525) and generateStructuredTurnStatusLabel (line 959) received Temperature: ptr.Ref(quickgenTemperature), but only the title test asserts temperature in the request body. This test uses the same OpenAI-compatible server pattern, inspects the same request body map, and asserts tool_choice without asserting temperature.
"The agent propagated the production change to both sites but didn't propagate the test change. Classic 'fix one, miss sibling' when the sibling is in the same file, same test pattern, one screen away." (Mafu-san)
Fix:
require.Equal(t, quickgenTemperature, body["temperature"],
"status-label generation should pin temperature for repeatable output")(Netero P3, Mafu-san P2)
🤖
🤖 This review was automatically generated with Coder Agents.
Also tighten the quickgenTemperature comment per review feedback.
|
/coder-agents-review |
There was a problem hiding this comment.
R1 findings addressed cleanly. CRF-1 (P2, missing temperature assertion) verified fixed in 259022e. CRF-3 (P3, verbose comment) verified fixed. CRF-2 (Nit, ptr.Ref vs new()) panel-closed: all 10 commenting reviewers accepted the consistency defense. CRF-4 (Note) acknowledged, closed.
1 P3, 1 Note new this round.
Bisky on the test quality: "These test additions are lovely. Honest work."
coderd/x/chatd/quickgen.go:844
Note [CRF-8] The title prompts (automatic and manual) now instruct the model to write in the user's language. The status-label prompt has no such instruction. For a non-English user, titles will be in their language but sidebar status labels ("Finished unit tests", "Submitted PR") may remain in English. Whether status labels should match the user's language is a product decision, not a code defect. (Razor)
🤖
🤖 This review was automatically generated with Coder Agents.
The four English examples created a behavioral prior toward English output that could undermine the same-language rule for non-English messages. Add a Spanish example demonstrating same-language preservation.
|
/coder-agents-review |
There was a problem hiding this comment.
CRF-6 (P3, English-only examples) addressed in 7b6ee18 with a Spanish example. Will verify in the next round.
CRF-8 (Note, status-label prompt lacks same-language rule) has no author response. This is a product decision observation, not a code defect. Please acknowledge, contest, or defer with a ticket so the review can proceed.
Further review is blocked until CRF-8 receives a response.
🤖 This review was automatically generated with Coder Agents.
|
Re CRF-8: acknowledged, deferring. Status labels are deliberately out of scope for this PR: it pins their temperature for repeatability but does not change their prompt. Whether sidebar status labels should match the user's language is a product decision affecting the fixed English example set in
|
|
/coder-agents-review |
There was a problem hiding this comment.
All findings resolved across 4 rounds. CRF-1 (P2, missing temperature assertion) and CRF-3 (P3, verbose comment) fixed in 259022e. CRF-6 (P3, English-only examples) fixed with a Spanish example in 7b6ee18. CRF-2 (Nit) panel-closed. CRF-4 and CRF-8 (Notes) acknowledged.
Clean, proportional change: temperature pinning, same-language rule, and few-shot examples each target a specific failure mode at the right causal level. Tests verify temperature reaches the wire format for both call sites.
Hisoka on the final state: "No other structured-output call sites exist in coderd/x/chatd/. The normalizeShortTextOutput and validateGeneratedTitle post-processing functions are rune-safe and do not strip non-ASCII, so non-English titles from the same-language rule survive normalization."
🤖 This review was automatically generated with Coder Agents.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7b6ee189a8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The example titles dropped the pq error name and the acme/webapp repo, teaching the model to generalize identifiers despite the preservation rule above.
|
@codex review |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Chat title generation produced noticeably different titles for the same input across runs, occasionally rewrote non-English messages into English, and handled vague messages poorly.
Pin temperature 0 on the title and turn-status-label
ObjectCalls so repeated runs over the same input produce stable output (fantasy providers drop the setting with a call warning for models that reject it, such as OpenAI reasoning models). Extend the title prompt with a same-language rule and four worked examples, and add the language rule to the manual regeneration prompt. The schema stays a single{title}field.Compared against decomposed-schema alternatives (
{verb, object, context}assembled in Go, and extract-then-compose) on a 20-prompt corpus at temperature 0 acrossgpt-4o-miniandclaude-haiku-4-5: decomposition added token cost and new failure modes (literal<UNKNOWN>placeholders, dropped subjects, ungrammatical assemblies) without quality gains, while few-shot plus temperature 0 fixed the observed issues.