feat: add reasoning effort to model overrides - #27061
Conversation
26325c8 to
3685d3c
Compare
3028ea1 to
578101a
Compare
3685d3c to
c1f9fc4
Compare
578101a to
19b4375
Compare
c1f9fc4 to
2f40dd5
Compare
19b4375 to
8996938
Compare
2f40dd5 to
35efe67
Compare
8996938 to
7d48361
Compare
35efe67 to
5286cc9
Compare
7d48361 to
a37dfd8
Compare
5286cc9 to
cc24930
Compare
a37dfd8 to
ff41fb0
Compare
cc24930 to
fd9e20c
Compare
ff41fb0 to
59813d7
Compare
fd9e20c to
226f860
Compare
59813d7 to
09834d6
Compare
226f860 to
1f8421d
Compare
09834d6 to
126499d
Compare
1f8421d to
d6a7d4c
Compare
126499d to
cd1a31a
Compare
d6a7d4c to
54c3345
Compare
cd1a31a to
d8d2038
Compare
54c3345 to
fd8c08a
Compare
d8d2038 to
63d311c
Compare
fd8c08a to
76f68fe
Compare
63d311c to
b56cf6e
Compare
|
/coder-agents-review |
|
Chat: Review in progress (6/6 reviewers complete) | View chat Review historydeep-review v0.9.0 | Round 3 | Last posted: Round 3, 17 findings (1 P0, 2 P2, 11 P3, 1 P4, 2 Note), APPROVE. Review Finding inventoryFinding InventoryFindings
Contested and acknowledgedCRF-2 (P3, title_override.go:18) - Triple uuid[:effort] parsing duplication
CRF-7 (P3, subagent.go:427) - withResolvedReasoningEffort silent error swallowing
CRF-8 (P3, exp_chats.go:564) - parseChatModelCallConfig duplicates unmarshalChatModelCallConfig
CRF-12 (Note, exp_chats.go:1308) - Personal override effort bypasses selectability at chat creation
CRF-13 (Note, PersonalModelOverrideRow.tsx:281) - Form/display effort divergence after admin config change
Round logRound 1Panel. 2 P2, 7 P3, 1 P4, 2 Note new. 3 dropped. Reviewed against 76f68fe..b56cf6e. Round 2Panel. CRF-1,3,4,5,6,9,10,11 addressed. CRF-2 panel closed (5/7). CRF-7 panel closed (4/6). CRF-8 panel closed (6/6). CRF-12,13 acknowledged. 1 P0, 3 P3 new. 1 dropped. Reviewed against 3c16e1e..b4d5ac0. Round 3Panel verification. CRF-17,18,19,20 addressed. No new findings. All open items resolved. Reviewed against 3c16e1e..5f12b74. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
The feature is well-structured: reasoning effort threads cleanly through deployment overrides, personal overrides, subagent child chats, and title generation, converging at ResolveReasoningEffort for runtime clamping. Test coverage is thorough for the happy paths, with 9 new test cases covering positive flows, negative validation, round-tripping, non-model mode rejection, child chat propagation, and deployment default reflection.
Two P2s, seven P3s, one P4, two Notes.
"Someone hits this at 2 AM from the admin override settings. They get 'Invalid reasoning_effort.' THAT'S IT. No list of valid values, no guidance, nothing. Meanwhile three other endpoints in the same file hand them the exact answer."
Severity summary: 2 P2 (API contract asymmetry, error message gaps), 7 P3 (test gaps, duplication, silent errors), 1 P4, 2 Notes.
coderd/x/chatd/personal_model_override_test.go:93
P3 [CRF-6] TestParseChatPersonalModelOverride unit test table was not extended for the new model:uuid:effort parsing paths.
The function was extended (personal_model_override.go:65-79) to parse model:<uuid>:<effort> and reject model:<uuid>: (trailing colon with empty effort). The table has seven rows, none of which exercise the ReasoningEffort field. Two missing cases:
model:<uuid>:highshould parse withReasoningEffort: ptr.Ref("high")model:<uuid>:(trailing colon, empty effort) should be malformed
The integration tests cover the HTTP round-trip, but the unit test is where a regression in the parsing logic would surface cheapest.
(Bisky P3, Mafu-san P2, Meruem P3, Chopper P3, Pariston Nit)
🤖
site/src/pages/AISettingsPage/CoderAgentsPage/CoderAgentsPageView.tsx:21
P3 [CRF-10] SaveModelOverride type alias declares only { readonly model_config_id: string }, but SubagentModelOverrideSettings sends { model_config_id, reasoning_effort } through this callback.
TypeScript's structural subtyping lets the wider object pass at the call site, so the runtime works. But the type is a lie: anyone refactoring this view who trusts the type and reconstructs the request from the declared shape will drop reasoning_effort silently, with no compiler complaint.
Fix: add readonly reasoning_effort?: string to the type alias.
(Nami)
🤖
coderd/x/chatd/subagent.go:964-965
P4 [CRF-11] The doc comment enumerates universal fields ("modelConfigIDOverride and planModeOverride apply to any subagent") but omits reasoningEffortOverride, which is also universal: subagent_catalog.go passes it for both general and explore subagents.
// modelConfigIDOverride, reasoningEffortOverride, and planModeOverride apply
// to any subagent.(Gon P2, Leorio Nit)
🤖
🤖 This review was automatically generated with Coder Agents.
| rawID, rawEffort, hasEffort := strings.Cut(trimmed, ":") | ||
| modelConfigID, err := uuid.Parse(rawID) | ||
| if err != nil || (hasEffort && rawEffort == "") { | ||
| return parsedChatModelOverride{}, xerrors.Errorf("parse chat model override: %w", err) |
There was a problem hiding this comment.
P3 [CRF-1] xerrors.Errorf wraps a nil error when the trailing-colon guard fires independently of uuid.Parse.
When hasEffort && rawEffort == "" is true but uuid.Parse succeeded (err == nil), xerrors.Errorf("parse chat model override: %w", nil) produces a non-nil error with message "parse chat model override: %!w(<nil>)". The error path is correct (an error is returned), but the message is nonsensical.
Fix: produce a specific error when the trailing-colon guard fires, e.g. xerrors.New("parse chat model override: trailing colon with empty reasoning effort").
(Netero)
🤖
There was a problem hiding this comment.
Coder Agents generated.
Fixed. The parser now returns a specific error for the trailing colon with empty reasoning effort case instead of wrapping a nil error.
There was a problem hiding this comment.
Verified fixed in b4d5ac05. The parser now returns a specific error via xerrors.New for the trailing colon case.
🤖
|
|
||
| const titleGenerationOverrideContext = "title_generation" | ||
|
|
||
| type parsedModelOverride struct { |
There was a problem hiding this comment.
P3 [CRF-2] parsedModelOverride / parseModelOverride duplicates parsedChatModelOverride / parseChatModelOverride from coderd/exp_chats.go:521, and a third copy exists in ParseChatPersonalModelOverride (personal_model_override.go:65-79).
All three implement the same uuid[:effort] parsing: strings.Cut(trimmed, ":") + uuid.Parse(rawID) + hasEffort && rawEffort == "" guard + effort extraction. The only differences are return types (pointer vs value UUID, error vs bool). A shared helper would eliminate the divergence risk.
(Netero, Robin)
🤖
There was a problem hiding this comment.
Coder Agents generated.
Not changed intentionally. These parsers have distinct API/storage semantics and return shapes, so sharing a helper here would add indirection without eliminating much code. Keeping them separate also avoids coupling chatd title override parsing to the HTTP storage parser.
There was a problem hiding this comment.
Panel closed (5/7 accept). Five reviewers confirmed the return types and error semantics differ enough to justify separate implementations. The coupling cost of merging outweighs the duplication risk for ~8 lines of shared logic.
🤖
| httpapi.Write(ctx, rw, status, *resp) | ||
| return | ||
| } | ||
| modelConfig, err := lookupEnabledChatModelConfigByID(ctx, api.Database, parsedModelConfigID) |
There was a problem hiding this comment.
P3 [CRF-3] Double DB lookup for the same model config in putUserChatPersonalModelOverride.
validateUserChatModelConfigAvailable (line 5319) internally looks up the model config via userCanUseChatModelConfig. lookupEnabledChatModelConfigByID (line 5324) then looks up the same parsedModelConfigID again to feed validateChatModelOverrideEffort. The model config could be returned from the first validation step. Compare with putChatModelOverride which uses the combined validateChatModelOverride that looks up once.
(Netero, Pariston)
🤖
There was a problem hiding this comment.
Coder Agents generated.
Fixed. validateUserChatModelConfigAvailable now returns the validated model config, and putUserChatPersonalModelOverride reuses it for reasoning effort validation instead of doing a second lookup.
There was a problem hiding this comment.
Verified fixed in b4d5ac05. validateUserChatModelConfigAvailable now returns the model config, eliminating the second DB lookup.
🤖
| }) | ||
| return | ||
| } | ||
| if parsed.reasoningEffort != nil && req.ReasoningEffort == nil { |
There was a problem hiding this comment.
P2 [CRF-4] putChatModelOverride accepts reasoning effort embedded in model_config_id, creating an undocumented secondary input channel and GET/PUT asymmetry.
parseChatModelOverride parses the storage format uuid:effort from req.ModelConfigID. When the field contains "uuid:high" and reasoning_effort is null, the embedded effort is silently promoted (lines 5075-5077). But the GET response always separates them (line 5042 passes nil for effort to formatChatModelOverride). This creates two asymmetries:
- GET/PUT mismatch: GET returns
{"model_config_id": "uuid", "reasoning_effort": "high"}, but PUT also accepts{"model_config_id": "uuid:high"}without the explicit field. - Endpoint mismatch:
putUserChatPersonalModelOverrideparsesreq.ModelConfigIDas a plain UUID (line 5305:uuid.Parse(rawModelConfigID)). Sending{"model_config_id": "uuid:high"}to the deployment override PUT succeeds; the same payload to the personal override PUT fails.
Fix: parse req.ModelConfigID as a plain UUID in the PUT handler, taking effort only from req.ReasoningEffort. Remove lines 5075-5077. This keeps the internal storage parser separate from API input parsing.
(Hisoka P2, Knov P2, Kite Nit)
🤖
There was a problem hiding this comment.
Coder Agents generated.
Fixed. putChatModelOverride now parses model_config_id as a plain UUID and rejects encoded uuid:effort API input. The stored override parser remains in place for reading the internal persisted format.
There was a problem hiding this comment.
Verified fixed in b4d5ac05. PUT now rejects : in model_config_id and accepts reasoning_effort as a separate field. GET/PUT symmetry restored. Test PUTRejectsEncodedModelConfigID confirms.
🤖
| } | ||
| if !chatprovider.IsValidReasoningEffort(*effort) { | ||
| return http.StatusBadRequest, &codersdk.Response{ | ||
| Message: "Invalid reasoning_effort.", |
There was a problem hiding this comment.
P2 [CRF-5] Error message "Invalid reasoning_effort." drops the Detail field and uses inconsistent wording compared to the same check elsewhere in this file.
Three sibling checks in the same file (lines 1312, 3241, 3409) use:
Message: "Invalid reasoning_effort value.",
Detail: "Must be one of none, minimal, low, medium, high, xhigh, max.",This new check uses only Message: "Invalid reasoning_effort." with no Detail. An admin hitting this at the override settings endpoint gets no guidance about valid values.
Fix: match the existing pattern.
(Leorio P2, Chopper P3)
🤖
There was a problem hiding this comment.
Coder Agents generated.
Fixed. The invalid reasoning effort response now uses the same message as the sibling checks and includes the valid value detail.
There was a problem hiding this comment.
Verified fixed in b4d5ac05. Error messages now include valid values in Detail, consistent with sibling checks. Note: CRF-20 flags an edge case in this fix where SelectableReasoningEfforts returns nil for non-reasoning models.
🤖
| } | ||
| callConfig := codersdk.ChatModelCallConfig{} | ||
| if len(modelConfig.Options) > 0 { | ||
| if err := json.Unmarshal(modelConfig.Options, &callConfig); err != nil { |
There was a problem hiding this comment.
P3 [CRF-7] withResolvedReasoningEffort silently drops the admin-configured reasoning effort override when modelConfig.Options fails to unmarshal or re-marshal, with no log or error signal.
Both json.Unmarshal (line 427) and json.Marshal (line 442) errors return the original unmodified modelConfig. The function has no logger access and no doc comment explaining the fail-open contract. Its single caller (title_override.go:90) receives a modelConfig that looks valid but carries the model's base reasoning effort instead of the admin override.
The Options field is validated at write time, so corruption is unlikely. But the pattern sets a precedent where config deserialization failures silently degrade behavior. At minimum, the caller has logger access and could log when the returned config doesn't reflect the requested effort. A doc comment explaining the intentional fail-open behavior would prevent the next editor from treating the silent returns as bugs.
Additionally, this function has zero test coverage. Four exit paths (nil effort, unmarshal failure, nil resolved effort, success), none tested.
(Hisoka P3, Mafu-san P3, Meruem P3, Chopper P3, Knov P3)
🤖
There was a problem hiding this comment.
Coder Agents generated.
Not changed intentionally. The fail-open behavior is low impact because Options is validated on write, and preserving the original config on unexpected marshal/unmarshal failure avoids breaking title generation for a corrupt row. I did not add the suggested broader logging/test changes in this PR5 fix pass.
There was a problem hiding this comment.
Panel closed (4/6 accept). The fail-open design is appropriate for the title generation best-effort path. Options corruption requires bypassing write-side validation. Test coverage is tracked separately as CRF-18.
🤖
| } | ||
|
|
||
| func validateChatModelOverrideID( | ||
| func parseChatModelCallConfig(options json.RawMessage) (*codersdk.ChatModelCallConfig, error) { |
There was a problem hiding this comment.
P3 [CRF-8] parseChatModelCallConfig duplicates unmarshalChatModelCallConfig at line 7744 of this file.
Both take json.RawMessage, unmarshal into *codersdk.ChatModelCallConfig, and check for empty input. The new function returns (ptr, error) with a zero struct on empty; the existing one returns nil on empty/error and also checks isZeroChatModelCallConfig. A shared core function returning (*ChatModelCallConfig, error) could serve both callers, with unmarshalChatModelCallConfig wrapping it to apply the zero check and swallow errors.
(Robin)
🤖
There was a problem hiding this comment.
Coder Agents generated.
Not changed intentionally. The helpers differ semantically: this validation path must surface JSON errors to callers, while the existing unmarshal helper intentionally returns nil for empty, invalid, or zero configs. Sharing the core would add branching without making the call sites clearer.
There was a problem hiding this comment.
Panel closed (6/6 accept). The two functions have genuinely different contracts: validation (surfaces errors) vs display (swallows errors). Sharing would require error-handling branching less clear than two short functions.
🤖
| selectableEfforts := chatprovider.SelectableReasoningEfforts(callConfig.ReasoningEffort) | ||
| if !slices.Contains(selectableEfforts, *effort) { | ||
| return http.StatusBadRequest, &codersdk.Response{ | ||
| Message: "Invalid reasoning_effort: value is not selectable for the model config.", |
There was a problem hiding this comment.
P3 [CRF-9] The selectability validation error doesn't include the selectable values despite having the computed slice available on line 595.
The message explains the constraint ("value is not selectable for the model config") but doesn't name the selectable set. The admin knows their value is wrong but has to guess what the model config allows.
Fix:
return http.StatusBadRequest, &codersdk.Response{
Message: "Invalid reasoning_effort: value is not selectable for the model config.",
Detail: fmt.Sprintf("Must be one of %s.", strings.Join(selectableEfforts, ", ")),
}(Leorio P3, Chopper P3)
🤖
There was a problem hiding this comment.
Coder Agents generated.
Fixed. The selectability validation response now includes a Detail listing the selectable reasoning effort values for the model config.
There was a problem hiding this comment.
Verified fixed in b4d5ac05. Selectability error now includes the selectable values via strings.Join. Note: CRF-20 flags an edge case where selectableEfforts is nil.
🤖
| if chatReasoningEffort == nil { | ||
| chatReasoningEffort = personalOverrideEffort | ||
| } | ||
| if chatReasoningEffort != nil && !chatprovider.IsValidReasoningEffort(*chatReasoningEffort) { |
There was a problem hiding this comment.
Note [CRF-12] When postChats uses a personal override's reasoning effort (line 1308), it validates with IsValidReasoningEffort (is this a known value?) but not SelectableReasoningEfforts (is this value within the model's current max?).
Scenario: user stores personal override with effort "high" for a model whose max is "high". Admin later lowers the model's max to "medium". User creates a chat. "high" passes validity, the chat stores LastReasoningEffort = "high", but generation_preparer.go clamps it to "medium" at runtime via ResolveReasoningEffort.
The runtime behavior is correct (clamped), and the same gap exists for the direct req.ReasoningEffort path at this line, so this is consistent with the existing design. The stored metadata diverges from what was applied, but there's no user-facing impact. Worth knowing if metadata correctness becomes important.
(Hisoka)
🤖
There was a problem hiding this comment.
Coder Agents generated.
Not changed intentionally. Runtime clamping remains consistent with the direct per-turn reasoning effort path, so stored metadata can be above the current max but execution resolves to the allowed value.
There was a problem hiding this comment.
Acknowledged. Runtime clamping is correct and consistent with per-turn effort path.
🤖
| @@ -258,6 +275,17 @@ export const PersonalModelOverrideRow: FC<PersonalModelOverrideRowProps> = ({ | |||
| form.values.mode === "model" | |||
There was a problem hiding this comment.
Note [CRF-13] When an admin lowers a model's max reasoning effort after a user saved their personal override, the form holds the stale stored value while the UI slider shows the clamped value from pickReasoningEffort. Saving without touching the slider submits the original "high" value, which the backend rejects as not selectable.
The backend validation is correct, so this isn't a data integrity issue. The window is narrow (requires admin config change after user override). The user would see a save failure and can adjust.
(Knov)
🤖
There was a problem hiding this comment.
Coder Agents generated.
Not changed intentionally. The stale UI case fails closed: if an admin lowers max effort after a saved override, the backend rejects the stale value on save and the user can adjust it. No data integrity issue is introduced.
There was a problem hiding this comment.
Acknowledged. Fails closed in a narrow window. No data integrity issue.
🤖
b56cf6e to
7b9043d
Compare
|
Coder Agents generated. Addressed the remaining top-level findings in the amended PR5 commit:
|
76f68fe to
ead7707
Compare
7b9043d to
e9d0d29
Compare
ead7707 to
0b08c0a
Compare
e9d0d29 to
508e5cc
Compare
0b08c0a to
6455ba5
Compare
508e5cc to
9eb79c5
Compare
6455ba5 to
3c16e1e
Compare
9eb79c5 to
b4d5ac0
Compare
|
/coder-agents-review |
There was a problem hiding this comment.
Good progress on R1 findings: 8 of 13 addressed cleanly in one commit. The CRF-4 fix (separating model_config_id from reasoning_effort in the PUT API) is particularly well done, including a test for the rejection case. CRF-5's fix consolidated the error Detail pattern into a shared helper. CRF-6's parser test additions are thorough.
Contested findings: panel closed CRF-2 (5/7 accept duplication defense), CRF-7 (4/6 accept fail-open design for title gen), CRF-8 (6/6 accept semantic differences). Test coverage for withResolvedReasoningEffort (raised separately as CRF-18) is tracked independently from the design concern.
One P0, three P3s new this round.
"The PR description lists
TestCreateChildSubagentChat_StoresReasoningEffortOverrideamong the validation commands. The test assertssql.NullStringagainstdatabase.NullChatReasoningEffort. These are different types." The test never passed.
Severity summary: 1 P0 (deterministic test failure), 3 P3 (test gaps, fix-chain regression).
🤖 This review was automatically generated with Coder Agents.
b4d5ac0 to
5f12b74
Compare
|
/coder-agents-review |
There was a problem hiding this comment.
All findings resolved across three rounds. R2 fixes (CRF-17 P0 test type mismatch, CRF-18/19 P3 test gaps, CRF-20 P3 empty error detail) verified fixed and passing. No new findings from Netero or the R3 panel (Bisky, Mafuuu, Pariston, Meruem).
The reasoning effort feature threads cleanly through deployment overrides, personal overrides, subagent child chats, and title generation. Write-time validation gates all override paths via validateChatModelOverrideEffort. Runtime clamping via ResolveReasoningEffort handles stale stored values when admin config changes. Test density rose from 34% (R1) to 44% (R3) with coverage across all new validation paths, parsers, and the withResolvedReasoningEffort helper.
Finding ledger: 21 total (2 P2, 8 P3, 2 P4, 3 Note, 3 Nit, 1 P0). 12 author-fixed, 3 panel-closed, 2 acknowledged, 4 dropped.
"These are real stones, not costume jewelry."
🤖 This review was automatically generated with Coder Agents.
Documentation CheckUpdates Needed
Automated review via Coder Agents |
3c16e1e to
1277de5
Compare
5f12b74 to
a4b850a
Compare
034b553
into
dm/reasoning-effort-2-per-turn
Summary
Validation
Generated by Coder Agents.