Conversation
|
Claude Security Review: no high-confidence findings. (run) |
There was a problem hiding this comment.
AgentCore Harness Review
Verdict: Looks good
Presentation-only change: adds group metadata to the flags on eval ondemand evaluate and eval ondemand simulate so --help groups them, and trims per-flag source: / filter: prefixes now that the group headings carry them. Verified:
flag()insrc/router/handler.tsxsupports{ group }andflags.tsxwires it tooption.helpGroup(...), so the change is a proper use of the existing help-group mechanism (same pattern already used bybatch-evaluation/{simulate,evaluate}/index.tsx).- No flag names, schemas, defaults, or handler behavior change — the reordering of
--evaluatorsto sit next to--ground-truthunder theEvaluation:heading is a declaration-order move only. - Tests in
src/handlers/eval/ondemand/*.test.tsxdon't assert on help output, so no snapshots need updating. - No telemetry surface is added; nothing to instrument.
Minor nit (non-blocking, not requesting changes): the group labels here (Target:, Invocation:, Session source:, Session filters:) don't perfectly match the labels in batch-evaluation/simulate (Runtime invocation:, Configuration:) or sessionSource.tsx (Session source (choose exactly one):, Source filters:). Aligning them across eval commands would be nice but is a follow-up.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## refactor #2300 +/- ##
=========================================
Coverage 96.98% 96.98%
=========================================
Files 579 579
Lines 39552 39574 +22
=========================================
+ Hits 38358 38380 +22
Misses 1194 1194 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Applies the router's help-group mechanism (#2262) to `eval ondemand evaluate` (9 options) and `eval ondemand simulate` (10 options), whose flat option lists had grown past skimming. Presentation only — no flag is added, removed, or renamed, and no request changes. - ondemand evaluate: Session source, Session filters, Evaluation. `--evaluators` moves next to `--ground-truth` so the Evaluation heading's flags declare contiguously. Descriptions lose their "source:"/"time filter:"/"filter:" prefixes, which the headings now carry. - ondemand simulate: Target, Invocation, Dataset, Evaluation. `--dataset` loses its "dataset source:" prefix for the same reason. The two API-shaped flags gain a "Parameter details" block, matching batch-evaluation: `--payload-template` reuses batch simulate's wording (same shape), and `--ground-truth` documents EvaluationReferenceInput[] — the raw per-reference shape on-demand passes straight to Evaluate, which differs from batch evaluate's session-keyed wrapper, so it gets its own block rather than a copy. A command that groups its flags gets its generated -h/--help moved to "Other options:".
9ab9346 to
aea569e
Compare
|
Claude Security Review: no high-confidence findings. (run) |
Applies the router's help-group mechanism from #2262 to the two
eval ondemandcommands whose flat option lists had grown past skimming. Presentation only — no flag is added, removed, or renamed, and no request changes.eval ondemand evaluateeval ondemand simulateNotes
ondemand evaluate:--evaluatorsis moved next to--ground-truthso theEvaluation:heading's flags declare contiguously (Commander orders headings by first-declared flag). Descriptions lose theirsource:/time filter:/filter:prefixes, which the headings now carry.ondemand simulate:--datasetloses itsdataset source:prefix for the same reason.--payload-templatereuses batch simulate's wording (identical shape);--ground-truthdocumentsEvaluationReferenceInput[]— the raw per-reference shape on-demand passes straight to Evaluate, which differs from batch evaluate's session-keyed wrapper, so it gets its own block rather than a copy.-h/--helpmoved toOther options:.Proposed
--helpagentcore eval ondemand evaluate --helpagentcore eval ondemand simulate --helpTesting
bun run typecheckcleanbun run lintcleanbun test src/handlers/eval/ondemand— 25 pass