Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(eval): grouped help for the ondemand commands - #2300

Open
jariy17 wants to merge 1 commit into
refactorfrom
feat/eval-ondemand-grouped-help
Open

jariy17 wants to merge 1 commit into
refactorfrom
feat/eval-ondemand-grouped-help

Conversation

@jariy17

@jariy17 jariy17 commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Applies the router's help-group mechanism from #2262 to the two eval ondemand commands whose flat option lists had grown past skimming. Presentation only — no flag is added, removed, or renamed, and no request changes.

Command Options Headings
eval ondemand evaluate 9 Session source · Session filters · Evaluation
eval ondemand simulate 10 Target · Invocation · Dataset · Evaluation

Notes

  • ondemand evaluate: --evaluators is moved next to --ground-truth so the Evaluation: heading's flags declare contiguously (Commander orders headings by first-declared flag). Descriptions lose their source: / time filter: / filter: prefixes, which the headings now carry.
  • ondemand simulate: --dataset loses its dataset source: prefix for the same reason.
  • Parameter details added for the two API-shaped flags, matching batch-evaluation: --payload-template reuses batch simulate's wording (identical shape); --ground-truth documents EvaluationReferenceInput[] — the raw per-reference shape on-demand passes straight to Evaluate, which differs from batch evaluate's session-keyed wrapper, so it gets its own block rather than a copy.
  • A command that groups its flags gets its generated -h/--help moved to Other options:.

Proposed --help

agentcore eval ondemand evaluate --help
Usage: agentcore eval ondemand evaluate [options]

evaluate existing sessions client-side (synchronous; prints scores)

Session source:
  --agent <agent>                  harness ID or Runtime ID whose sessions to
                                   evaluate
  --endpoint <endpoint>            Runtime endpoint qualifier (default DEFAULT)

Session filters:
  --lookback-days <lookback-days>  evaluate sessions from the last N days
  --start-time <start-time>        window start (ISO-8601, with --end-time)
  --end-time <end-time>            window end (ISO-8601, with --start-time)
  --session-ids <session-ids...>   specific session IDs
  --trace-id <trace-id>            a single trace ID (session ID is read off the
                                   span)

Evaluation:
  --evaluators <evaluators...>     evaluator ID(s) to apply
  --ground-truth <ground-truth>    expected answers (JSON
                                   EvaluationReferenceInput[])

Other options:
  -h, --help                       display help for command

Global Options:
  --region <region>                AWS region
  --debug                          debug logging (default: false)
  --json                           JSON output (default: false)
  --endpoint-url <endpoint-url>    endpoint URL override

Parameter details:

  --ground-truth (JSON: list of objects)
      Expected answers for the sessions being evaluated, so an evaluator can score a
      response against a reference instead of judging it on its own. Each entry is an
      EvaluationReferenceInput bound to a session (and optionally a trace) by its span
      context; omit an entry for a session that has no reference answer.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        [
          {
            "context": {                        // [required] what this reference applies to
              "spanContext": {
                "sessionId": "string",          // [required]
                "traceId": "string"             // set for a trace-level reference
              }
            },
            "expectedResponse": { "text": "string" },   // trace-level; needs a traceId
            "assertions": [                              // session-level
              { "text": "string" },
              ...
            ],
            "expectedTrajectory": {                      // session-level
              "toolNames": ["string", ...]    // tools the agent should have called
            }
          },
          ...
        ]

      Example:
        --ground-truth '[{"context":{"spanContext":{"sessionId":"session-123"}},"assertions":[{"text":"acknowledges the shipping delay"}]}]'

        --ground-truth file://ground-truth.json

agentcore eval ondemand simulate --help
Usage: agentcore eval ondemand simulate [options]

replay a dataset against a Runtime, then evaluate the sessions client-side

Target:
  --runtime-id <runtime-id>                Runtime ID to invoke per scenario
  --endpoint <endpoint>                    Runtime endpoint qualifier (default DEFAULT)

Invocation:
  --payload-template <payload-template>    request body per example (JSON object); {input} is replaced with the input
  --header <header...>                     an ordered application header (repeatable)
  --bearer-token <bearer-token>            CUSTOM_JWT bearer token (for JWT-auth Runtimes)
  --user-id <user-id>                      Runtime user ID

Dataset:
  --dataset <dataset>                      local JSONL path or a dataset ID
  --dataset-version <dataset-version>      dataset version (with a dataset ID)

Evaluation:
  --evaluators <evaluators...>             evaluator ID(s) to apply
  --ingestion-wait-ms <ingestion-wait-ms>  ms to wait for span ingestion before grading (default 180000; 0 to skip)

Other options:
  -h, --help                               display help for command

Global Options:
  --region <region>                        AWS region
  --debug                                  debug logging (default: false)
  --json                                   JSON output (default: false)
  --endpoint-url <endpoint-url>            endpoint URL override

Parameter details:

  --payload-template (JSON object)
      The request body sent to the Runtime for each dataset example. Every occurrence
      of {input} is replaced with that example's input, so the template describes the
      shape your agent expects and {input} marks where the prompt goes.

      Example:
        --payload-template '{"prompt":"{input}"}'

        --payload-template '{"messages":[{"role":"user","content":"{input}"}],"stream":false}'

Testing

  • bun run typecheck clean
  • bun run lint clean
  • bun test src/handlers/eval/ondemand — 25 pass

@github-actions github-actions Bot added the size/s PR size: S label Sep 14, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added agentcore-harness-reviewing AgentCore Harness review in progress claude-security-reviewing Claude Code /security-review in progress labels Sep 14, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed claude-security-reviewing Claude Code /security-review in progress agentcore-harness-reviewing AgentCore Harness review in progress labels Sep 14, 2026

@agentcore-devx-automation agentcore-devx-automation Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AgentCore Harness Review

Verdict: Looks good

Presentation-only change: adds group metadata to the flags on eval ondemand evaluate and eval ondemand simulate so --help groups them, and trims per-flag source: / filter: prefixes now that the group headings carry them. Verified:

  • flag() in src/router/handler.tsx supports { group } and flags.tsx wires it to option.helpGroup(...), so the change is a proper use of the existing help-group mechanism (same pattern already used by batch-evaluation/{simulate,evaluate}/index.tsx).
  • No flag names, schemas, defaults, or handler behavior change — the reordering of --evaluators to sit next to --ground-truth under the Evaluation: heading is a declaration-order move only.
  • Tests in src/handlers/eval/ondemand/*.test.tsx don't assert on help output, so no snapshots need updating.
  • No telemetry surface is added; nothing to instrument.

Minor nit (non-blocking, not requesting changes): the group labels here (Target:, Invocation:, Session source:, Session filters:) don't perfectly match the labels in batch-evaluation/simulate (Runtime invocation:, Configuration:) or sessionSource.tsx (Session source (choose exactly one):, Source filters:). Aligning them across eval commands would be nice but is a follow-up.

@codecov-commenter

codecov-commenter commented Sep 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.98%. Comparing base (e7a7aae) to head (aea569e).

Additional details and impacted files
@@            Coverage Diff            @@
##           refactor    #2300   +/-   ##
=========================================
  Coverage     96.98%   96.98%           
=========================================
  Files           579      579           
  Lines         39552    39574   +22     
=========================================
+ Hits          38358    38380   +22     
  Misses         1194     1194           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Applies the router's help-group mechanism (#2262) to `eval ondemand evaluate`
(9 options) and `eval ondemand simulate` (10 options), whose flat option lists
had grown past skimming. Presentation only — no flag is added, removed, or
renamed, and no request changes.

- ondemand evaluate: Session source, Session filters, Evaluation. `--evaluators`
  moves next to `--ground-truth` so the Evaluation heading's flags declare
  contiguously. Descriptions lose their "source:"/"time filter:"/"filter:"
  prefixes, which the headings now carry.
- ondemand simulate: Target, Invocation, Dataset, Evaluation. `--dataset` loses
  its "dataset source:" prefix for the same reason.

The two API-shaped flags gain a "Parameter details" block, matching
batch-evaluation: `--payload-template` reuses batch simulate's wording (same
shape), and `--ground-truth` documents EvaluationReferenceInput[] — the raw
per-reference shape on-demand passes straight to Evaluate, which differs from
batch evaluate's session-keyed wrapper, so it gets its own block rather than a
copy.

A command that groups its flags gets its generated -h/--help moved to
"Other options:".
@jariy17
jariy17 force-pushed the feat/eval-ondemand-grouped-help branch from 9ab9346 to aea569e Compare September 14, 2026 22:12
@github-actions github-actions Bot added size/m PR size: M and removed size/s PR size: S size/m PR size: M labels Sep 14, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 14, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 14, 2026
@jariy17
jariy17 marked this pull request as ready for review September 14, 2026 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/m PR size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants