Review: behavior-E2E harness (#1431)

Summary — Solid, well-targeted addition. The side-effect-not-reply premise, planted unguessable tokens, N× repetition, and false_success-as-hard-fail directly attack the #1428 bug class ("agent claims success, tool never ran"), and the classification logic is cleanly split from the live BehaviorHarness so 24 deterministic unit tests cover it without a server. All four REST contracts the harness calls were verified against the real UI server and match (/api/sessions→id, /api/chat/send→content, /api/agents→agents[].id, /api/health). No security concerns, no blocking issues in the runtime path. The one thing worth doing before merge: stop reimplementing the builder's agent-id normalization and import the real function — a verification harness that derives its expected path with a copy of the logic it's verifying can silently drift into green. The two CI-workflow bugs below are latent (the workflow is gated off until #1297) but will break the step the day it activates.

Issues

🟡 Side-effect check duplicates the builder's slug logic instead of reusing it (src/gaia/eval/behavior_harness.py:246-249, again at 418-422)
The real builder already exposes _normalize_agent_id (src/gaia/agents/builder/agent.py:355), and crucially runs _split_camel_case(name.strip()) before it (agent.py:392-395). The harness reimplements only the normalize half, inline, in two places — so the docstring claim "Derive the agent id the same way the builder does" is already not quite true, and if the builder's rules change the check will look in the wrong directory and emit wrong verdicts. That's the exact false-signal failure mode this harness exists to prevent. For the current test-<hex> tokens it happens to agree, so it's a latent risk, not a live bug — but reuse removes both the risk and the duplication. The unit-test docstring (test_behavior_harness.py:793) already refers to _normalize_agent_id() as the canonical function; make it real:

from gaia.agents.builder.agent import _normalize_agent_id, _split_camel_case
...
agent_id = _normalize_agent_id(_split_camel_case(planted_name.strip()))

Apply in both _builder_side_effect_check and the registry-recheck block in run_scenario.

🟡 gaia init --check is not a valid flag (.github/workflows/test_agent_behavior_e2e.yml:30)
init has no --check; the closest is --skip-lemonade ("Skip Lemonade installation check"). As written, argparse exits non-zero and the step fails. The workflow is gated off until #1297, so this isn't breaking CI today — but it'll break the step the moment it activates. Likely intended gaia init --skip-lemonade (or a Lemonade health probe).

🟡 Workflow venv is created but never activated (.github/workflows/test_agent_behavior_e2e.yml:25-35)
uv venv && uv pip install -e ".[dev]" runs in one step; the later gaia init … and python -m pytest … steps run in fresh shells with no source .venv/bin/activate (or VIRTUAL_ENV/uv run), so gaia and the installed package won't be on PATH. Same caveat: deferred, but won't run as-is. Prefer uv run python -m pytest … and uv run gaia …, or set VIRTUAL_ENV once.

🟢 rstrip("/api/v1") mangles non-default URLs (tests/integration/eval/test_behavior_e2e.py:521)
str.rstrip treats the argument as a char set, not a suffix — it works for the default URL by luck but corrupts any base URL whose host/port ends in a/p/i/v/1/. Use:

    health_url = base_url.removesuffix("/api/v1").rstrip("/") + "/api/v1/health"

🟢 except Exception: return False is a silent fallback (src/gaia/eval/behavior_harness.py:377)
GAIA's "fail loudly" rule (CLAUDE.md) applies repo-wide, eval infra included. Here a server error is swallowed and read as "agent not listed", quietly downgrading a run to honest_failure and masking the very kind of infra problem the harness should surface. At minimum log it with context — mirror the exc_info=True pattern you already use in run_scenario (behavior_harness.py:432).

🟢 Dead field Scenario._planted_names (src/gaia/eval/behavior_harness.py:217)
Declared but never read or written — the harness re-extracts the planted name from the prompt via regex (behavior_harness.py:406) instead. Drop it, or have prompt_factory return (prompt, planted_name) so the regex round-trip (and its quoting coupling at :406/:801) goes away entirely.

Strengths

• The core idea is right and the design supports it. Verdict taxonomy (true/false_success, honest_failure, error) with false_success dominating aggregation is the correct shape, and the aggregation rules are exhaustively unit-tested (TestAggregateVerdicts).
• Testability-first split: pure helpers (_success_markers, _aggregate_verdicts, side_effect_check) vs. the live BehaviorHarness, so 24 tests run with no model/network — and the real_model marker is registered in pyproject.toml with an auto-skip fixture, so normal CI stays green.
• Docs landed with the code (docs/sdk/testing.mdx) and clearly explain the false-success rationale and how new agents plug in a Scenario.

Verdict

Approve with suggestions. Nothing blocks the runtime path. Please reuse _normalize_agent_id/_split_camel_case before merge (cheap, and it closes the harness-drift gap), and fix the two workflow bugs so the step is correct when #1297 turns it on.

All items addressed in the rebase+fix commit (05e74296):

🟡 _normalize_agent_id / _split_camel_case reuse — Both slug-derivation blocks in behavior_harness.py (_builder_side_effect_check and the registry-recheck in run_scenario) now import and call the canonical builder functions instead of re-implementing them inline.

🟡 gaia init --check — Changed to uv run gaia init --skip-lemonade (the valid flag for skipping the Lemonade install check).

🟡 Workflow venv not activated — All run: steps that need the package now use uv run prefix so no explicit source .venv/bin/activate is required.

🟢 rstrip("/api/v1") mangling — Changed to base_url.removesuffix("/api/v1").rstrip("/") in the integration test fixture.

🟢 except Exception: return False — Now logs exc_info=True via logger.warning before returning False, matching the pattern already used in run_scenario.

🟢 Dead Scenario._planted_names field — Removed from the dataclass and its docstring entry.

Also removed three unused imports (dataclasses.field, tempfile, pathlib.Path) that became dead after the above changes. 24/24 unit tests still pass.

🟡 src/gaia/eval/behavior_harness.py:276 — silent fallback violates CLAUDE.md

_agent_listed catches Exception broadly and returns False, which the harness treats as "agent not in registry → downgrade true_success → honest_failure". This masks API failures silently and is prohibited by the No-Silent-Fallbacks rule.

        except requests.RequestException as exc:
            logger.warning(
                "_agent_listed: /api/agents request failed: %s",
                exc,
                exc_info=True,
            )
            raise

Or, since this is an optional extra check after the file-exists check already confirmed success, you could simply let the exception propagate to the outer except Exception as exc in run_scenario (which logs and records Verdict.error), keeping the harness honest about infrastructure failures vs. agent failures.

Removed the try/except from _agent_listed entirely — exceptions from _get now propagate to the outer run_scenario handler, which logs and records Verdict.error. No silent downgrade.

🟡 .github/workflows/test_agent_behavior_e2e.yml:41 — artifact upload path will always be empty on failure.

The workflow uploads path: /tmp/gaia-behavior-e2e-*/, but the harness writes transcripts to pytest's tmp_path / "artifacts" — a path like /tmp/pytest-0/test_builder_creates_agent_file0/artifacts/. The glob never matches, so the behavior-e2e-artifacts archive will be uploaded empty every time the test fails, making CI failures undebuggable.

          path: /tmp/pytest-*/test_builder_creates_agent_file*/**/artifacts/

Or, more robustly, pass a fixed artifact_dir that matches the upload glob:

# in test_builder_creates_agent_file
import tempfile, pathlib
artifact_dir = pathlib.Path(tempfile.mkdtemp(prefix="gaia-behavior-e2e-")) / "artifacts"

then the original glob /tmp/gaia-behavior-e2e-*/ works as intended.

Fixed — updated the upload glob to /tmp/pytest-*/test_builder_creates_agent_file*/**/artifacts/ which matches pytest's actual tmp_path layout.