feat: per-class ModelPinMetadata + cost-by-prompt-purpose dashboard + prompt-drift CI#2511
Conversation
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI (base), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (15)
📜 Recent review details⏰ Context from checks skipped due to timeout. (32)
|
| Check name | Status | Explanation |
|---|---|---|
| Title check | ✅ Passed | The title clearly summarizes the main changes: per-class metadata, prompt-purpose analytics, and CI drift checks. |
| Description check | ✅ Passed | The description is directly related and accurately outlines the metadata rollout, dashboard, and CI drift work. |
| Linked Issues check | ✅ Passed | The summarized changes cover the linked objectives for metadata rollout, prompt-purpose analytics, and prompt-drift CI. |
| Out of Scope Changes check | ✅ Passed | The added docs, tests, and hardening changes all support the listed objectives; no clear unrelated changes stand out. |
| Docstring Coverage | ✅ Passed | Docstring coverage is 85.94% which is sufficient. The required threshold is 40.00%. |
Comment @coderabbitai help to get the list of available commands.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request completes epic #2490 by standardizing ModelPinMetadata across all prompt classes and populating prompt_class_id throughout the cost-recording infrastructure. These changes facilitate a new cost and latency breakdown dashboard by prompt purpose and establish a robust prompt-drift regression gate in CI to ensure model pin consistency. Highlights
Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2511 +/- ##
==========================================
+ Coverage 88.84% 88.86% +0.01%
==========================================
Files 3146 3148 +2
Lines 159649 160041 +392
==========================================
+ Hits 141847 142227 +380
- Misses 17802 17814 +12 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive prompt-class metadata system and a corresponding budget breakdown dashboard. Key changes include the implementation of a declarative model pin registry (model_pins.py), the requirement for all LLM-invoking prompt classes to expose a metadata property returning ModelPinMetadata, and the addition of two new CI/pre-push gates (check_prompt_class_metadata.py and check_pin_golden_fresh.py) to enforce these conventions and prevent prompt-drift regressions. On the API and persistence layers, cost record querying and analytics have been updated to support filtering by prompt_class_id, backed by new database indexes. Finally, the frontend has been enhanced with a new "Cost by prompt purpose" section on the budget page, displaying detailed cost, latency, and reliability metrics per prompt class. No review comments were provided, so I have no additional feedback to offer.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
There was a problem hiding this comment.
Actionable comments posted: 12
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
src/synthorg/budget/tracker_protocol.py (1)
83-89: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winUpdate the stale pagination wording.
collect_all_records()no longer drains successive pages; it delegates to the new one-snapshotcollect_records()contract. Please update this protocol docstring so implementers do not preserve the old TOCTOU-prone page-walk semantics.Suggested wording
- cursor walk is repeatable. Callers needing every matching record - drain successive pages via :func:`collect_all_records`. Unbounded - materialisation over a long-lived cost log is a memory hazard. + cursor walk is repeatable. Callers needing every matching record + should use :meth:`collect_records` or :func:`collect_all_records` + to read one atomic snapshot.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/synthorg/budget/tracker_protocol.py` around lines 83 - 89, The pagination docstring in tracker_protocol’s record-batching contract is stale and still describes page-walk behavior via collect_all_records; update the wording to reflect that collect_all_records now delegates to the one-snapshot collect_records contract. Make sure the text around the bounded page return behavior and any mention of successive pages or repeatable cursor walks no longer encourages TOCTOU-prone semantics, and align the description with the current protocol intent for the relevant method/documentation block.src/synthorg/budget/call_analytics.py (1)
99-114: 🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy liftPass
prompt_class_idinto the orchestration-ratio query as well.When
prompt_class_idis set, Line 104 filters the record set, but Lines 108-114 still computeorchestration_ratioover the broader window. That makes a filtered aggregation internally inconsistent: counts/costs are prompt-class-specific whileorchestration_ratiois not. Thread the same filter throughget_orchestration_ratio(...), or omit that field for prompt-class-filtered requests.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/synthorg/budget/call_analytics.py` around lines 99 - 114, The aggregation in CallAnalytics is inconsistent because `collect_all_records(...)` is filtered by `prompt_class_id` but `self._tracker.get_orchestration_ratio(...)` is not. Update the `CallAnalytics` flow to pass the same `prompt_class_id` filter into `get_orchestration_ratio(...)` (or skip the ratio when `prompt_class_id` is set), so the record metrics and orchestration ratio are computed over the same scope.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/check_prompt_class_metadata.py`:
- Around line 99-119: _update _resolve_project_root in
check_prompt_class_metadata.py to fail closed when the provided --repo-root is
not the actual repository root. After resolving the path, verify it contains the
expected scan root used by _scan_all (for example src/synthorg); if that check
fails, raise ProjectRootError so the CLI exits with code 2 instead of silently
treating an empty directory as success. Keep the existing resolve/dir
validation, but add the repository-structure validation in _resolve_project_root
so misconfigured callers cannot bypass the enforcement gate._
In `@src/synthorg/api/controllers/meta_analytics.py`:
- Around line 92-95: The collector-disabled guard in the meta analytics
controller is logging a warning on a supported configuration path, which creates
noisy request-level logs. Update the `_collector is None` branch in
`meta_analytics.py` to stop using
`logger.warning(XDEPLOY_COLLECTOR_UNAVAILABLE)`; either raise
`ServiceUnavailableError` directly or downgrade the log to debug and keep any
disabled-state signal at startup in `configure_analytics_controller()`.
In `@src/synthorg/budget/call_analytics_models.py`:
- Around line 160-184: The _zero_calls_have_zero_rates validator in
CallAnalyticsModel only blocks non-zero cost and rate fields, but zero-call rows
can still slip through with impossible token or latency values. Update the
model_validator logic to also reject any non-zero token/count/latency aggregates
on call_count == 0 rows, using the same prompt_class_id error context. Keep the
check centralized in _zero_calls_have_zero_rates so all internally inconsistent
zero-call rows fail validation before the API returns them.
In `@src/synthorg/hr/evaluation/pin_golden_compute.py`:
- Around line 35-39: The shared golden computation in pin_golden_compute.py is
missing the same prompt-class invariant enforced by
ModelPinValidationBenchmark.grade(), so it can silently fingerprint cases whose
metadata prompt class does not match case.id. Update the helper that iterates
benchmark.load_test_cases() to validate each case by comparing case.id against
pin.prompt_class_id derived from pin_from_case_metadata(case.metadata), and fail
fast before calling runner.run_case() or writing into golden when they differ.
Keep the existing fingerprint_for and sorted return behavior unchanged, but make
sure the mismatch surfaces as an error instead of producing a wrong golden.
In `@src/synthorg/llm/model_pins.py`:
- Around line 104-152: Duplicate PromptPurposeId entries in _PIN_SPECS can still
overwrite silently because the dict literal collapses duplicates before the
missing-spec check runs. Refactor model_pins.py to follow the
model_tier_policy.py pattern: store the pin definitions in a sequence first,
validate that no PromptPurposeId appears more than once, then construct
_PIN_SPECS from that validated data. Keep the import-time guard near _PIN_SPECS
and the _missing_pin_specs validation so both duplicates and omissions fail
fast.
In `@src/synthorg/providers/cost_recording.py`:
- Around line 193-197: Validate and coerce `purpose` to a real `PromptPurposeId`
before emitting `PROVIDER_PROMPT_PURPOSE_INVOKED` in `cost_recording` so the
trackerless path shares the same contract as `CostRecordingContext`. Move the
`purpose` normalization/validation ahead of the `logger.debug(...)` call, and
use the validated value for both the trackerless event and the tracked path to
prevent raw strings from reaching eval/analytics logs.
In `@src/synthorg/providers/management/service.py`:
- Around line 313-319: `ProviderManagementService` is already well over the
service-module LOC budget, so this prompt-metadata logic should not stay in the
class. Move the `_PURPOSE_ID`/`metadata` behavior out of
`ProviderManagementService` into an existing mixin or a small dedicated
helper/service, and keep the class focused on provider management. Make sure the
new location still exposes the same `ModelPinMetadata` behavior used by
`pin_for` and `PromptPurposeId.PROVIDERS_TEST_CONNECTION`.
In `@tests/unit/budget/test_call_analytics.py`:
- Around line 324-327: The test for prompt_class_id filtering currently passes
even if the filter is ignored because the fixture only includes a matching
record. Update test_aggregation_accepts_prompt_class_filter in
test_call_analytics.py to prove the filter is applied by adding at least one
non-matching record to the _make_service fixture or by asserting the tracker
call kwargs from get_aggregation; keep the existing matching record and verify
only the matching data contributes to agg.total_calls.
In `@tests/unit/scripts/test_check_pin_golden_fresh.py`:
- Line 46: The typed cast in the test helper is using a bare class name that
triggers Ruff TC006; update the `cast()` call in `test_check_pin_golden_fresh`
to use the quoted forward reference form for `_ScriptModule`. Keep the change
localized to the `cast` usage around the module return so the type annotation
remains correct while satisfying the linter.
In `@web/src/mocks/handlers/budget.ts`:
- Around line 11-19: `getPromptClassBreakdown` is only used in a type position,
so the MSW handler import should not bring it in as a runtime value. Update the
imports in the budget handler module by moving `getPromptClassBreakdown` to a
separate `import type` from `@/api/endpoints/budget`, keeping the value imports
limited to the functions actually called at runtime. Verify the handler
definitions that reference `typeof getPromptClassBreakdown` still type-check
after the split.
In `@web/src/pages/BudgetPage.tsx`:
- Around line 259-262: PromptClassSection is currently rendered without the
active budget window, so it keeps fetching the unbounded prompt-class breakdown
while the rest of the dashboard is period-scoped. Update BudgetPage to pass the
selected period start/end into PromptClassSection, then have PromptClassSection
include those values in its request to /budget/prompt-class-breakdown. Make sure
the prop wiring and fetch/query logic use the existing PromptClassSection symbol
so the table stays aligned with the currently selected period.
In `@web/src/utils/budget.ts`:
- Around line 78-88: Keep the reserved “Other” color out of the normal rotating
palette in budget utilities. In `computeCostBreakdown()`, the modulo-based slice
coloring should only use the regular category colors, so remove
`DONUT_COLOR_OTHER` from `DONUT_COLORS` and reserve it only for the aggregated
overflow bucket. Verify the palette remains aligned with the intended category
colors and that the “Other” slice is assigned explicitly where it is created.
---
Outside diff comments:
In `@src/synthorg/budget/call_analytics.py`:
- Around line 99-114: The aggregation in CallAnalytics is inconsistent because
`collect_all_records(...)` is filtered by `prompt_class_id` but
`self._tracker.get_orchestration_ratio(...)` is not. Update the `CallAnalytics`
flow to pass the same `prompt_class_id` filter into
`get_orchestration_ratio(...)` (or skip the ratio when `prompt_class_id` is
set), so the record metrics and orchestration ratio are computed over the same
scope.
In `@src/synthorg/budget/tracker_protocol.py`:
- Around line 83-89: The pagination docstring in tracker_protocol’s
record-batching contract is stale and still describes page-walk behavior via
collect_all_records; update the wording to reflect that collect_all_records now
delegates to the one-snapshot collect_records contract. Make sure the text
around the bounded page return behavior and any mention of successive pages or
repeatable cursor walks no longer encourages TOCTOU-prone semantics, and align
the description with the current protocol intent for the relevant
method/documentation block.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 2c04b8da-5177-4925-afa4-7694190b37c9
📒 Files selected for processing (119)
.github/workflows/ci.yml.opencode/agents/python-reviewer.md.opencode/commands/babysit-pr.md.opencode/plugins/synthorg-hooks.tsCLAUDE.mddata/architecture_report.jsondata/feature_index.jsondata/runtime_stats.yamldocs/design/evaluation-loop.mddocs/design/page-structure.mddocs/guides/monitoring.mddocs/reference/convention-gates.mddocs/reference/model-tier-policy.mdevals/models/scorecard.pyevals/run.pyevals/runner/execution.pyscripts/check_pin_golden_fresh.pyscripts/check_prompt_class_metadata.pyscripts/convention_gate_map.yamlscripts/cost_scope_purpose_baseline.txtscripts/refresh_model_pin_golden.pyscripts/run_prepush_python_gates.pysrc/synthorg/api/controllers/budget.pysrc/synthorg/api/controllers/meta_analytics.pysrc/synthorg/api/rate_limits/policies.pysrc/synthorg/api/services/analytics_read_service.pysrc/synthorg/budget/_tracker_helpers.pysrc/synthorg/budget/call_analytics.pysrc/synthorg/budget/call_analytics_models.pysrc/synthorg/budget/tracker.pysrc/synthorg/budget/tracker_protocol.pysrc/synthorg/client/generators/llm.pysrc/synthorg/engine/classification/semantic_detectors.pysrc/synthorg/engine/evolution/proposers/separate_analyzer.pysrc/synthorg/engine/intake/strategies/agent_intake.pysrc/synthorg/engine/intervention/proposer.pysrc/synthorg/engine/quality/decomposers/llm.pysrc/synthorg/engine/quality/graders/llm.pysrc/synthorg/engine/workspace/semantic_llm.pysrc/synthorg/hr/evaluation/pin_golden.jsonsrc/synthorg/hr/evaluation/pin_golden_compute.pysrc/synthorg/hr/evaluation/pin_probe.pysrc/synthorg/hr/evaluation/pin_validation_benchmark.pysrc/synthorg/hr/performance/llm_calibration_sampler.pysrc/synthorg/hr/performance/llm_judge_quality_strategy.pysrc/synthorg/hr/training/curation/llm_curated.pysrc/synthorg/knowledge/synthesis/llm_synthesizer.pysrc/synthorg/llm/__init__.pysrc/synthorg/llm/model_pins.pysrc/synthorg/llm/model_tier_policy.pysrc/synthorg/llm/prompt_purpose.pysrc/synthorg/memory/consolidation/abstractive.pysrc/synthorg/memory/consolidation/compressor.pysrc/synthorg/memory/consolidation/llm_op.pysrc/synthorg/memory/embedding/fine_tune_query.pysrc/synthorg/memory/procedural/proposer.pysrc/synthorg/memory/procedural/success_proposer.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pysrc/synthorg/memory/retrieval/reranking/llm_reranker.pysrc/synthorg/meta/charter/strategy.pysrc/synthorg/meta/chief_of_staff/chat.pysrc/synthorg/meta/chief_of_staff/narrative/synthesiser.pysrc/synthorg/meta/chief_of_staff/propose.pysrc/synthorg/meta/chief_of_staff/routing.pysrc/synthorg/meta/strategies/code_modification.pysrc/synthorg/meta/toolsmith/strategy.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/observability/events/api.pysrc/synthorg/observability/events/cross_deployment.pysrc/synthorg/observability/events/evals.pysrc/synthorg/observability/events/provider.pysrc/synthorg/persistence/cost_record_protocol.pysrc/synthorg/persistence/postgres/cost_record_repo.pysrc/synthorg/persistence/postgres/revisions/20260628000000_cost_record_prompt_class_id_idx.sqlsrc/synthorg/persistence/postgres/schema.sqlsrc/synthorg/persistence/sqlite/cost_record_repo.pysrc/synthorg/persistence/sqlite/revisions/20260628000000_cost_record_prompt_class_id_idx.sqlsrc/synthorg/persistence/sqlite/schema.sqlsrc/synthorg/providers/cost_recording.pysrc/synthorg/providers/management/service.pysrc/synthorg/research/planning/llm_planner.pysrc/synthorg/research/synthesis/llm_synthesizer.pysrc/synthorg/research/triage/llm.pysrc/synthorg/security/llm_evaluator.pysrc/synthorg/security/redteam/grounding/substrate.pysrc/synthorg/security/safety_classifier.pysrc/synthorg/security/uncertainty.pysrc/synthorg/security/visionverify/verifiers/llm_vision.pytests/conformance/persistence/test_core_repositories.pytests/evals_spine/test_scorecard_emit.pytests/unit/api/controllers/test_activities.pytests/unit/api/controllers/test_budget.pytests/unit/api/controllers/test_departments_health.pytests/unit/budget/test_call_analytics.pytests/unit/budget/test_call_analytics_properties.pytests/unit/budget/test_forecast_history.pytests/unit/budget/test_pareto_assignments.pytests/unit/engine/classification/test_semantic_detectors.pytests/unit/hr/evaluation/test_pin_probe_runner.pytests/unit/hr/evaluation/test_pin_validation_benchmark.pytests/unit/hr/test_activity_list_recent.pytests/unit/hr/test_activity_service.pytests/unit/llm/test_model_pins.pytests/unit/providers/test_cost_recording_chokepoint.pytests/unit/scripts/test_check_pin_golden_fresh.pytests/unit/scripts/test_check_prompt_class_metadata.pytests/unit/settings/test_convention_rollout_settings.pyweb/src/__tests__/pages/budget/PromptClassSection.test.tsxweb/src/api/endpoints/budget.tsweb/src/api/types/budget.tsweb/src/api/types/dtos.gen.tsweb/src/api/types/openapi.gen.tsweb/src/mocks/handlers/budget.tsweb/src/pages/BudgetPage.tsxweb/src/pages/budget/CostBreakdownChart.stories.tsxweb/src/pages/budget/CostBreakdownChart.tsxweb/src/pages/budget/PromptClassSection.stories.tsxweb/src/pages/budget/PromptClassSection.tsxweb/src/utils/budget.ts
💤 Files with no reviewable changes (1)
- scripts/cost_scope_purpose_baseline.txt
| <ErrorBoundary level="section"> | ||
| <PromptClassSection /> | ||
| </ErrorBoundary> | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Scope PromptClassSection to the selected period.
This section is mounted without any time window, so it can only fetch the unbounded /budget/prompt-class-breakdown view. Once the user changes the page period, this table will still show lifetime totals while the surrounding budget widgets stay period-scoped, which makes the dashboard internally inconsistent. Thread the active window down here and include start / end in the section request. This matches the new time-windowed breakdown API added in this PR.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@web/src/pages/BudgetPage.tsx` around lines 259 - 262, PromptClassSection is
currently rendered without the active budget window, so it keeps fetching the
unbounded prompt-class breakdown while the rest of the dashboard is
period-scoped. Update BudgetPage to pass the selected period start/end into
PromptClassSection, then have PromptClassSection include those values in its
request to /budget/prompt-class-breakdown. Make sure the prop wiring and
fetch/query logic use the existing PromptClassSection symbol so the table stays
aligned with the currently selected period.
3a99bc8 to
e183339
Compare
e183339 to
37cfe80
Compare
There was a problem hiding this comment.
Actionable comments posted: 6
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/unit/hr/test_activity_service.py (1)
210-244: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winApply
prompt_class_idinside the cost-tracker fake.Both mocks accept
prompt_class_idand then ignore it, so the tests return the same records for every prompt class. That makes the new filter path impossible to validate and can let broken wiring pass.This matches the new prompt-class filter contract in the provided change summary.Proposed fix
async def _get( # noqa: PLR0913 -- mirrors CostTrackerProtocol.get_records *, agent_id: str | None = None, task_id: str | None = None, provider: str | None = None, prompt_class_id: str | None = None, start: datetime, end: datetime, limit: int = 100, offset: int = 0, ) -> tuple[CostRecord, ...]: - _ = (task_id, provider, prompt_class_id) + _ = (task_id, provider) matched = tuple( r for r in records if (agent_id is None or str(r.agent_id) == agent_id) + and ( + prompt_class_id is None + or r.prompt_class_id == prompt_class_id + ) and start <= r.timestamp <= end ) return matched[offset : offset + limit] async def _collect( # noqa: PLR0913 -- mirrors CostTrackerProtocol.collect_records *, agent_id: str | None = None, task_id: str | None = None, provider: str | None = None, prompt_class_id: str | None = None, start: datetime, end: datetime, ) -> tuple[CostRecord, ...]: - _ = (task_id, provider, prompt_class_id) + _ = (task_id, provider) return tuple( r for r in records if (agent_id is None or str(r.agent_id) == agent_id) + and ( + prompt_class_id is None + or r.prompt_class_id == prompt_class_id + ) and start <= r.timestamp <= end )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit/hr/test_activity_service.py` around lines 210 - 244, The cost-tracker fakes in the activity service tests ignore prompt_class_id, so the mock still returns all records regardless of prompt class. Update both the _get and _collect helpers in test_activity_service to filter records by prompt_class_id in addition to agent_id and timestamp, and keep the tracker wiring through CostTrackerProtocol/AsyncMock aligned with that behavior so the new prompt-class path is actually exercised.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@evals/models/scorecard.py`:
- Around line 149-162: The prompt_class_usage map in the scorecard model
currently validates only non-negative counts, so blank prompt class ids can
still pass through and create unusable buckets. Update the relevant scorecard
models, including the prompt_class_usage field and its validation logic in
ScorecardReport and any related shared definitions, so keys are treated as
NotBlankStr and blank/whitespace-only ids are rejected at the model boundary.
Keep the existing non-negative count check, but add a key validation step using
the shared prompt_class_id identifier path so downstream consumers only receive
valid prompt class ids.
In `@scripts/check_prompt_class_metadata.py`:
- Around line 329-341: cmd_scan_all currently assumes the passed project_root is
valid and only handles GateSourceError from _scan_all, which lets direct callers
get a false clean 0 for a non-repo directory. Reuse the same repository-root
validation used by main() at the start of cmd_scan_all, before calling
_scan_all, and return 2 with the existing stderr error path when the provided
root is outside the repo so the helper preserves its configuration-error
contract.
In `@src/synthorg/budget/call_analytics.py`:
- Around line 290-297: The mixed-currency branch in call_analytics should not
use the success event for a failure case; in the except
MixedCurrencyAggregationError handler, replace the
ANALYTICS_AGGREGATION_COMPUTED warning with the appropriate failure event so
downstream telemetry can distinguish it from successful breakdowns. Keep the
existing prompt_class_id and note context in the logger.warning call, and make
the change in the prompt-class breakdown path where assert_currencies_match
triggers the exception.
In `@src/synthorg/memory/retrieval/hierarchical/supervisor.py`:
- Around line 179-192: The router metadata currently only covers `_PURPOSE_ID`,
so the retry prompt class tagged with `_RETRY_PURPOSE_ID` is missing from the
class-level metadata contract. Update `supervisor.py` in the `metadata` property
(and any related class metadata registration used by `MEMORY_RETRIEVAL_RETRY`)
so both `PromptPurposeId.MEMORY_RETRIEVAL_ROUTE` and
`PromptPurposeId.MEMORY_RETRIEVAL_RETRY` are exposed consistently, likely by
returning or aggregating metadata for both prompt purposes instead of only
calling `pin_for(self._PURPOSE_ID)`.
In `@src/synthorg/persistence/cost_record_protocol.py`:
- Around line 29-31: The Field definition for prompt_class_id in the cost record
protocol still uses the old description text, so update the description to match
the prompt_class_id contract instead of “prompt purpose id.” Keep the change
localized to the prompt_class_id field in the persistence model so generated
schema/docs reflect the correct identifier name.
In `@tests/unit/providers/test_cost_recording_chokepoint.py`:
- Around line 185-215: The test only verifies the inner CostTracker, so it
misses regressions where the outer cost_recording_scope also records the call.
Tighten test_explicit_none_purpose_does_not_inherit_outer in
test_cost_recording_chokepoint.py by asserting tracker_outer stays empty after
provider.complete and drain_pending_records, alongside the existing inner
assertions. Use the existing cost_recording_scope, CostTracker, and
prompt_class_id checks to confirm explicit purpose=None does not inherit or emit
attribution on the outer scope.
---
Outside diff comments:
In `@tests/unit/hr/test_activity_service.py`:
- Around line 210-244: The cost-tracker fakes in the activity service tests
ignore prompt_class_id, so the mock still returns all records regardless of
prompt class. Update both the _get and _collect helpers in test_activity_service
to filter records by prompt_class_id in addition to agent_id and timestamp, and
keep the tracker wiring through CostTrackerProtocol/AsyncMock aligned with that
behavior so the new prompt-class path is actually exercised.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 80952b15-de16-4cf1-ae70-c80c4530998f
📒 Files selected for processing (119)
.github/workflows/ci.yml.opencode/agents/python-reviewer.md.opencode/commands/babysit-pr.md.opencode/plugins/synthorg-hooks.tsCLAUDE.mddata/architecture_report.jsondata/feature_index.jsondata/runtime_stats.yamldocs/design/evaluation-loop.mddocs/design/page-structure.mddocs/guides/monitoring.mddocs/reference/convention-gates.mddocs/reference/model-tier-policy.mdevals/models/scorecard.pyevals/run.pyevals/runner/execution.pyscripts/check_pin_golden_fresh.pyscripts/check_prompt_class_metadata.pyscripts/convention_gate_map.yamlscripts/cost_scope_purpose_baseline.txtscripts/refresh_model_pin_golden.pyscripts/run_prepush_python_gates.pysrc/synthorg/api/controllers/budget.pysrc/synthorg/api/controllers/meta_analytics.pysrc/synthorg/api/rate_limits/policies.pysrc/synthorg/api/services/analytics_read_service.pysrc/synthorg/budget/_tracker_helpers.pysrc/synthorg/budget/call_analytics.pysrc/synthorg/budget/call_analytics_models.pysrc/synthorg/budget/tracker.pysrc/synthorg/budget/tracker_protocol.pysrc/synthorg/client/generators/llm.pysrc/synthorg/engine/classification/semantic_detectors.pysrc/synthorg/engine/evolution/proposers/separate_analyzer.pysrc/synthorg/engine/intake/strategies/agent_intake.pysrc/synthorg/engine/intervention/proposer.pysrc/synthorg/engine/quality/decomposers/llm.pysrc/synthorg/engine/quality/graders/llm.pysrc/synthorg/engine/workspace/semantic_llm.pysrc/synthorg/hr/evaluation/pin_golden.jsonsrc/synthorg/hr/evaluation/pin_golden_compute.pysrc/synthorg/hr/evaluation/pin_probe.pysrc/synthorg/hr/evaluation/pin_validation_benchmark.pysrc/synthorg/hr/performance/llm_calibration_sampler.pysrc/synthorg/hr/performance/llm_judge_quality_strategy.pysrc/synthorg/hr/training/curation/llm_curated.pysrc/synthorg/knowledge/synthesis/llm_synthesizer.pysrc/synthorg/llm/__init__.pysrc/synthorg/llm/model_pins.pysrc/synthorg/llm/model_tier_policy.pysrc/synthorg/llm/prompt_purpose.pysrc/synthorg/memory/consolidation/abstractive.pysrc/synthorg/memory/consolidation/compressor.pysrc/synthorg/memory/consolidation/llm_op.pysrc/synthorg/memory/embedding/fine_tune_query.pysrc/synthorg/memory/procedural/proposer.pysrc/synthorg/memory/procedural/success_proposer.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pysrc/synthorg/memory/retrieval/reranking/llm_reranker.pysrc/synthorg/meta/charter/strategy.pysrc/synthorg/meta/chief_of_staff/chat.pysrc/synthorg/meta/chief_of_staff/narrative/synthesiser.pysrc/synthorg/meta/chief_of_staff/propose.pysrc/synthorg/meta/chief_of_staff/routing.pysrc/synthorg/meta/strategies/code_modification.pysrc/synthorg/meta/toolsmith/strategy.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/observability/events/api.pysrc/synthorg/observability/events/cross_deployment.pysrc/synthorg/observability/events/evals.pysrc/synthorg/observability/events/provider.pysrc/synthorg/persistence/cost_record_protocol.pysrc/synthorg/persistence/postgres/cost_record_repo.pysrc/synthorg/persistence/postgres/revisions/20260628000000_cost_record_prompt_class_id_idx.sqlsrc/synthorg/persistence/postgres/schema.sqlsrc/synthorg/persistence/sqlite/cost_record_repo.pysrc/synthorg/persistence/sqlite/revisions/20260628000000_cost_record_prompt_class_id_idx.sqlsrc/synthorg/persistence/sqlite/schema.sqlsrc/synthorg/providers/cost_recording.pysrc/synthorg/providers/management/service.pysrc/synthorg/research/planning/llm_planner.pysrc/synthorg/research/synthesis/llm_synthesizer.pysrc/synthorg/research/triage/llm.pysrc/synthorg/security/llm_evaluator.pysrc/synthorg/security/redteam/grounding/substrate.pysrc/synthorg/security/safety_classifier.pysrc/synthorg/security/uncertainty.pysrc/synthorg/security/visionverify/verifiers/llm_vision.pytests/conformance/persistence/test_core_repositories.pytests/evals_spine/test_scorecard_emit.pytests/unit/api/controllers/test_activities.pytests/unit/api/controllers/test_budget.pytests/unit/api/controllers/test_departments_health.pytests/unit/budget/test_call_analytics.pytests/unit/budget/test_call_analytics_properties.pytests/unit/budget/test_forecast_history.pytests/unit/budget/test_pareto_assignments.pytests/unit/engine/classification/test_semantic_detectors.pytests/unit/hr/evaluation/test_pin_probe_runner.pytests/unit/hr/evaluation/test_pin_validation_benchmark.pytests/unit/hr/test_activity_list_recent.pytests/unit/hr/test_activity_service.pytests/unit/llm/test_model_pins.pytests/unit/providers/test_cost_recording_chokepoint.pytests/unit/scripts/test_check_pin_golden_fresh.pytests/unit/scripts/test_check_prompt_class_metadata.pytests/unit/settings/test_convention_rollout_settings.pyweb/src/__tests__/pages/budget/PromptClassSection.test.tsxweb/src/api/endpoints/budget.tsweb/src/api/types/budget.tsweb/src/api/types/dtos.gen.tsweb/src/api/types/openapi.gen.tsweb/src/mocks/handlers/budget.tsweb/src/pages/BudgetPage.tsxweb/src/pages/budget/CostBreakdownChart.stories.tsxweb/src/pages/budget/CostBreakdownChart.tsxweb/src/pages/budget/PromptClassSection.stories.tsxweb/src/pages/budget/PromptClassSection.tsxweb/src/utils/budget.ts
💤 Files with no reviewable changes (1)
- scripts/cost_scope_purpose_baseline.txt
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
src/synthorg/memory/retrieval/hierarchical/supervisor.py (1)
340-345: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winUse
self.metadata.prompt_class_idon the routing scope too.The retry path now reads its identifier from class metadata, but the routing path still hard-codes
_PURPOSE_ID. That leaves this rollout using two identifier sources in one class instead of the single metadata contract the prompt-class work is introducing.Minimal fix
async with cost_recording_scope( cost_tracker=self._cost_tracker, agent_id=query.agent_id, task_id=NotBlankStr("system:memory:retrieval_route"), - purpose=self._PURPOSE_ID, + purpose=self.metadata.prompt_class_id, call_category=LLMCallCategory.SYSTEM, ):🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/synthorg/memory/retrieval/hierarchical/supervisor.py` around lines 340 - 345, The routing cost-recording scope in supervisor.py still hard-codes the identifier instead of using the prompt-class metadata contract. Update the `cost_recording_scope` call in `Supervisor` to use `self.metadata.prompt_class_id` for the routing path, matching the retry path and keeping both paths on the same identifier source.tests/unit/providers/test_cost_recording_chokepoint.py (1)
222-259: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winExercise the provider chokepoint in these trackerless tests.
Both tests leave the scope body empty, so they never hit the
complete()path described in the comments. A regression that skipsPROVIDER_PROMPT_PURPOSE_INVOKEDonly whencost_tracker=Noneduring provider invocation would still pass here.Minimal fix
async def test_purpose_logged_without_tracker(self) -> None: + provider = _StubProvider() with capture_logs() as logs: async with cost_recording_scope( cost_tracker=None, agent_id=NotBlankStr("agent-1"), task_id=NotBlankStr("task-1"), purpose=PromptPurposeId.MEMORY_RERANK, call_category=LLMCallCategory.SYSTEM, ): - pass + await provider.complete([_msg()], "test-model") @@ async def test_no_purpose_log_when_purpose_none(self) -> None: + provider = _StubProvider() with capture_logs() as logs: async with cost_recording_scope( cost_tracker=None, agent_id=NotBlankStr("agent-1"), task_id=NotBlankStr("task-1"), call_category=LLMCallCategory.SYSTEM, ): - pass + await provider.complete([_msg()], "test-model")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit/providers/test_cost_recording_chokepoint.py` around lines 222 - 259, The trackerless tests are not exercising the provider chokepoint because the scope bodies are empty, so they never drive the `complete()` path where `PROVIDER_PROMPT_PURPOSE_INVOKED` should be emitted. Update `test_purpose_logged_without_tracker` and `test_no_purpose_log_when_purpose_none` in `test_cost_recording_chokepoint.py` to invoke the actual provider path through `cost_recording_scope`/`complete()` instead of only entering and exiting the context, while keeping the same assertions on `PROVIDER_PROMPT_PURPOSE_INVOKED`, `PromptPurposeId.MEMORY_RERANK`, and the no-purpose case.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/synthorg/memory/retrieval/hierarchical/supervisor.py`:
- Around line 340-345: The routing cost-recording scope in supervisor.py still
hard-codes the identifier instead of using the prompt-class metadata contract.
Update the `cost_recording_scope` call in `Supervisor` to use
`self.metadata.prompt_class_id` for the routing path, matching the retry path
and keeping both paths on the same identifier source.
In `@tests/unit/providers/test_cost_recording_chokepoint.py`:
- Around line 222-259: The trackerless tests are not exercising the provider
chokepoint because the scope bodies are empty, so they never drive the
`complete()` path where `PROVIDER_PROMPT_PURPOSE_INVOKED` should be emitted.
Update `test_purpose_logged_without_tracker` and
`test_no_purpose_log_when_purpose_none` in `test_cost_recording_chokepoint.py`
to invoke the actual provider path through `cost_recording_scope`/`complete()`
instead of only entering and exiting the context, while keeping the same
assertions on `PROVIDER_PROMPT_PURPOSE_INVOKED`,
`PromptPurposeId.MEMORY_RERANK`, and the no-purpose case.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 64813ff9-af2a-47ee-a2f6-b0f4a5b56ad8
📒 Files selected for processing (8)
evals/models/scorecard.pyscripts/check_prompt_class_metadata.pysrc/synthorg/budget/call_analytics.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/persistence/cost_record_protocol.pytests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.py
📜 Review details
⏰ Context from checks skipped due to timeout. (30)
- GitHub Check: Dashboard Test (shard 2/4)
- GitHub Check: Dashboard Test (shard 1/4)
- GitHub Check: Dashboard Test (shard 3/4)
- GitHub Check: Dashboard Test (shard 4/4)
- GitHub Check: Dashboard Build
- GitHub Check: Dashboard Type Check
- GitHub Check: Build Backend
- GitHub Check: Build Fine-Tune (gpu, fine-tune-gpu)
- GitHub Check: Build Fine-Tune (cpu, fine-tune-cpu)
- GitHub Check: CodSpeed Python benchmarks
- GitHub Check: CodSpeed Web benchmarks
- GitHub Check: Lighthouse Dashboard
- GitHub Check: Lighthouse Site
- GitHub Check: Test Integration (shard 2)
- GitHub Check: Test Integration (shard 4)
- GitHub Check: Test Integration (shard 3)
- GitHub Check: Dashboard E2E (Playwright)
- GitHub Check: Test Conformance (SQLite)
- GitHub Check: Test Integration (shard 1)
- GitHub Check: Test E2E
- GitHub Check: Test Unit (shard 1)
- GitHub Check: Test Unit (shard 2)
- GitHub Check: Test Unit (shard 4)
- GitHub Check: Test Unit (shard 3)
- GitHub Check: Gates (pre-commit all-files + parity)
- GitHub Check: Build Web Assets (melange)
- GitHub Check: Build Preview
- GitHub Check: CLI Test (windows-latest)
- GitHub Check: pyright (advisory)
- GitHub Check: Analyze (python)
⚠️ CI failures not shown inline (2)
GitHub Actions: CLA / CLA Signature Check: feat: per-class ModelPinMetadata + cost-by-prompt-purpose dashboard + prompt-drift CI
Conclusion: failure
##[group]Run set -euo pipefail
�[36;1mset -euo pipefail�[0m
�[36;1m�[0m
�[36;1m# ── Helper: gh_api_retry ──�[0m
�[36;1m# Wraps `gh api` with bounded exponential-with-cap retry on�[0m
�[36;1m# transient failures (network errors, 5xx, EPIPE-class).�[0m
�[36;1m# Definitive client errors (4xx) fail fast: retrying a missing�[0m
�[36;1m# branch or revoked token wastes the retry budget.�[0m
�[36;1m# Backoff: 15, 30, 60, 120, 120, 120, 120 = ~10 min over�[0m
�[36;1m# 8 attempts (7 sleeps). Job timeout is 12 min for headroom.�[0m
�[36;1mgh_api_retry() {�[0m
�[36;1m local attempt=0�[0m
�[36;1m local max_attempts=8�[0m
�[36;1m local delay=15�[0m
�[36;1m local rc�[0m
�[36;1m while : ; do�[0m
�[36;1m attempt=$((attempt + 1))�[0m
�[36;1m if gh api "$@" 2>/tmp/gh-api-err; then�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m rc=$?�[0m
�[36;1m # 401 is intentionally NOT in this list. On ``pull_request_target``�[0m
�[36;1m # the workflow runs against ``GITHUB_TOKEN`` from the repo, which�[0m
�[36;1m # has full scope -- a true credentials failure is impossible.�[0m
�[36;1m # Empirically the GHA auth service occasionally returns 401 on�[0m
�[36;1m # the very first API call right after a job starts, and the�[0m
�[36;1m # next attempt with the same freshly-issued token succeeds. So�[0m
�[36;1m # 401 here means "GitHub auth-service hiccup", same backoff�[0m
�[36;1m # ladder as 5xx.�[0m
�[36;1m if grep -qE 'HTTP 4(00|03|04|09|22)' /tmp/gh-api-err; then�[0m
�[36;1m echo "::error::gh api definitive 4xx (no retry): $(cat /tmp/gh-api-err)" >&2�[0m
GitHub Actions: CLA / 0_CLA Signature Check.txt: feat: per-class ModelPinMetadata + cost-by-prompt-purpose dashboard + prompt-drift CI
Conclusion: failure
##[group]Run set -euo pipefail
�[36;1mset -euo pipefail�[0m
�[36;1m�[0m
�[36;1m# ── Helper: gh_api_retry ──�[0m
�[36;1m# Wraps `gh api` with bounded exponential-with-cap retry on�[0m
�[36;1m# transient failures (network errors, 5xx, EPIPE-class).�[0m
�[36;1m# Definitive client errors (4xx) fail fast: retrying a missing�[0m
�[36;1m# branch or revoked token wastes the retry budget.�[0m
�[36;1m# Backoff: 15, 30, 60, 120, 120, 120, 120 = ~10 min over�[0m
�[36;1m# 8 attempts (7 sleeps). Job timeout is 12 min for headroom.�[0m
�[36;1mgh_api_retry() {�[0m
�[36;1m local attempt=0�[0m
�[36;1m local max_attempts=8�[0m
�[36;1m local delay=15�[0m
�[36;1m local rc�[0m
�[36;1m while : ; do�[0m
�[36;1m attempt=$((attempt + 1))�[0m
�[36;1m if gh api "$@" 2>/tmp/gh-api-err; then�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m rc=$?�[0m
�[36;1m # 401 is intentionally NOT in this list. On ``pull_request_target``�[0m
�[36;1m # the workflow runs against ``GITHUB_TOKEN`` from the repo, which�[0m
�[36;1m # has full scope -- a true credentials failure is impossible.�[0m
�[36;1m # Empirically the GHA auth service occasionally returns 401 on�[0m
�[36;1m # the very first API call right after a job starts, and the�[0m
�[36;1m # next attempt with the same freshly-issued token succeeds. So�[0m
�[36;1m # 401 here means "GitHub auth-service hiccup", same backoff�[0m
�[36;1m # ladder as 5xx.�[0m
�[36;1m if grep -qE 'HTTP 4(00|03|04|09|22)' /tmp/gh-api-err; then�[0m
�[36;1m echo "::error::gh api definitive 4xx (no retry): $(cat /tmp/gh-api-err)" >&2�[0m
🧰 Additional context used
📓 Path-based instructions (7)
src/synthorg/persistence/**
📄 CodeRabbit inference engine (CLAUDE.md)
Only files under
src/synthorg/persistence/may import sqlite/psycopg or emit raw SQL; new repository protocols must inherit a generic category frompersistence/_generics.py; bespoke repository methods are only allowed under ADR-0001 D7; protocols must remain@runtime_checkable.
Files:
src/synthorg/persistence/cost_record_protocol.py
**/*.{py,ts,tsx,js,jsx,go}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{py,ts,tsx,js,jsx,go}: No hardcoded values: numerics should live insettings/definitions/, with only the stated allowlist exceptions.
EachErrorCodemust map to exactly oneDomainErrorsubclass, with only the documented inheritance alias andSHAREABLE_CODESexemptions; per-line opt-out uses# lint-allow: error-code-uniqueness.
Everycost_recording_scope()call must passpurpose=(aPromptPurposeIdor explicitNone) so cost attribution does not silently regress.
Every prompt class that tags an LLM chokepoint with a non-Nonepurpose=must expose a@property metadata -> ModelPinMetadatareturningpin_for(self._PURPOSE_ID); pins must live only inllm/model_pins.py.
Keep modules within the tiered LOC budgets defined by# module-kind:(with the stated exemptions for declarative/generated files).
Respect import layering and architecture-drift rules: use declarative.importlintercontracts, avoid raw-SQL boundary violations and DTO leaks, preserve dependency inversion, keep hub__init__files light, and place shared types incore.*/execution.*leaves.
Files:
src/synthorg/persistence/cost_record_protocol.pytests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pyevals/models/scorecard.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.pyscripts/check_prompt_class_metadata.py
**/*.{py,js,ts,tsx,go}
📄 CodeRabbit inference engine (CLAUDE.md)
After an issue, branch, commit, and push before PR review; use
/pre-pr-reviewand then/aurelio-review-prafter the PR, fixing all valid issues without deferring.
Files:
src/synthorg/persistence/cost_record_protocol.pytests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pyevals/models/scorecard.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.pyscripts/check_prompt_class_metadata.py
**/*.py
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.py: Python code must follow the repository conventions: nofrom __future__ import annotations; keep type-only imports at module level; add type hints to public functions; use Google-style docstrings; keep functions under 50 lines; use the prescribed error hierarchy, Pydantic v2 settings, args models, immutability, async, clock seam, prompt safety, repository CRUD, and UTC datetime helpers.
Usefrom synthorg.observability import get_logger; name the logger variablelogger; avoidimport loggingandprint()in app code; use structured event constants and redact secrets in error logging.
For startup, MCP, telemetry, resilience, and conversational-org interface code, follow the documented boot phases, tool-handler contracts, allowed telemetry properties, retry policy, and feature-gating rules.
Follow the testing implementation conventions in Python tests: async fixtures, timeout settings, coverage expectations, Hypothesis determinism, and no flaky skip/xfail patterns.
Files:
src/synthorg/persistence/cost_record_protocol.pytests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pyevals/models/scorecard.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.pyscripts/check_prompt_class_metadata.py
**/*
📄 CodeRabbit inference engine (CLAUDE.md)
Use the prescribed commit/branch/signing/pre-commit workflow and clear failed push markers with the provided script rather than deleting them manually.
Files:
src/synthorg/persistence/cost_record_protocol.pytests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pyevals/models/scorecard.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.pyscripts/check_prompt_class_metadata.py
src/**/*.py
⚙️ CodeRabbit configuration file
This project uses Python 3.14+ with PEP 758 except syntax: "except A, B:" (comma-separated, no parentheses) is correct and mandatory -- do NOT flag it as a typo or suggest parenthesized form. The "except builtins.MemoryError, RecursionError: raise" pattern is intentional project convention for system-error propagation. When evaluating the 50-line function limit, count only the function body excluding the signature lines, decorators, and docstring. Functions 1-5 lines over due to docstrings or multi-line signatures should not be flagged. Do not suggest extracting single-use helper functions called exactly once -- this reduces readability without improving maintainability.
Files:
src/synthorg/persistence/cost_record_protocol.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.py
tests/**/*.py
📄 CodeRabbit inference engine (CLAUDE.md)
Test code must use the approved markers, async/test-timeout conventions, xdist defaults, Windows event-loop rules, double-selection rules, entity-id helpers, vendor-agnostic names, Hypothesis practices, and dual-backend conformance fixtures described in the testing guidance.
Files:
tests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.py
⚙️ CodeRabbit configuration file
Test files do not require Google-style docstrings on classes or functions -- ruff D rules are only enforced on src/. A bare
@settings() decorator with no arguments on Hypothesis property tests is a no-op and should not be suggested -- the HYPOTHESIS_PROFILE env var controls example counts via registered profiles, which@given() honors automatically.
Files:
tests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.py
🧠 Learnings (24)
📓 Common learnings
Learnt from: CR
Repo: Aureliolo/synthorg
Timestamp: 2026-06-28T15:57:46.826Z
Learning: Read `docs/design/` before implementing; any deviation from the design spec requires approval.
Learnt from: CR
Repo: Aureliolo/synthorg
Timestamp: 2026-06-28T15:57:46.826Z
Learning: Present every plan for accept/deny before coding.
Learnt from: CR
Repo: Aureliolo/synthorg
Timestamp: 2026-06-28T15:57:46.826Z
Learning: Use regional defaults: no privileged region/currency/locale, use metric units, and prefer British English.
Learnt from: CR
Repo: Aureliolo/synthorg
Timestamp: 2026-06-28T15:57:46.826Z
Learning: Every convention PR must ship its enforcement gate.
Learnt from: CR
Repo: Aureliolo/synthorg
Timestamp: 2026-06-28T15:57:46.826Z
Learning: Do not add AGPL/GPL (non-LGPL) dependencies; LGPL dependencies must be attributed in `NOTICE`; `golangci-lint` must remain an external binary; exclude `pymupdf`/`fitz`/`pymupdf4llm`.
Learnt from: CR
Repo: Aureliolo/synthorg
Timestamp: 2026-06-28T15:57:46.826Z
Learning: Configuration precedence must be DB > env > code default (Cat-1), env > code default (Cat-2), and Cat-3 bootstrap secrets must come only from environment variables; `os.environ.get` is forbidden outside startup/entry-point/dynamic-secret/Cat-3 allowlists or explicit `# lint-allow: env-read`.
📚 Learning: 2026-05-05T09:04:46.195Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 1760
File: scripts/_dual_backend_parity_lib.py:215-216
Timestamp: 2026-05-05T09:04:46.195Z
Learning: This repository targets Python 3.14+ and follows PEP 758. Therefore, reviewer tooling should NOT treat unparenthesized multi-exception `except` clauses written without an `as` clause (e.g., `except MemoryError, RecursionError:`) as syntax errors. Only flag `except`-clause problems when they are genuinely invalid for Python 3.14+.
Applied to files:
src/synthorg/persistence/cost_record_protocol.pytests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pyevals/models/scorecard.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.pyscripts/check_prompt_class_metadata.py
📚 Learning: 2026-05-21T22:55:20.496Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2035
File: src/synthorg/meta/toolsmith/models.py:114-114
Timestamp: 2026-05-21T22:55:20.496Z
Learning: In this repo’s “magic number” review standard, the existing gate in `scripts/check_no_magic_numbers.py` intentionally does NOT flag numeric literals used as raw call-site arguments. So, do not flag numeric literals passed as keyword arguments to Pydantic `Field()` (e.g., `Field(ge=0, le=100)` / `Field(ge=1, le=50)`)—this is an established idiom. Only treat numeric literals as “magic numbers” when they occur in the locations the gate checks (module-level assignments and function/method parameter defaults).
Applied to files:
src/synthorg/persistence/cost_record_protocol.pytests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pyevals/models/scorecard.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.pyscripts/check_prompt_class_metadata.py
📚 Learning: 2026-05-29T08:50:58.380Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2160
File: src/synthorg/persistence/sqlite/escalation_repo.py:370-370
Timestamp: 2026-05-29T08:50:58.380Z
Learning: In this repo, Ruff flake8-unused-arguments (ARG002) already suppresses unused-argument warnings on parameters of methods decorated with `override` (from `typing`). Therefore, if you see `# noqa: ARG002` (or equivalent) on parameters of an `override`-decorated method, treat it as stale/unused and remove it. Do not recommend re-adding `# noqa: ARG002` in these cases, because Ruff will flag the redundant directive (RUF100) and fail the Ruff CI gate.
Applied to files:
src/synthorg/persistence/cost_record_protocol.pytests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pyevals/models/scorecard.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.pyscripts/check_prompt_class_metadata.py
📚 Learning: 2026-06-09T09:22:47.752Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2283
File: src/synthorg/engine/evolution/config.py:278-280
Timestamp: 2026-06-09T09:22:47.752Z
Learning: This repository uses ruff rule TC006 (`runtime-cast-value`), which requires the quoted string-literal form for the type argument in `cast()` calls (e.g., `cast("object", value)` rather than `cast(object, value)`). During code review, do not suggest removing the quotes from the `cast(<type>, ...)` first argument; the unquoted form will be auto-reverted by ruff on commit, so the quoted form should be treated as lint-compliant.
Applied to files:
src/synthorg/persistence/cost_record_protocol.pytests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pyevals/models/scorecard.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.pyscripts/check_prompt_class_metadata.py
📚 Learning: 2026-05-21T22:55:09.289Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2035
File: src/synthorg/meta/toolsmith/config.py:29-30
Timestamp: 2026-05-21T22:55:09.289Z
Learning: For this repo’s Pydantic configuration idiom, do not treat numeric literals passed directly as arguments to `pydantic.Field(...)` as “magic numbers” during review. This includes call-site usages like `Field(default=0.2, ge=0.0, le=1.0)` (e.g., in config models such as `ToolAuthoringConfig`, `ToolValidationConfig`, `ToolsmithConfig`). Do not request extracting those `Field(...)` numeric arguments into named constants, since the repo’s `scripts/check_no_magic_numbers.py` intentionally excludes call-site `Field(...)` numerics and relies on `Field(...)` as the canonical way to express these constraints/defaults.
Applied to files:
src/synthorg/persistence/cost_record_protocol.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.py
📚 Learning: 2026-05-31T18:00:32.445Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2180
File: src/synthorg/engine/intervention/models.py:182-203
Timestamp: 2026-05-31T18:00:32.445Z
Learning: In this repository, `NotBlankStr` is a Pydantic `Annotated[str, ...]` type alias (defined in `synthorg/core/types.py`). At runtime, calling `NotBlankStr(value)` acts like an identity/cast to `str(value)` and does not execute the `StringConstraints` or `AfterValidator(...)`. Therefore, during code review, do not treat `NotBlankStr(x)` used inside non-Pydantic model methods as a place that would raise `ValidationError`; it won’t. Similarly, when `tuple[NotBlankStr, ...]` values are involved, `NotBlankStr` erases to `str` at runtime, so membership tests/comparisons can be done with raw `str` values.
Applied to files:
src/synthorg/persistence/cost_record_protocol.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.py
📚 Learning: 2026-06-10T12:09:37.293Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2295
File: src/synthorg/project_brain/service.py:77-80
Timestamp: 2026-06-10T12:09:37.293Z
Learning: Do not flag or recommend moving imports out of `if TYPE_CHECKING:` in `src/synthorg` when the imported class is a collaborator type that’s intentionally “concrete-faked” in tests (duck-typed stubs injected into code under constructor/typing annotations). This pattern is used to avoid runtime import side effects where runtime type enforcement (e.g., `typeguard`/`isinstance` checks against the concrete type) would cause the test fakes to be rejected. If the code relies only on annotations for those collaborators and tests provide duck-typed stubs, keep the import under `TYPE_CHECKING` rather than promoting it to a module-level runtime import.
Applied to files:
src/synthorg/persistence/cost_record_protocol.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.py
📚 Learning: 2026-06-10T12:09:46.221Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2295
File: src/synthorg/settings/dispatcher.py:43-46
Timestamp: 2026-06-10T12:09:46.221Z
Learning: In this repository’s Python modules, do not treat `if TYPE_CHECKING:` imports as violations of any “hoist to module level” rule when they are deliberately used to support duck-typed test fakes (“concrete-faked collaborators”).
Concretely: if the code includes a comment/docstring indicating that tests drive the module using a duck-typed stub (e.g., “Concrete-faked collaborator: tests drive … with a duck-typed … stub, so a runtime import would make typeguard reject the fake”), then keep that collaborator import/type reference under `if TYPE_CHECKING:` rather than importing it at runtime. The intent is to avoid runtime typeguard validation rejecting the test fake; these guards are intentional and should not be flagged.
Applied to files:
src/synthorg/persistence/cost_record_protocol.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.py
📚 Learning: 2026-06-20T14:08:03.848Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2423
File: src/synthorg/api/lifecycle.py:292-292
Timestamp: 2026-06-20T14:08:03.848Z
Learning: When reviewing calls to `provider_credential_catalog_of(app_state)` (from `src/synthorg/integrations/state.py`), treat it as a soft/optional accessor: it returns `None` when the provider credential catalog is not wired and does not raise. Do not flag such uses as potentially raising exceptions or as hard dependencies that would disable functionality if the catalog is absent. (By contrast, `connection_catalog_of(app_state)` is a hard accessor that raises via `require_service`.)
Applied to files:
src/synthorg/persistence/cost_record_protocol.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.py
📚 Learning: 2026-06-03T11:43:13.104Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2200
File: tests/unit/engine/artifacts/test_service.py:42-45
Timestamp: 2026-06-03T11:43:13.104Z
Learning: For the D7 protocol method `save_returning_outcome(artifact: Artifact) -> bool` defined in `src/synthorg/persistence/artifact_protocol.py`, any implementation—including fake/stub/test doubles—must use the exact same parameter name `artifact` (i.e., `save_returning_outcome(self, artifact=...)` / `save_returning_outcome(self, artifact: Artifact)`), not `entity`. This name must match for typeguard positional-or-keyword name conformance. Do not suggest renaming the protocol method’s parameter to `entity`.
Applied to files:
src/synthorg/persistence/cost_record_protocol.pytests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.py
📚 Learning: 2026-06-03T11:43:33.228Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2200
File: src/synthorg/persistence/sqlite/flight_recorder_repo.py:250-252
Timestamp: 2026-06-03T11:43:33.228Z
Learning: In Aureliolo/synthorg persistence repositories under src/synthorg/persistence/**/*.py, follow the existing error-path logging conventions:
1) For operational persistence failures (e.g., sqlite3/aiosqlite errors), emit a logger.warning before raising QueryError, using the form logger.warning(EVENT, error_type=..., error=safe_error_description(exc)), then raise QueryError.
2) For input-validation/guard paths (e.g., precondition checks like naive datetime checks such as if threshold.tzinfo is None: raise QueryError(...)), raise QueryError directly without any preceding logger.warning call.
Do not suggest adding a warning log before raise on input-validation guard paths.
Applied to files:
src/synthorg/persistence/cost_record_protocol.py
📚 Learning: 2026-06-03T11:43:33.228Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2200
File: src/synthorg/persistence/sqlite/flight_recorder_repo.py:250-252
Timestamp: 2026-06-03T11:43:33.228Z
Learning: When reviewing naive-datetime guards in persistence repositories under src/synthorg/persistence/** (e.g., in SQLite/Postgres flight recorder repos), follow the project’s deliberate consistency rule: guard naive datetimes using only `if threshold.tzinfo is None:` and do not add an additional `threshold.utcoffset() is None` check. Avoid suggesting the `utcoffset()` branch to keep behavior consistent with sibling repositories.
Applied to files:
src/synthorg/persistence/cost_record_protocol.py
📚 Learning: 2026-06-03T14:28:35.042Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2200
File: src/synthorg/persistence/sqlite/idempotency_repo.py:90-90
Timestamp: 2026-06-03T14:28:35.042Z
Learning: In the synthorg persistence layer, treat `normalize_utc(value: datetime)` as “coerce-by-design”: for valid `datetime` inputs it should not raise (naive datetimes are assigned UTC via `value.replace(tzinfo=UTC)`, aware datetimes are converted via `value.astimezone(UTC)`). During code review, don’t recommend wrapping `normalize_utc(...)` calls (e.g., `normalize_utc(now)`) in try/except to convert exceptions into persistence `QueryError`s, because that would mask programming bugs (non-`datetime` inputs). Instead, enforce the actual input-validation boundary used in this repo’s `purge_before` methods: use the explicit guard `if threshold.tzinfo is None: raise QueryError(...)` rather than relying on exception handling around `normalize_utc`.
Applied to files:
src/synthorg/persistence/cost_record_protocol.py
📚 Learning: 2026-06-09T08:55:30.949Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2282
File: src/synthorg/persistence/postgres/flight_recorder_repo.py:13-13
Timestamp: 2026-06-09T08:55:30.949Z
Learning: In Aureliolo/synthorg persistence modules, keep “type-only” imports (e.g., `from psycopg_pool import AsyncConnectionPool`) at module scope when they are deliberately hoisted so runtime annotation checking (e.g., via `typeguard`) can resolve annotations. During code review, do not suggest moving these imports under `TYPE_CHECKING` unless you can point to a proven import cycle or a concrete runtime failure in a supported install/configuration.
Applied to files:
src/synthorg/persistence/cost_record_protocol.py
📚 Learning: 2026-06-27T22:56:39.679Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2506
File: src/synthorg/persistence/sqlite/model_pin_validation_repo.py:122-171
Timestamp: 2026-06-27T22:56:39.679Z
Learning: In Aureliolo/synthorg, the `check_persistence_boundary` gate forbids repository-layer mutation logging inside persistence repositories (e.g., `src/synthorg/persistence/**`). Do not suggest adding post-commit `INFO` logs for `save`/`delete` from within repository modules. Instead, emit state-transition/audit `INFO` logs for writes from the service/ledger/benchmark layer that performs the persistence call. For the model pin validation flow specifically, the approved `INFO` log is `MODEL_PIN_VALIDATION_STAMPED`, emitted one layer up after the persistence write (not from the repository).
Applied to files:
src/synthorg/persistence/cost_record_protocol.py
📚 Learning: 2026-06-27T22:56:37.515Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2506
File: src/synthorg/persistence/postgres/model_pin_validation_repo.py:89-132
Timestamp: 2026-06-27T22:56:37.515Z
Learning: In Aureliolo/synthorg persistence repository implementations, avoid recommending INFO (or other) logs for successful mutation/write operations inside repository-layer methods (e.g., `save()`, `delete()`). Repository-layer mutation logging is disallowed by the `check_persistence_boundary` gate; instead, require mutation/state-transition logging to be implemented in the service/ledger/benchmark layer.
Applied to files:
src/synthorg/persistence/cost_record_protocol.py
📚 Learning: 2026-06-09T10:06:53.040Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2283
File: src/synthorg/engine/middleware/coordination_constraints.py:152-152
Timestamp: 2026-06-09T10:06:53.040Z
Learning: Use International/British English spellings throughout the repository (code comments, docstrings, and documentation). British spellings such as "Analyse" (not "Analyze"), "Behaviour" (not "Behavior"), and "Colour" (not "Color") are mandatory and should not be flagged as spelling errors or inconsistencies during code review. This repo’s style is enforced via `vale` (see `CLAUDE.md` → "Regional Defaults (MANDATORY)"). If an American spelling appears, prefer the corresponding British spelling.
Applied to files:
src/synthorg/persistence/cost_record_protocol.pytests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pyevals/models/scorecard.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.pyscripts/check_prompt_class_metadata.py
📚 Learning: 2026-06-13T08:51:11.124Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2348
File: docs/guides/custom-mcp-server-dev.md:155-161
Timestamp: 2026-06-13T08:51:11.124Z
Learning: When reviewing Aureliolo/synthorg usage of `mcp_descriptor()` (from `src/synthorg/meta/mcp/feature_descriptors.py`), ensure all call sites pass the keyword argument `handlers=...` (type `Callable[[], Mapping[str, object]]`). `mcp_descriptor()` maps this `handlers` value internally to the descriptor’s `handlers_factory` field; `handlers_factory=...` is not a valid keyword and will raise `TypeError: unexpected keyword argument`. Also ensure documentation and code examples do not suggest renaming `handlers` to `handlers_factory`; always show `handlers=`.
Applied to files:
src/synthorg/persistence/cost_record_protocol.pytests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pyevals/models/scorecard.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/budget/call_analytics.pyscripts/check_prompt_class_metadata.py
📚 Learning: 2026-06-27T16:49:03.704Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2501
File: src/synthorg/persistence/trust_state_protocol.py:1-11
Timestamp: 2026-06-27T16:49:03.704Z
Learning: In Aureliolo/synthorg, protocol modules under `src/synthorg/persistence/*_protocol.py` may intentionally omit the `# module-kind:` header and rely on the module-size gate’s default `code` / 500 LOC classification. When reviewing these sibling protocol modules (e.g., `audit_chain_protocol.py`, `promotion_history_protocol.py`, `hiring_request_protocol.py`, `agent_contribution_protocol.py`, `trust_state_protocol.py`), do not flag the missing `# module-kind:` header if the file follows this established `*_protocol.py` convention.
Applied to files:
src/synthorg/persistence/cost_record_protocol.py
📚 Learning: 2026-05-23T12:24:00.128Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2080
File: tests/_shared/test_postgres_proxy.py:19-48
Timestamp: 2026-05-23T12:24:00.128Z
Learning: When creating test doubles for Python typing.Protocols in tests, prefer a hand-written Protocol fake (a concrete class that explicitly implements the Protocol) over `mock_of[T]` if the Protocol only defines annotation-only attributes (e.g., `username: str`, `password: str`, `dbname: str`) with no class-level values/assignments. This is because `mock_of[T]` relies on `create_autospec(..., spec_set=True)`, which enumerates members via `dir(spec)`; annotation-only attributes are not included, so `mock_of`’s kwarg-based attribute setting can raise `AttributeError: attribute not present on spec type`. In that annotation-only case, don’t recommend `mock_of[T]`—use an explicit fake class instead.
Applied to files:
tests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.py
📚 Learning: 2026-06-11T11:04:22.231Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2311
File: tests/conformance/persistence/test_workflow_service.py:77-77
Timestamp: 2026-06-11T11:04:22.231Z
Learning: When using Aureliolo/synthorg test-id helpers, treat `sid(label)`/`as_uuid(label)` as label-only functions that deterministically hash a human-readable label into a UUID (uuid5). Do NOT call them on an already-canonical UUID value (string or `UUID` object), since this re-hashes into a different UUID and breaks key matching/persistence. Instead:
- Use `coerce_id(value)` / `as_pk(value)` for pass-through-or-hash behavior (accept label or canonical UUID/`UUID` object and return the canonical form without re-hashing).
- If you already have a `UUID` object (e.g., `definition.id`), use `str(definition.id)` (or `coerce_id(definition.id)` / `as_pk(definition.id)`) for string-key conversion—never `sid(str(definition.id))`.
Applied to files:
tests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.py
📚 Learning: 2026-06-19T14:28:19.662Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2415
File: tests/unit/api/controllers/test_decomposition.py:118-139
Timestamp: 2026-06-19T14:28:19.662Z
Learning: For pytest test modules in this repo, if the module defines a module-level `pytestmark = pytest.mark.<marker>` (e.g., `pytest.mark.unit`, `pytest.mark.integration`, `pytest.mark.e2e`, `pytest.mark.slow`), then individual test functions in that module do not need their own per-function `pytest.mark.<marker>` decorators to satisfy the repo’s marker requirement. Only flag a missing per-function marker when the module does not set `pytestmark` and the test function also lacks the required `pytest.mark.<marker>` marker(s).
Applied to files:
tests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.py
📚 Learning: 2026-06-19T14:28:49.107Z
Learnt from: Aureliolo
Repo: Aureliolo/synthorg PR: 2415
File: tests/unit/observability/audit_chain/test_signer.py:61-93
Timestamp: 2026-06-19T14:28:49.107Z
Learning: For pytest test modules under tests/, if the module defines a module-level marker like `pytestmark = pytest.mark.unit`, then the repository requirement “tests carry the unit marker” is satisfied for that module. In that case, do not flag individual test functions for missing `pytest.mark.unit` decorators solely because they lack per-function decorators—per-function markers would be redundant unless the module-level `pytestmark` is missing or not applicable.
Applied to files:
tests/unit/hr/test_activity_service.pytests/unit/providers/test_cost_recording_chokepoint.py
Per-class ModelPinMetadata: every prompt class that tags an LLM chokepoint with a non-None purpose exposes a metadata property returning pin_for, backed by the declarative llm/model_pins.py registry, enforced by check_prompt_class_metadata.py (the F1 gate). Cost attribution by prompt_class_id: cost records carry prompt_class_id, the analytics layer aggregates a per-prompt-class breakdown (cost, latency percentiles, cache-hit, retry, success), exposed via GET /budget/prompt-class-breakdown and rendered in a new budget dashboard section with per-purpose alerts. Prompt-drift regression: a CI eval fingerprints each class's pin through the deterministic scripted provider, with a tier-change canary and golden-eval prompt_class linkage; pin_golden_compute.py is the shared source for the regen tool and the freshness gate. Includes round-1 review/CI fixes: fail-closed repo-root validation in the F1 gate, zero-call-row token/latency validation, duplicate pin-spec detection, trackerless purpose coercion, orchestration ratio computed from the same filtered snapshot as the counts, and the PromptClassSection test corrected for the non-nullable retry_rate field.
All 7 are CodeRabbit findings on head 37cfe80: - scorecard: prompt_class_usage keys are NotBlankStr so blank/whitespace prompt-class ids are rejected at the model boundary. - check_prompt_class_metadata: cmd_scan_all revalidates the repo root (validation centralised there) so direct callers fail closed too. - call_analytics: the mixed-currency breakdown failure logs a dedicated ANALYTICS_BREAKDOWN_MIXED_CURRENCY event instead of the success event. - supervisor: expose retry_metadata for the reflective-retry prompt class and route its chokepoint through it, completing the metadata contract. - cost_record_protocol: field description matches prompt_class_id. - test_cost_recording_chokepoint: assert the outer scope records nothing (the inner scope fully shadows it). - test_activity_service: the cost-tracker fakes filter by prompt_class_id so the new filter path is actually exercised.
…tadata CodeRabbit (outside-diff): the reflective-retry path now reads its prompt class from self.retry_metadata (round 3), but the routing path still hard-coded _PURPOSE_ID. Route it through self.metadata.prompt_class_id so both chokepoints in the class use the single metadata contract.
b795459 to
c056cc5
Compare
There was a problem hiding this comment.
Actionable comments posted: 9
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/synthorg/hr/evaluation/pin_validation_benchmark.py (1)
114-123: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winKey the cases and golden lookup by
pin.prompt_class_id.This generator still uses
purpose.idforEvalTestCase.idandself._golden[...], butgrade()treatspin.prompt_class_idas the canonical identity. If those ever diverge, the benchmark will fail as a malformed case instead of validating drift for the right prompt class.Proposed fix
for purpose in PROMPT_PURPOSE_REGISTRY.all_purposes(): pid = purpose.id pin = pin_for(pid) + class_id = str(pin.prompt_class_id) yield EvalTestCase( - id=NotBlankStr(str(pid)), + id=NotBlankStr(class_id), behavior_tags=_CASE_TAGS, input_data=probe_input_data(pid), - expected_output=self._golden.get(str(pid), ""), + expected_output=self._golden.get(class_id, ""), metadata={PIN_META_KEY: pin_metadata_payload(pin)}, )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/synthorg/hr/evaluation/pin_validation_benchmark.py` around lines 114 - 123, The benchmark case identity and golden lookup in the EvalTestCase generator are still keyed by purpose.id instead of the canonical pin.prompt_class_id used by grade(). Update the loop in the generator to derive the case id and self._golden lookup from pin.prompt_class_id, while keeping pin_for(pid) and probe_input_data(pid) as-is so the generated cases align with the grading identity.
♻️ Duplicate comments (1)
src/synthorg/hr/evaluation/pin_golden_compute.py (1)
39-50: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winValidate
case.metadatabefore running the probe.Line 40 still executes
runner.run_case(case)before the mismatch check on Lines 45-50, so a malformed case can run the wrong prompt class before this helper fails.Minimal fix
golden: dict[str, str] = {} async for case in benchmark.load_test_cases(): - output = await runner.run_case(case) pin = pin_from_case_metadata(case.metadata) # Mirror ``ModelPinValidationBenchmark.grade``: a case whose metadata # pins a different prompt class than its id is malformed; fail loudly # rather than write a wrong golden the freshness canary would trust. if str(pin.prompt_class_id) != str(case.id): msg = ( "malformed pin-validation case: " f"id {case.id} != pinned prompt_class_id {pin.prompt_class_id}" ) raise ValueError(msg) + output = await runner.run_case(case) golden[str(case.id)] = fingerprint_for(pin, output)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/synthorg/hr/evaluation/pin_golden_compute.py` around lines 39 - 50, Validate the case metadata before invoking the probe in pin_golden_compute’s loop. In the async loop around benchmark.load_test_cases(), move the pin_from_case_metadata and prompt_class_id vs case.id check ahead of runner.run_case(case), so malformed cases fail immediately without executing the wrong prompt class. Keep the existing ValueError path in pin_golden_compute and use the same case.metadata/pin variables to gate the run.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/check_prompt_class_metadata.py`:
- Around line 261-279: The _is_metadata_property helper is currently allowing
async metadata accessors, which can incorrectly validate classes that will fail
when metadata is accessed. Update _is_metadata_property to reject
ast.AsyncFunctionDef and only accept synchronous ast.FunctionDef nodes for the
metadata property, while keeping the existing checks for the property decorator,
name, and return annotation.
In `@src/synthorg/api/controllers/budget.py`:
- Around line 82-89: The `start`/`end` query parameters in the budget controller
are accepted as `datetime` values but can still be naive, so they should be
normalized or rejected before reaching the breakdown scan. Update the controller
boundary that uses `_StartFilter` and `_EndFilter` to run `normalize_utc()` on
these values before passing them into the service, and reject naive datetimes if
normalization is not possible. Apply the same handling anywhere the controller
forwards these timestamps so comparisons against stored UTC records stay
consistent.
In `@src/synthorg/budget/call_analytics_models.py`:
- Around line 135-190: The zero-call guard in _zero_calls_have_zero_rates still
allows an all-zero placeholder row through; tighten the validation so a row with
call_count == 0 is rejected outright in this model. Update the after-model
validator to raise a ValueError whenever call_count is 0, and keep the existing
prompt_class_id context in the message so callers can locate the bad row.
In `@src/synthorg/budget/tracker_protocol.py`:
- Around line 94-109: The protocol contract for collect_records is missing the
oldest-first ordering guarantee, which can let backends return arbitrary row
order and change behavior for full-snapshot callers. Update the collect_records
signature/docstring in CostTrackerProtocol to explicitly require oldest-first
results, and keep collect_all_records aligned with that contract since it
delegates to collect_records.
In `@src/synthorg/engine/quality/decomposers/llm.py`:
- Around line 371-373: The explicit purpose=None in the decomposer’s prompt
metadata is preventing prompt-class attribution and causing these calls to drop
out of the breakdown/drift pipeline. Update the LLMPrompt metadata in the
decomposer path to pass a real metadata.prompt_class_id from the dedicated
system prompt, or add a clearly documented exemption only if this prompt is
intentionally outside prompt-class inventory. Use the surrounding
cost_recording_scope() and the decomposer’s prompt construction site to locate
the change.
In `@src/synthorg/security/redteam/grounding/substrate.py`:
- Around line 108-113: The extraction and entailment paths are currently sharing
the same prompt_class_id, which merges two different prompt classes into one
tracking bucket. Update the completion flow in substrate.py so _run_completion()
receives the prompt purpose from each call site, and have _complete_extraction()
and _complete_entailment() pass distinct ids matching their different prompt
envelopes/tool contracts. Use the existing _PURPOSE_ID / metadata pattern as the
reference point and ensure the prompt_class_id recorded for each path is unique
and specific to that method.
In `@tests/unit/providers/test_cost_recording_chokepoint.py`:
- Around line 244-251: The trackerless test is only exercising the
default-argument path for cost_recording_scope, so update the calls in
test_no_purpose_log_when_purpose_none to pass purpose=None explicitly and keep
the no-purpose behavior covered. Make the same change in the related invocation
in the same test block so the contract is validated through the
cost_recording_scope API rather than omission.
In `@web/src/__tests__/pages/budget/PromptClassSection.test.tsx`:
- Around line 76-89: The PromptClassSection test is asserting the wrong
null-cell placeholder string, so it can pass even when the UI renders the
correct em dash. Update the assertion in PromptClassSection.test.tsx to expect
the actual em dash placeholder used by PromptClassSection for the nullable
latency/rate cells, and keep the check scoped to the four nullable fields
rendered by the component.
In `@web/src/pages/budget/PromptClassSection.tsx`:
- Around line 38-39: The table in PromptClassSection uses an arbitrary min-width
value that should be centralized instead of hardcoded. Replace the min-w-[48rem]
usage on the table element with an existing shared design token or utility
class, keeping the sizing consistent with the project’s design system. Locate
the table markup in PromptClassSection and update the className so the width
comes from a reusable token rather than a one-off layout value.
---
Outside diff comments:
In `@src/synthorg/hr/evaluation/pin_validation_benchmark.py`:
- Around line 114-123: The benchmark case identity and golden lookup in the
EvalTestCase generator are still keyed by purpose.id instead of the canonical
pin.prompt_class_id used by grade(). Update the loop in the generator to derive
the case id and self._golden lookup from pin.prompt_class_id, while keeping
pin_for(pid) and probe_input_data(pid) as-is so the generated cases align with
the grading identity.
---
Duplicate comments:
In `@src/synthorg/hr/evaluation/pin_golden_compute.py`:
- Around line 39-50: Validate the case metadata before invoking the probe in
pin_golden_compute’s loop. In the async loop around benchmark.load_test_cases(),
move the pin_from_case_metadata and prompt_class_id vs case.id check ahead of
runner.run_case(case), so malformed cases fail immediately without executing the
wrong prompt class. Keep the existing ValueError path in pin_golden_compute and
use the same case.metadata/pin variables to gate the run.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 5c2a50fb-2281-4a95-9d8d-ab6767198ef4
📒 Files selected for processing (119)
.github/workflows/ci.yml.opencode/agents/python-reviewer.md.opencode/commands/babysit-pr.md.opencode/plugins/synthorg-hooks.tsCLAUDE.mddata/architecture_report.jsondata/feature_index.jsondata/runtime_stats.yamldocs/design/evaluation-loop.mddocs/design/page-structure.mddocs/guides/monitoring.mddocs/reference/convention-gates.mddocs/reference/model-tier-policy.mdevals/models/scorecard.pyevals/run.pyevals/runner/execution.pyscripts/check_pin_golden_fresh.pyscripts/check_prompt_class_metadata.pyscripts/convention_gate_map.yamlscripts/cost_scope_purpose_baseline.txtscripts/refresh_model_pin_golden.pyscripts/run_prepush_python_gates.pysrc/synthorg/api/controllers/budget.pysrc/synthorg/api/controllers/meta_analytics.pysrc/synthorg/api/rate_limits/policies.pysrc/synthorg/api/services/analytics_read_service.pysrc/synthorg/budget/_tracker_helpers.pysrc/synthorg/budget/call_analytics.pysrc/synthorg/budget/call_analytics_models.pysrc/synthorg/budget/tracker.pysrc/synthorg/budget/tracker_protocol.pysrc/synthorg/client/generators/llm.pysrc/synthorg/engine/classification/semantic_detectors.pysrc/synthorg/engine/evolution/proposers/separate_analyzer.pysrc/synthorg/engine/intake/strategies/agent_intake.pysrc/synthorg/engine/intervention/proposer.pysrc/synthorg/engine/quality/decomposers/llm.pysrc/synthorg/engine/quality/graders/llm.pysrc/synthorg/engine/workspace/semantic_llm.pysrc/synthorg/hr/evaluation/pin_golden.jsonsrc/synthorg/hr/evaluation/pin_golden_compute.pysrc/synthorg/hr/evaluation/pin_probe.pysrc/synthorg/hr/evaluation/pin_validation_benchmark.pysrc/synthorg/hr/performance/llm_calibration_sampler.pysrc/synthorg/hr/performance/llm_judge_quality_strategy.pysrc/synthorg/hr/training/curation/llm_curated.pysrc/synthorg/knowledge/synthesis/llm_synthesizer.pysrc/synthorg/llm/__init__.pysrc/synthorg/llm/model_pins.pysrc/synthorg/llm/model_tier_policy.pysrc/synthorg/llm/prompt_purpose.pysrc/synthorg/memory/consolidation/abstractive.pysrc/synthorg/memory/consolidation/compressor.pysrc/synthorg/memory/consolidation/llm_op.pysrc/synthorg/memory/embedding/fine_tune_query.pysrc/synthorg/memory/procedural/proposer.pysrc/synthorg/memory/procedural/success_proposer.pysrc/synthorg/memory/retrieval/hierarchical/supervisor.pysrc/synthorg/memory/retrieval/reranking/llm_reranker.pysrc/synthorg/meta/charter/strategy.pysrc/synthorg/meta/chief_of_staff/chat.pysrc/synthorg/meta/chief_of_staff/narrative/synthesiser.pysrc/synthorg/meta/chief_of_staff/propose.pysrc/synthorg/meta/chief_of_staff/routing.pysrc/synthorg/meta/strategies/code_modification.pysrc/synthorg/meta/toolsmith/strategy.pysrc/synthorg/observability/events/analytics.pysrc/synthorg/observability/events/api.pysrc/synthorg/observability/events/cross_deployment.pysrc/synthorg/observability/events/evals.pysrc/synthorg/observability/events/provider.pysrc/synthorg/persistence/cost_record_protocol.pysrc/synthorg/persistence/postgres/cost_record_repo.pysrc/synthorg/persistence/postgres/revisions/20260628000000_cost_record_prompt_class_id_idx.sqlsrc/synthorg/persistence/postgres/schema.sqlsrc/synthorg/persistence/sqlite/cost_record_repo.pysrc/synthorg/persistence/sqlite/revisions/20260628000000_cost_record_prompt_class_id_idx.sqlsrc/synthorg/persistence/sqlite/schema.sqlsrc/synthorg/providers/cost_recording.pysrc/synthorg/providers/management/service.pysrc/synthorg/research/planning/llm_planner.pysrc/synthorg/research/synthesis/llm_synthesizer.pysrc/synthorg/research/triage/llm.pysrc/synthorg/security/llm_evaluator.pysrc/synthorg/security/redteam/grounding/substrate.pysrc/synthorg/security/safety_classifier.pysrc/synthorg/security/uncertainty.pysrc/synthorg/security/visionverify/verifiers/llm_vision.pytests/conformance/persistence/test_core_repositories.pytests/evals_spine/test_scorecard_emit.pytests/unit/api/controllers/test_activities.pytests/unit/api/controllers/test_budget.pytests/unit/api/controllers/test_departments_health.pytests/unit/budget/test_call_analytics.pytests/unit/budget/test_call_analytics_properties.pytests/unit/budget/test_forecast_history.pytests/unit/budget/test_pareto_assignments.pytests/unit/engine/classification/test_semantic_detectors.pytests/unit/hr/evaluation/test_pin_probe_runner.pytests/unit/hr/evaluation/test_pin_validation_benchmark.pytests/unit/hr/test_activity_list_recent.pytests/unit/hr/test_activity_service.pytests/unit/llm/test_model_pins.pytests/unit/providers/test_cost_recording_chokepoint.pytests/unit/scripts/test_check_pin_golden_fresh.pytests/unit/scripts/test_check_prompt_class_metadata.pytests/unit/settings/test_convention_rollout_settings.pyweb/src/__tests__/pages/budget/PromptClassSection.test.tsxweb/src/api/endpoints/budget.tsweb/src/api/types/budget.tsweb/src/api/types/dtos.gen.tsweb/src/api/types/openapi.gen.tsweb/src/mocks/handlers/budget.tsweb/src/pages/BudgetPage.tsxweb/src/pages/budget/CostBreakdownChart.stories.tsxweb/src/pages/budget/CostBreakdownChart.tsxweb/src/pages/budget/PromptClassSection.stories.tsxweb/src/pages/budget/PromptClassSection.tsxweb/src/utils/budget.ts
💤 Files with no reviewable changes (1)
- scripts/cost_scope_purpose_baseline.txt
| it('renders an em-dash placeholder for null latency and rate cells', async () => { | ||
| serveRows([ | ||
| row({ | ||
| avg_latency_ms: null, | ||
| p95_latency_ms: null, | ||
| cache_hit_rate: null, | ||
| success_rate: null, | ||
| }), | ||
| ]) | ||
| render(<PromptClassSection />) | ||
| await screen.findByText('system:cos:chat') | ||
| // retry_rate is always defined (0.0 over zero calls), so only the four | ||
| // genuinely nullable latency/rate cells render the '--' placeholder. | ||
| expect(screen.getAllByText('--')).toHaveLength(4) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Assert the actual em dash placeholder.
This test says the null-cell placeholder is an em dash, but it currently accepts "--". That weakens the contract and can fail against a correct — render.
Suggested fix
- // retry_rate is always defined (0.0 over zero calls), so only the four
- // genuinely nullable latency/rate cells render the '--' placeholder.
- expect(screen.getAllByText('--')).toHaveLength(4)
+ // retry_rate is always defined (0.0 over zero calls), so only the four
+ // genuinely nullable latency/rate cells render the '—' placeholder.
+ expect(screen.getAllByText('—')).toHaveLength(4)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| it('renders an em-dash placeholder for null latency and rate cells', async () => { | |
| serveRows([ | |
| row({ | |
| avg_latency_ms: null, | |
| p95_latency_ms: null, | |
| cache_hit_rate: null, | |
| success_rate: null, | |
| }), | |
| ]) | |
| render(<PromptClassSection />) | |
| await screen.findByText('system:cos:chat') | |
| // retry_rate is always defined (0.0 over zero calls), so only the four | |
| // genuinely nullable latency/rate cells render the '--' placeholder. | |
| expect(screen.getAllByText('--')).toHaveLength(4) | |
| it('renders an em-dash placeholder for null latency and rate cells', async () => { | |
| serveRows([ | |
| row({ | |
| avg_latency_ms: null, | |
| p95_latency_ms: null, | |
| cache_hit_rate: null, | |
| success_rate: null, | |
| }), | |
| ]) | |
| render(<PromptClassSection />) | |
| await screen.findByText('system:cos:chat') | |
| // retry_rate is always defined (0.0 over zero calls), so only the four | |
| // genuinely nullable latency/rate cells render the '—' placeholder. | |
| expect(screen.getAllByText('—')).toHaveLength(4) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@web/src/__tests__/pages/budget/PromptClassSection.test.tsx` around lines 76 -
89, The PromptClassSection test is asserting the wrong null-cell placeholder
string, so it can pass even when the UI renders the correct em dash. Update the
assertion in PromptClassSection.test.tsx to expect the actual em dash
placeholder used by PromptClassSection for the nullable latency/rate cells, and
keep the check scoped to the four nullable fields rendered by the component.
| <div className="overflow-x-auto"> | ||
| <table className="w-full min-w-[48rem] text-sm" aria-label="Cost by prompt purpose"> |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Replace the arbitrary table width with a shared token/class.
min-w-[48rem] bakes a one-off layout size into page code. Please switch this to an existing design token or shared utility so table sizing stays centrally managed. As per coding guidelines, "Do not hardcode hex colors, fonts, pixel spacing, Motion durations, BCP 47 locale literals, or currency symbols; use design tokens, @/lib/motion presets, @/utils/format, and DEFAULT_CURRENCY from @/utils/currencies."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@web/src/pages/budget/PromptClassSection.tsx` around lines 38 - 39, The table
in PromptClassSection uses an arbitrary min-width value that should be
centralized instead of hardcoded. Replace the min-w-[48rem] usage on the table
element with an existing shared design token or utility class, keeping the
sizing consistent with the project’s design system. Locate the table markup in
PromptClassSection and update the className so the width comes from a reusable
token rather than a one-off layout value.
Source: Coding guidelines
Split the red-team grounding substrate into two prompt classes so cost and drift attribute extraction and entailment separately: add RED_TEAM_GROUNDING_ENTAILMENT (tier + pin mirror the extraction class), thread the purpose through _run_completion, expose entailment_metadata, and regenerate the pin golden (39 fingerprints). Other findings: - Reject an async metadata property in check_prompt_class_metadata (an async property returns a coroutine, breaking self.metadata access). - Coerce naive ISO query datetimes to UTC at the budget controller boundary before the breakdown scan compares them to aware timestamps. - Constrain PromptClassBreakdownRow.call_count to gt=0; a row always aggregates at least one record, so the zero-call validator is removed. - Document the oldest-first ordering guarantee on collect_records. - Pass purpose=None explicitly in the trackerless chokepoint test. - Correct the misleading "em-dash" placeholder test name (the cell renders "--").
…l) (#2517) Closes #2478 Closes out the genuinely-remaining tail of the backend hardening epic. The bulk (~95%) already landed via #2510; the typed-boundary call-site fixes were moved to #2482 and the `ModelPinMetadata` per-class rollout is owned by #2511. This PR implements only what remained, after a line-by-line audit of every checklist item against current `main`. ## What changed - **S7 — MilestoneCriteria fail-closed.** A `model_validator` rejects `auto_promote=True` with zero `time_active_days`/`clean_history_days`; the default flips to opt-out. The previous silent runtime coercion is removed (the invariant now lives in the type). - **G7 — Verification grading live.** New `VerificationReviewStage` decomposes acceptance criteria into atomic probes and grades against a rubric with a *separate* evaluator identity, mapped onto the review pipeline's PASS/FAIL/SKIP. A grader or rubric fault fails OPEN (SKIP) so a verifier defect can never block task completion. Gated by `simulations.verification_review_enabled` (default on). - **G6 — Agent middleware chain un-parked + wired.** Root-caused the typeguard crash that had parked it: `builtin.py` referenced `CostTrackerProtocol`/`ApprovalGate`/`SecurityInterceptionStrategy` in `__init__` annotations while importing them under `if TYPE_CHECKING:`, so `register_agent_defaults` → `inspect.signature` → `__annotate__` raised `NameError`. Moving the three to module-level imports fixes it. The chain is now built at boot (gated by `engine.enable_agent_middleware`) and its `before_agent`/`after_agent` hooks fire at the engine boundary; the live effect is the authority-deference justification-header injection. - **F3/F4 — Eval-loop pluggable + LLM strategies + dispatcher.** Extracted the IDENTIFY/PROPOSE steps out of the (near-cap) coordinator behind `PatternIdentifier`/`FixProposer` protocols. Shipped defaults stay deterministic (threshold counting + a static action table); `hr.eval_loop_*_mode = llm` (with a model configured) swaps in provider-backed `LlmPatternIdentifier`/`LlmFixProposer` that degrade to the deterministic strategy on any provider or parsing failure. A concrete `RemediationActionDispatcher` now routes proposed actions to operators via the notification dispatcher (previously a no-op). Two new prompt purposes + tier/pin entries + a regenerated pin golden. - **L11 — won't-do (kept).** The hr/ single-implementation protocols are the pluggable seam by design; no code change. ## Also in this PR - Reverted a stray edit that had flipped the CLI setup-wizard telemetry default to opt-in (telemetry must be opt-in, off by default). - Kept `agent_engine.py` and the engine assembly under their module-size caps by extracting `_classification_assembly.py` and collapsing imports (no baseline edits). ## Review coverage Pre-reviewed by 7 agents (code/python/security/conventions reviewers, silent-failure-hunter, issue-resolution-verifier, ghost-wiring boot-reachability). issue-resolution-verifier: PASS (S7/G7/G6/F3/F4 all RESOLVED); ghost-wiring: all new components REACHABLE at boot; conventions: clean. 15 valid findings, all in this PR's own new code, all fixed — including two genuine bugs (the LLM proposer's actions were silently discarded at dispatch; a missing-default-rubric KeyError could escape the fail-open guard) and security hardening (agent ids redacted at the prompt boundary; model-returned action ids validated against a snake_case allowlist before reaching notification sinks). ## Test plan - `uv run python -m pytest tests/ -m unit` (full unit suite green via the pre-push gate). - New coverage: middleware before/after firing + typeguard-no-crash, verification stage (PASS/FAIL/REFER/SKIP, grader-fault and missing-default-rubric fail-open), LLM strategy happy-path + degrade-to-deterministic on empty/malformed/retryable error + non-retryable surface, action-id injection filter, dispatcher notification routing, MilestoneCriteria validator, boot wiring (deterministic default + llm mode + degrade-on-missing-provider). - mypy strict, ruff, every convention gate, schema-drift, pin-golden freshness, golangci/go, vale, lychee, import-linter, deptry, vulture, knip, dpdm all green on push.
<!-- HIGHLIGHTS_START --> ## Highlights > _AI-generated summary (model: `mistral-large-latest` via Mistral). Commit-based changelog below._ ### What you'll notice - Alerts now show their purpose and validate responses end-to-end. - Settings changes apply immediately without restarting the system. - Failed tool calls now show clear error messages in the model matcher. - Dashboard loads faster and works better with screen readers. ### What's new - Track costs by prompt purpose and model in the dashboard. - Ask questions directly in the knowledge synthesis (generative RAG) surface. - Setup wizard now ranks embedders and matches models automatically. ### Under the hood - Security and durability improvements across backend services. - Marketing site hardened for accessibility, SEO, and CSP compliance. <!-- HIGHLIGHTS_END --> :robot: I have created a release *beep* *boop* --- ## [0.9.2](v0.9.1...v0.9.2) (2026-06-30) ### Features * complete epic [#2490](#2490) (per-purpose alerts + end-to-end validation) ([#2518](#2518)) ([494e72b](494e72b)) * cost attribution by prompt_class_id + purpose gate ([#2504](#2504)) ([4339a6f](4339a6f)), closes [#2494](#2494) * dashboard API-type consolidation, correctness, and accessibility ([#2485](#2485)) ([d936b7f](d936b7f)) * dashboard UX pass (capabilities, MCP auto-reload, org-chart, dev-auth) ([166688c](166688c)) * feature-enablement overhaul (on-by-default posture, per-feature models, wizard surfaces) ([#2474](#2474)) ([f646a39](f646a39)), closes [#2471](#2471) * improve the setup wizard (model matching, embedder ranking, pure-API resume) ([#2472](#2472)) ([3fd3351](3fd3351)) * migrate marketing site to Astro 7 with fonts/CSP/SEO/a11y hardening ([#2465](#2465)) ([cfec64a](cfec64a)) * optional knowledge synthesis (generative RAG) ask surface ([#2473](#2473)) ([105de61](105de61)), closes [#2470](#2470) * per-class ModelPinMetadata + cost-by-prompt-purpose dashboard + prompt-drift CI ([#2511](#2511)) ([22ca74a](22ca74a)), closes [#2498](#2498) * prompt-purpose registry + eval-loop activation ([#2499](#2499)) ([92fb989](92fb989)), closes [#2493](#2493) * runtime tool-call failure feedback for the model matcher ([#2475](#2475)) ([331b0c0](331b0c0)), closes [#2469](#2469) * validated model pins (tier policy, drift benchmark, validator) ([#2506](#2506)) ([e6fb823](e6fb823)), closes [#2495](#2495) ### Bug Fixes * API layer error mapping, auth edges, and lifecycle fixes ([#2486](#2486)) ([47efc70](47efc70)), closes [#2479](#2479) * backend correctness, durability, and wiring hardening ([#2478](#2478) tail) ([#2517](#2517)) ([235a768](235a768)) * backend correctness, durability, and wiring hardening ([#2501](#2501)) ([f5a46c3](f5a46c3)) * backend dedup, wiring, and security hardening ([#2478](#2478) tail) ([#2510](#2510)) ([5d20a77](5d20a77)) * correctness, security and observability hardening ([#2464](#2464)) ([1ef9608](1ef9608)), closes [#2460](#2460) * settings hot-reload hardening (dispatcher kill switch + ghost fixes + restart-bound conversions) ([#2523](#2523)) ([49ef73c](49ef73c)) * tooling, tests, CI, gate, CLI, and typed-boundary call-site fixes ([#2484](#2484)) ([cc7f902](cc7f902)) * web dashboard audit bundle (UX, FE contract, a11y, TS strictness) ([#2455](#2455)) ([72c24de](72c24de)) ### Refactoring * [#2503](#2503) structural tail (classifier wiring, decision/transition events, layering) ([#2512](#2512)) ([83ac5b4](83ac5b4)) * audit fixes for backend API, lifecycle, security, auth, integrations + boot-wiring ([#2459](#2459)) ([d41d466](d41d466)) * audit fixes for engine, meta, HR, providers, memory, communication ([#2456](#2456)) ([e8366e0](e8366e0)) * audit fixes for persistence parity, budget, observability, settings ([#2457](#2457)) ([6a7788b](6a7788b)) * mechanical sweeps (dedup/centralisation, factory error-path logging, docstring accuracy) ([#2508](#2508)) ([1443862](1443862)) * prompt-class sampling-parameter cleanups ([#2500](#2500)) ([76a85bf](76a85bf)), closes [#2492](#2492) * remove redundant property Returns: docstrings ([#2509](#2509)) ([#2513](#2513)) ([dce87c4](dce87c4)) * **settings:** make per-request/per-tick knobs hot-reloadable + restart-required gate ([#2522](#2522)) ([d3281ca](d3281ca)), closes [#2514](#2514) * **settings:** make research/knowledge/hr/simulations subsystems hot via ghost-wire + live-gate ([#2520](#2520)) ([9101172](9101172)), closes [#2515](#2515) * **settings:** make self_improvement + chief_of_staff settings hot (live meta config overlay) ([#2521](#2521)) ([52e3185](52e3185)), closes [#2516](#2516) * structural rework, docs accuracy & cleanup ([#2463](#2463)) ([fecab21](fecab21)), closes [#2461](#2461) ### Documentation * align documentation and website with current implementation ([#2483](#2483)) ([2901aac](2901aac)), closes [#2481](#2481) * slim CLAUDE.md files and move detail into reference docs ([#2476](#2476)) ([010a9aa](010a9aa)) ### CI/CD * retry GHCR prune on transient 401 and track persistent failures ([#2466](#2466)) ([b5fe9c5](b5fe9c5)) * run GHCR prune weekly off the release path, drop wretry ([#2468](#2468)) ([7bd1c46](7bd1c46)) * update apko lockfiles ([#2454](#2454)) ([05645df](05645df)) * update apko lockfiles ([#2519](#2519)) ([71a2c98](71a2c98)) * wrap remaining transient registry ops in retry across the publish + verify path ([#2462](#2462)) ([7c22362](7c22362)) ### Maintenance * adopt eslint checkRelationalComparisons and harden review-dep-pr adopt-gate ([#2491](#2491)) ([b1fa2d0](b1fa2d0)) * harden codebase-audit skill against false positives and coverage gaps ([#2447](#2447)) ([b46c569](b46c569)) * Lock file maintenance ([#2507](#2507)) ([d444c9b](d444c9b)) * Update Infrastructure dependencies ([#2488](#2488)) ([6aafc09](6aafc09)) * Update Web dependencies ([#2489](#2489)) ([f82ee26](f82ee26)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: synthorg-repo-bot[bot] <279117679+synthorg-repo-bot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Completes epic #2490: every prompt class now exposes
ModelPinMetadata,CostRecord.prompt_class_idis populated across the prompt classes, and one identifier (prompt_class_id) feeds both the dashboard and the drift regression gate. The three issues ship together because the prompt classes and cost sites are largely the same files.#2496 — per-class ModelPinMetadata rollout + F1 gate
PromptPurposeIdmembers (38 purposes total), tierJUDGE_GRADE_VERIFY.llm/model_pins.py,pin_for(pid)); every prompt class exposesmetadata -> ModelPinMetadataand tags itscost_recording_scope(purpose=...).check_prompt_class_metadata.pygate (mirrors the temperature gate) with per-classlint-allowopt-out, wired into pre-push + CI + the gate map + CLAUDE.md.#2497 — cost/latency-by-prompt-purpose dashboard
prompt_class_idfilter threaded through the cost-record repos, tracker, call-analytics, and/budget/records+/budget/call-analytics.GET /budget/prompt-class-breakdown(full quality row: cost, tokens, calls, avg+p95 latency, cache-hit/retry/success) +PromptClassSectionpanel (pure API consumer, design tokens only, accessible table, abort-guarded fetch).#2498 — prompt-drift regression in CI + canary + golden-eval linkage
pin-drift-regressionCI job validating the real pins againstpin_golden.jsonfor all 38 purposes.check_pin_golden_fresh.pytier-change canary (policy/pin table changed without a golden regen fails).prompt_class_usageper brief + suite rollup.Review-round hardening (last commit)
clear()resets the lock;collect_all_recordsreads ONE atomic snapshot (collect_records) to close the offset-pagination TOCTOU under concurrent prune.analytics_read_service+ meta-analytics; canary friendly-failure messages.Nonenon-inheritance, PromptClassSection skeleton + null-cell states.Closes #2496
Closes #2497
Closes #2498