Render sandbox.firewall models.json in AWF step summaries#34088
Conversation
Co-authored-by: pelikhan <[email protected]>
Co-authored-by: pelikhan <[email protected]>
|
@copilot also render the model alias information |
Co-authored-by: pelikhan <[email protected]>
Co-authored-by: pelikhan <[email protected]>
There was a problem hiding this comment.
Pull request overview
Extends the AWF step-summary generator to include additional sandbox/firewall model inventory information so operators can see configured /reflect endpoints and runtime-discovered models in the GitHub Actions step summary.
Changes:
- Read and render
/tmp/gh-aw/sandbox/firewall/models.jsonas an additional “Runtime models.json” table in the step summary. - Add normalization/parsing helpers for runtime model payload shapes and model-ID extraction.
- Add Vitest coverage for reading/normalizing the new payload and for combined summary output.
Show a summary per file
| File | Description |
|---|---|
| actions/setup/js/awf_reflect_summary.cjs | Adds runtime models.json + awf-config parsing helpers and renders additional tables in the step summary output. |
| actions/setup/js/awf_reflect_summary.test.cjs | Adds tests for new file-reading helpers, normalization behavior, and the updated summary output. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 2/2 changed files
- Comments generated: 3
| * | ||
| * Supported payload shapes: | ||
| * - { endpoints: [...] } | ||
| * - { providers: { name: ... } } |
| pushRow(runtimeModelsData.provider, runtimeModelsData); | ||
| } | ||
|
|
||
| return rows.filter(row => row.provider).sort((a, b) => a.provider.localeCompare(b.provider) || a.endpoint.localeCompare(b.endpoint)); |
| if (modelAliasRows.length > 0) { | ||
| lines.push(""); | ||
| lines.push("Model aliases"); | ||
| lines.push(""); | ||
| lines.push("| Alias | Resolution order |"); | ||
| lines.push("|-------|------------------|"); | ||
| for (const row of modelAliasRows) { | ||
| lines.push(`| ${row.label} | ${formatModelList(row.targets, maxModels)} |`); | ||
| } | ||
| } |
|
Please post a brief blocker summary once those are resolved.
|
|
🧪 Test Quality Sentinel completed test quality analysis. |
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅ |
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. No ADR enforcement needed: PR #34088 does not have the implementation label and has 0 new lines of code in business logic directories (default threshold: 100). |
|
✅ PR Code Quality Reviewer completed the code quality review. |
There was a problem hiding this comment.
Code Quality Review
The implementation is well-structured and thoroughly tested, but there are a few maintainability issues that should be addressed before merging.
Review summary
Issues found:
- Scope mismatch — PR adds undocumented Model aliases feature from awf-config.json
- Inaccurate JSDoc — Comment doesn't match implementation for
providersfield shape - Redundant filter — Filter that will never remove any rows
What's done well:
- Comprehensive test coverage for all new functions
- Progressive disclosure with null-safe reads
- Flexible normalization handling multiple payload shapes
- Reuse of existing formatting helpers
🔎 Code quality review by PR Code Quality Reviewer · ● 653.4K
| for (const row of modelAliasRows) { | ||
| lines.push(`| ${row.label} | ${formatModelList(row.targets, maxModels)} |`); | ||
| } | ||
| } |
There was a problem hiding this comment.
Scope mismatch: PR adds Model aliases section from awf-config.json without mentioning it in title/description.
💡 Suggested approach
The PR title and description only mention extending summaries with sandbox/firewall/models.json, but lines 235-243 also add a "Model aliases" table sourced from /tmp/gh-aw/awf-config.json. This additional feature is undocumented in the PR description and example output.
Options:
- Update PR description and example to document the model aliases table
- Remove the model aliases feature and defer to a separate PR to keep scope focused
Adding undocumented features makes code review harder and creates incomplete changelogs.
| pushRow(runtimeModelsData.provider, runtimeModelsData); | ||
| } | ||
|
|
||
| return rows.filter(row => row.provider).sort((a, b) => a.provider.localeCompare(b.provider) || a.endpoint.localeCompare(b.endpoint)); |
There was a problem hiding this comment.
Redundant filter: filter(row => row.provider) always passes because provider defaults to "unknown".
💡 Suggested fix
The pushRow function on line 103 always assigns a non-empty provider string:
provider: String(provider || entry?.provider || entry?.name || "unknown")This means row.provider will never be falsy, so the filter on line 127 is redundant and will never remove any rows.
Options:
- Remove the filter:
.sort((a, b) => ...) - If the intent is to exclude unknown providers, allow empty provider values and filter explicitly:
.filter(row => row.provider && row.provider !== "unknown")
| /** | ||
| * Normalize runtime model entries to a common table-friendly shape. | ||
| * | ||
| * Supported payload shapes: |
There was a problem hiding this comment.
Inaccurate JSDoc: Comment lists incorrect shape for providers field.
💡 Suggested fix
The JSDoc on line 82 states:
* - { providers: { name: ... } }But the implementation on lines 113-117 treats providers as a map keyed by provider name, not an object with a name field:
for (const [provider, entry] of Object.entries(runtimeModelsData.providers)) {
pushRow(provider, entry);
}Update the JSDoc to reflect the actual supported shape:
* - { providers: { [providerName]: { endpoint: ..., models: [...] } } }There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /tdd and /grill-with-docs — requesting changes on domain language clarity and edge case coverage.
📋 Key Themes & Highlights
Key Themes
- Domain language drift — Five different field names for model lists suggests unclear vocabulary. Document the canonical schema and mark legacy fallbacks explicitly.
- Silent error handling — Malformed payload entries are filtered silently. Add edge case tests and consider debug logging for dropped data.
- Function declaration order — Helper functions called before declaration hurts readability and could break if refactored to arrow functions.
- Missing boundary tests — No tests for empty payloads, null endpoints, or malformed model entries.
Positive Highlights
- ✅ Comprehensive test coverage — Every new function has corresponding tests
- ✅ Progressive disclosure — Extends existing summary without breaking behavior
- ✅ Defensive parsing — All file reads gracefully handle missing/invalid files
- ✅ Clean separation — Runtime data vs. config data vs. aliases are clearly distinct
- ✅ Sorting consistency — Provider rows are deterministically sorted
🔍 Review Scope
Skills selected: /tdd (test-driven development) + /grill-with-docs (domain language consistency)
Focus areas:
- Test coverage for edge cases and boundary conditions
- Domain vocabulary clarity and consistency
- Error handling visibility and debugging support
- Code organization and readability
Not reviewed: Performance, security (no sensitive data), accessibility (markdown output)
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · ● 1M
| }); | ||
|
|
||
| describe("normalizeRuntimeModelRows", () => { | ||
| it("normalizes endpoint payloads into sorted rows", () => { |
There was a problem hiding this comment.
[/tdd] Missing edge case tests for empty and invalid runtime payloads.
💡 Add boundary condition tests
The tests cover valid payloads but not edge cases that could occur in production:
it('returns empty array for null or undefined', () => {
expect(module.normalizeRuntimeModelRows(null)).toEqual([]);
expect(module.normalizeRuntimeModelRows(undefined)).toEqual([]);
});
it('returns empty array for empty endpoints', () => {
expect(module.normalizeRuntimeModelRows({ endpoints: [] })).toEqual([]);
});
it('filters out null and undefined endpoints', () => {
const data = {
endpoints: [null, undefined, { provider: 'valid', models: [] }],
};
const rows = module.normalizeRuntimeModelRows(data);
expect(rows).toHaveLength(1);
expect(rows[0].provider).toBe('valid');
});These edge cases can happen if the firewall crashes mid-write or emits partial JSON.
🧪 Test Quality Sentinel Report✅ Test Quality Score: 86/100 — Excellent
📊 Metrics & Test Classification (11 tests analyzed)
Test Classification Details
Language SupportTests analyzed:
Verdict
📖 Understanding Test ClassificationsDesign Tests (High Value) verify what the system does:
Implementation Tests (Low Value) verify how the system does it:
Goal: Shift toward tests that describe the system's behavioral contract — the promises it makes to its users and collaborators.
|
|
|
|
@copilot review all comments and review. Resolve reviews you fix or ignore with a reason. |
Co-authored-by: pelikhan <[email protected]>
Handled in 56e7bdf. I fixed the two actionable review points in code: the |
AWF step summaries already showed
/reflectendpoint state, but not thesandbox.firewall/models.jsonpayload that captures the provider endpoints and runtime-discovered models actually available to the sandbox. This change extends the existing summary so operators can see both the configured proxy view and the emitted runtime model inventory in one place.Summary integration
awf_reflect_summary.cjsto read/tmp/gh-aw/sandbox/firewall/models.jsonalongsideawf-reflect.jsonRuntime models rendering
models.jsonshapes into a single table modelParsing support
models.jsonsafelyCoverage
models.jsonExample summary output:
pr-sous-chef: branch updated for maintainer follow-up.
Run: