feat(report): expose max_issue_severity so a normalized verdict cannot hide a HIGH finding - #398
Merged
Merged
Conversation
…t hide a HIGH finding risk_assessment.severity is a normalized, confidence-weighted verdict and can read LOW/SAFE while issues[] contains a HIGH finding: a single HIGH scores below the HIGH band. The smoothing is intentional, but it was invisible, so every consumer that wanted to gate on the worst finding had to walk issues[] and re-implement the severity ranking. max_issue_severity reports the highest severity present in issues[], or NONE when there are none. Additive: no existing field changes value and no scoring is touched. _SEVERITY_RANK is kept separate from _SEVERITY_POINTS on purpose — the latter are scoring weights that may be retuned, this is an ordering consumers will depend on. Suppressed findings do not raise the value, since a finding excluded by a baseline is not a reported issue. Refs NVIDIA#397 Signed-off-by: Mark2Mac <[email protected]>
rng1995
approved these changes
Aug 20, 2026
rng1995
left a comment
Collaborator
There was a problem hiding this comment.
[SkillSpector Review]
Approved. max_issue_severity is additive, derived from active reported findings after suppression, and kept separate from scoring weights. Tests cover worst severity, no findings, and suppressed findings; required CI is green.
rng1995
enabled auto-merge (squash)
August 20, 2026 22:30
rng1995
disabled auto-merge
August 20, 2026 22:38
rng1995
enabled auto-merge (squash)
August 20, 2026 22:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #397.
What
Adds
risk_assessment.max_issue_severityto the JSON report: the highest severity present inissues[], or"NONE"when there are none.Additive only. No existing field changes value, no scoring is touched, no behaviour changes.
Why
risk_assessment.severityis a normalised, confidence-weighted verdict, and it can read LOW /SAFE on a report whose
issues[]contains a HIGH finding. A single HIGH scores below theHIGH band, so the summary and the findings disagree — and nothing in the report says so.
I am not asking for the normalisation to go away. With rules that fire on documentation prose (see
#396), a max-severity verdict would read HIGH on almost everything, which is its own kind of
useless. The problem is that the smoothing is silent: every consumer that wants to gate on the
worst finding has to walk
issues[]and re-implement the severity ranking, and each one does itslightly differently.
This is not hypothetical for me. I run SkillSpector as a pre-install gate. The gate keyed on
risk_assessment.severity, and on a corpus of deliberately malicious fixtures it stopped 0 of5 — every one reported LOW/SAFE with HIGH findings listed underneath. Keying on the worst
finding took it to 2 of 5 with static analysis alone. The remaining gap is detection, which is
fair. The first gap was the report telling me SAFE.
Design notes
_SEVERITY_RANKis deliberately separate from_SEVERITY_POINTS. The latter are scoringweights and may be retuned; this is an ordering that consumers will depend on, so it should not
move when scoring does.
findingslist theissues[]array is built from, sosuppressed findings do not raise it. A finding excluded by a baseline is not a reported
issue, and a field that counted it would make baselines useless for exactly the consumers this
field is for. There is a test for this.
"NONE"rather thannullor"LOW"for the empty case:nullinvites.get(...)returninga falsy value that compares oddly, and
"LOW"would be indistinguishable from a report thatreally does have a LOW finding.
Tests
Three, in
tests/nodes/test_report.py:max_issue_severity == "HIGH"while the verdict normalises to LOW"NONE""MEDIUM"All three fail without the source change and pass with it (verified by reverting the one-line
addition and re-running: 3 failed → 3 passed).
One existing test needed updating:
tests/nodes/analyzers/test_sc8_shipped_bytecode.py:61compares
risk_assessmentagainst an exact dict, so any added key fails it. I only foundthis by running the full suite — worth knowing if other PRs add fields there.
ruff check,ruff format --checkandmypyare clean on the touched files.Full suite on this branch: 2107 passed, 14 skipped, 4 xfailed, 1 failed in 26m22s. The one
failure is
tests/unit/test_mcp_server.py::test_mcp_stdio_initialize_registers_scan_skill, andit is not from this change: it fails identically with the change reverted on the same
machine (45.02s with, 47.86s without, load average 31-33). Its
asyncio.wait_for(session.initialize(), timeout=15)is a hardcoded budget smaller than the workunder load — the MCP subprocess does not finish the handshake in 15 s on a busy machine. Happy to
open that separately if it is not already known.
Not in this PR
The human-readable output still prints
SAFEand then lists a[HIGH]line below it withnothing connecting the two. A one-line note there would help, but it is a presentation change with
its own review surface, so I left it out rather than bundle it.