Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(report): expose max_issue_severity so a normalized verdict cannot hide a HIGH finding - #398

Merged
rng1995 merged 3 commits into
NVIDIA:mainfrom
Mark2Mac:report-max-issue-severity
Aug 20, 2026
Merged

rng1995 merged 3 commits into
NVIDIA:mainfrom
Mark2Mac:report-max-issue-severity

Conversation

@Mark2Mac

Copy link
Copy Markdown
Contributor

Closes #397.

What

Adds risk_assessment.max_issue_severity to the JSON report: the highest severity present in
issues[], or "NONE" when there are none.

Additive only. No existing field changes value, no scoring is touched, no behaviour changes.

Why

risk_assessment.severity is a normalised, confidence-weighted verdict, and it can read LOW /
SAFE
on a report whose issues[] contains a HIGH finding. A single HIGH scores below the
HIGH band, so the summary and the findings disagree — and nothing in the report says so.

I am not asking for the normalisation to go away. With rules that fire on documentation prose (see
#396), a max-severity verdict would read HIGH on almost everything, which is its own kind of
useless. The problem is that the smoothing is silent: every consumer that wants to gate on the
worst finding has to walk issues[] and re-implement the severity ranking, and each one does it
slightly differently.

This is not hypothetical for me. I run SkillSpector as a pre-install gate. The gate keyed on
risk_assessment.severity, and on a corpus of deliberately malicious fixtures it stopped 0 of
5
— every one reported LOW/SAFE with HIGH findings listed underneath. Keying on the worst
finding took it to 2 of 5 with static analysis alone. The remaining gap is detection, which is
fair. The first gap was the report telling me SAFE.

Design notes

  • _SEVERITY_RANK is deliberately separate from _SEVERITY_POINTS. The latter are scoring
    weights and may be retuned; this is an ordering that consumers will depend on, so it should not
    move when scoring does.
  • The value is computed from the same findings list the issues[] array is built from, so
    suppressed findings do not raise it. A finding excluded by a baseline is not a reported
    issue, and a field that counted it would make baselines useless for exactly the consumers this
    field is for. There is a test for this.
  • "NONE" rather than null or "LOW" for the empty case: null invites .get(...) returning
    a falsy value that compares oddly, and "LOW" would be indistinguishable from a report that
    really does have a LOW finding.

Tests

Three, in tests/nodes/test_report.py:

  • a single HIGH finding → max_issue_severity == "HIGH" while the verdict normalises to LOW
  • no findings → "NONE"
  • a CRITICAL finding suppressed by baseline plus a reported MEDIUM → "MEDIUM"

All three fail without the source change and pass with it (verified by reverting the one-line
addition and re-running: 3 failed → 3 passed).

One existing test needed updating: tests/nodes/analyzers/test_sc8_shipped_bytecode.py:61
compares risk_assessment against an exact dict, so any added key fails it. I only found
this by running the full suite — worth knowing if other PRs add fields there.

ruff check, ruff format --check and mypy are clean on the touched files.

Full suite on this branch: 2107 passed, 14 skipped, 4 xfailed, 1 failed in 26m22s. The one
failure is tests/unit/test_mcp_server.py::test_mcp_stdio_initialize_registers_scan_skill, and
it is not from this change: it fails identically with the change reverted on the same
machine (45.02s with, 47.86s without, load average 31-33). Its
asyncio.wait_for(session.initialize(), timeout=15) is a hardcoded budget smaller than the work
under load — the MCP subprocess does not finish the handshake in 15 s on a busy machine. Happy to
open that separately if it is not already known.

Not in this PR

The human-readable output still prints SAFE and then lists a [HIGH] line below it with
nothing connecting the two. A one-line note there would help, but it is a presentation change with
its own review surface, so I left it out rather than bundle it.

…t hide a HIGH finding

risk_assessment.severity is a normalized, confidence-weighted verdict and can
read LOW/SAFE while issues[] contains a HIGH finding: a single HIGH scores below
the HIGH band. The smoothing is intentional, but it was invisible, so every
consumer that wanted to gate on the worst finding had to walk issues[] and
re-implement the severity ranking.

max_issue_severity reports the highest severity present in issues[], or NONE
when there are none. Additive: no existing field changes value and no scoring is
touched.

_SEVERITY_RANK is kept separate from _SEVERITY_POINTS on purpose — the latter
are scoring weights that may be retuned, this is an ordering consumers will
depend on. Suppressed findings do not raise the value, since a finding excluded
by a baseline is not a reported issue.

Refs NVIDIA#397

Signed-off-by: Mark2Mac <[email protected]>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Approved. max_issue_severity is additive, derived from active reported findings after suppression, and kept separate from scoring weights. Tests cover worst severity, no findings, and suppressed findings; required CI is green.

@rng1995
rng1995 enabled auto-merge (squash) August 20, 2026 22:30
@rng1995
rng1995 disabled auto-merge August 20, 2026 22:38
@rng1995
rng1995 enabled auto-merge (squash) August 20, 2026 22:41
@rng1995
rng1995 merged commit d2f1832 into NVIDIA:main Aug 20, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

risk_assessment can say LOW/SAFE while issues[] contains a HIGH finding, with nothing in the report flagging the disagreement

2 participants