Symptom
The CodeQL gate check fails on PRs with a title of the form:
190 new alerts including 190 high severity security vulnerabilities
"New alerts in code changed by this pull request"
Observed on #1405 at 04f43384. The alerts span ~21 backend route files. That PR changes exactly one route file (backend/routes/auth.ts), and every route it adds is rate-limited.
The attribution is false
Measured on #1405, comparing distinct (rule, file) pairs:
gh api --paginate ".../code-scanning/alerts?ref=refs/pull/1405/head&state=open"
gh api --paginate ".../code-scanning/alerts?ref=refs/heads/main&state=open"
| ref |
open alerts |
distinct (rule, file) pairs |
refs/pull/1405/head |
190 |
29 |
refs/heads/main |
505 |
85 |
comm -23 <pr_pairs> <main_pairs> returns empty — there is no (rule, file) pair on the PR head that is not already open on main. Every alert the gate calls "new in code changed by this pull request" is pre-existing repo baseline, dominated by js/missing-rate-limiting across pods.ts, skills.ts, admin/globalIntegrations.ts, admin/users.ts, integrations.ts, and others that the PR never touches.
Why this matters
This is not cosmetic. The failure is worded as a specific accusation against the PR under review, at high severity, with a plausible count. The likely responses are all bad:
- An author "fixes" alerts they did not introduce, expanding an unrelated PR.
- A reviewer treats the red as blocking and holds a clean PR.
- Everyone learns the CodeQL gate is noise and stops reading it — which is how a genuinely new alert gets waved through.
(3) is the expensive one. A gate that cries wolf at high severity on every PR is worse than no gate, because it trains the exact reflex it exists to prevent.
Likely cause
The PR analysis appears to have no base analysis to diff against, so the full open-alert set at the head is reported as the delta. Worth checking whether CodeQL is running on push to main often enough to keep a current baseline database, or whether the main analysis has been failing/skipped — note main currently carries 505 open alerts, so the baseline exists as alerts even where the comparison is not happening.
Suggested fix
- Confirm CodeQL runs and completes on
push: main, so PR runs have a baseline to diff.
- Until the diff is trustworthy, consider not making
CodeQL a required/blocking status, or scoping the gate to alerts whose location falls inside the PR's changed paths.
Separately: the baseline itself
505 open alerts on main, overwhelmingly js/missing-rate-limiting, is its own decision to make — either those routes get limiters or the rule gets a documented repo-level dismissal. Right now it is neither, and the volume is what makes the misattribution above so easy to miss. Filing that observation here rather than as a second issue, since the fix for this one will surface it.
Found while gating #1405 (review), where the red gate was the single most alarming signal on an otherwise clean PR.
Symptom
The
CodeQLgate check fails on PRs with a title of the form:Observed on #1405 at
04f43384. The alerts span ~21 backend route files. That PR changes exactly one route file (backend/routes/auth.ts), and every route it adds is rate-limited.The attribution is false
Measured on #1405, comparing distinct (rule, file) pairs:
refs/pull/1405/headrefs/heads/maincomm -23 <pr_pairs> <main_pairs>returns empty — there is no (rule, file) pair on the PR head that is not already open on main. Every alert the gate calls "new in code changed by this pull request" is pre-existing repo baseline, dominated byjs/missing-rate-limitingacrosspods.ts,skills.ts,admin/globalIntegrations.ts,admin/users.ts,integrations.ts, and others that the PR never touches.Why this matters
This is not cosmetic. The failure is worded as a specific accusation against the PR under review, at high severity, with a plausible count. The likely responses are all bad:
(3) is the expensive one. A gate that cries wolf at high severity on every PR is worse than no gate, because it trains the exact reflex it exists to prevent.
Likely cause
The PR analysis appears to have no base analysis to diff against, so the full open-alert set at the head is reported as the delta. Worth checking whether CodeQL is running on
pushtomainoften enough to keep a current baseline database, or whether themainanalysis has been failing/skipped — note main currently carries 505 open alerts, so the baseline exists as alerts even where the comparison is not happening.Suggested fix
push: main, so PR runs have a baseline to diff.CodeQLa required/blocking status, or scoping the gate to alerts whose location falls inside the PR's changed paths.Separately: the baseline itself
505 open alerts on
main, overwhelminglyjs/missing-rate-limiting, is its own decision to make — either those routes get limiters or the rule gets a documented repo-level dismissal. Right now it is neither, and the volume is what makes the misattribution above so easy to miss. Filing that observation here rather than as a second issue, since the fix for this one will surface it.Found while gating #1405 (review), where the red gate was the single most alarming signal on an otherwise clean PR.