feat: log rate-limited external auth token validation - #26754
Conversation
c5b1cd3 to
b2c420f
Compare
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
17610a0 to
8c94582
Compare
|
/coder-agents-review |
|
Chat: Review posted | View chat Review history
deep-review v0.9.0 | Round 1 | Last posted: Round 1, 7 findings (3 P2, 1 P3, 2 Nit, 1 Note), COMMENT. Review Finding inventoryFindings
Round logRound 1Panel (16 reviewers). 3 P2, 1 P3, 2 Nit, 1 Note posted. 3 dropped. Reviewed against 14a6104..8c94582. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
Clean observability addition. The shared xhttp.IsGitHubRateLimited predicate prevents metric/validation divergence by construction, the test helper builds through ConvertConfig matching production wiring, and negative-path coverage is thorough. 70.8% test density on an observability-only change.
3 P2, 1 P3, 2 Nit, 1 Note. The P2s are documentation accuracy: the commit message references promoauth.IsRateLimited (the function is xhttp.IsGitHubRateLimited), the Logger field doc overstates its scope, and NewGithub's doc omits the new counter. The P3 is an observability gap: the metric interceptor only fires for GitHub providers, but ValidateToken calls IsGitHubRateLimited for all provider types, so log and metric diverge for non-GitHub configs.
As Bisky put it: "Oh, this is a nice set. Genuinely nice."
coderd/promoauth/oauth2.go:174
P2 [CRF-3] NewGithub now also wires the externalRequestRateLimited counter via its interceptor, but the doc still says "It tracks rate limits as well as just the external request counts." The new rate-limited response counter is omitted.
Suggest: // NewGithub returns an instrumented config for GitHub, tracking rate limits, rate-limited response counts, and request totals.
(Gon)
🤖
🤖 This review was automatically generated with Coder Agents.
| cfg := f.New(name, under) | ||
| cfg.interceptors = append(cfg.interceptors, func(resp *http.Response, err error) { | ||
| cfg.interceptors = append(cfg.interceptors, func(source Oauth2Source, resp *http.Response, err error) { | ||
| if xhttp.IsGitHubRateLimited(resp) { |
There was a problem hiding this comment.
P2 [CRF-1] The commit message body says "The detection moves into promoauth as the exported IsRateLimited." The PR description says "The rate-limit detection is unified as promoauth.IsRateLimited." The actual function is xhttp.IsGitHubRateLimited in coderd/util/xhttp. The name is different, the package is different. A reader searching for promoauth.IsRateLimited will find nothing.
The commit subject also lists labels as {name,source} but the metric has {name,source,status_code}.
The behavioral claim is accurate: ValidateToken routes through the instrumented client, and the function is shared between the interceptor and validation. The description of what the code does is correct; the description of where the code lives is wrong. Commit messages are permanent records.
(Mafu-san P2, Leorio)
🤖
6ea4b66 to
6205719
Compare
6205719 to
16f1506
Compare
16f1506 to
6f7b41f
Compare
ReviewOne P2 and two P3s. No blockers. P2 — The PR description says "the valid/invalid decision is unchanged," but the new shared coder/coderd/util/xhttp/xhttp.go Lines 15 to 29 in d4ad8f6 That feeds directly into coder/coderd/externalauth/externalauth.go Lines 488 to 505 in d4ad8f6 Confirmed by the new test table itself ( P3 — coder/coderd/promoauth/oauth2.go Lines 174 to 176 in d4ad8f6 Also flagged in the earlier coder/coderd/promoauth/oauth2.go Lines 297 to 321 in d4ad8f6 P3 — pre-1.25
coder/coderd/externalauth/externalauth_internal_test.go Lines 70 to 81 in d4ad8f6 🤖 Generated with Claude Code |
BobbyHo
left a comment
There was a problem hiding this comment.
The overall changes LGTM. I left a couple of nit comments and shared an analysis from an AI agent. I believe those findings are also non-blocking, but they may be worth considering before merging the PR.
| @@ -560,6 +537,57 @@ func (c *Config) ValidateToken(ctx context.Context, link *oauth2.Token) (bool, * | |||
| return true, user, nil | |||
| } | |||
|
|
|||
| // rateLimitLogInterval is the minimum time between rate-limited validation | |||
| // warnings emitted per Config. | |||
| const rateLimitLogInterval = time.Minute | |||
There was a problem hiding this comment.
Nit: I wonder if we should use time.Minute as the default while allowing users to override it with another value.
| @@ -96,6 +102,16 @@ func NewFactory(registry prometheus.Registerer) *Factory { | |||
| "source", | |||
| "status_code", | |||
| }), | |||
| externalRequestRateLimited: factory.NewCounterVec(prometheus.CounterOpts{ | |||
| Namespace: "coderd", | |||
There was a problem hiding this comment.
NIT: It might be worth defining constants for the namespace and subsystem values, such as "coderd" and "oauth2", so we don’t have to repeat these strings across multiple blocks.
When ValidateToken keeps a token because the validation endpoint was
rate-limited (a 403 with rate-limit headers or a 429), it returned
valid=true silently, so operators could not distinguish a
provider-confirmed token from one kept optimistically during a rate
limit.
Add a Logger to externalauth.Config and emit a Warn (with provider_id,
provider_type, status_code, reason) on the rate-limit branches, threading
a logger through ConvertConfig. The unambiguous outcomes remain visible
via coderd_oauth2_external_requests_total{source="ValidateToken",
status_code}. Observability-only: the valid/invalid decision is unchanged.
Add coderd_oauth2_external_requests_rate_limited_total{name,source,status_code},
incremented in the instrumented round tripper when a response is
rate-limited (a 429, or a 403 with rate-limit headers). Detection lives in
the shared xhttp.IsRateLimited (coderd/util/xhttp), used by both the tripper
and externalauth.ValidateToken so the metric and the validation decision
share one definition. No externalauth wiring is needed: ValidateToken
already routes through the instrumented client with source=ValidateToken.
3294766 to
8ee8bd1
Compare
When
ValidateTokenkeeps a token because the external auth validation endpoint was rate-limited (a403with rate-limit headers or a429), it returnsvalid=truewithout provider confirmation. Previously this happened silently, so operators couldn't tell a provider-confirmed token from one kept optimistically during a rate limit.This adds a
Loggertoexternalauth.Configand emits aWarn(withprovider_id,provider_type,status_code, andreason) on those rate-limit branches. It also adds acoderd_oauth2_external_requests_rate_limited_total{name, source, status_code}counter, incremented in the instrumented round tripper whenever a provider returns a rate-limited response. The rate-limit detection is the sharedxhttp.IsRateLimited(incoderd/util/xhttp), used by both the tripper andValidateTokenso the metric and the validation decision share one definition; no extra wiring is needed sinceValidateTokenalready routes through the instrumented client withsource="ValidateToken".One deliberate behavioral change rides along: rate-limit detection now also recognizes the unprefixed
RateLimit-Remainingheader (GitLab, and the IETF draft rate-limit headers), so a403withRateLimit-Remaining: 0is treated as optimistically valid where it was previously treated as revoked. All other valid/invalid decisions are unchanged.TestValidateTokenasserts the warning's fields on the rate-limited cases and no warning for revocations,401, and confirmed responses;promoauthandxhttptests cover the detector and the new counter.Manual testing
The signals fire on the external-auth status check (
GET /api/v2/external-auth/{id}), which callsValidateToken. To force a rate-limited response, point a provider'svalidate_urlat a mock that returns the rate-limit shape:429on one path and403+X-RateLimit-Remaining: 0on another.coder serverwith--prometheus-enableand external auth providers whosevalidate_urlpoint at those mock paths (e.g.CODER_EXTERNAL_AUTH_0_VALIDATE_URL=http://127.0.0.1:5599/429).external_auth_linkswith a futureoauth_expiry(token contents are irrelevant; the mock rejects regardless).curlthe status endpoint with a session token, then check:Warn(reason=status_codefor429,reason=rate_limit_headersfor403),coderd_oauth2_external_requests_rate_limited_total{...,status_code="429"|"403"}.Notes:
scripts/testidp -429only rate-limits/oauth2/userinfo, not the/external-auth-validate/...path, so it does not exercise this; use a mockvalidate_url. The default Prometheus port2112may already be taken on dogfood workspaces, setCODER_PROMETHEUS_ADDRESSto a free port.🤖 Generated with the help of Coder Agents on behalf of @jscottmiller.