feat!: resolve agent external auth by template, not config order - #27854
Conversation
Hostname-only external auth requests, which is what GIT_ASKPASS sends, scanned every provider configured on the deployment and returned the last one whose regex matched the hostname. The requesting workspace's own template-declared provider was never consulted, so reordering CODER_EXTERNAL_AUTH_<N>_* silently changed which OAuth client's token a plain git operation received. Resolve the calling agent's workspace and build before selecting a provider, then narrow the candidates to the providers declared by that build's template version. When exactly one of them matches the hostname, use it regardless of deployment config order. When none of the declared providers match, fall back to the existing deployment-wide scan, so a template that declares only a GitHub provider can still clone an unrelated host. When several declared providers match the same hostname, return 409 naming them rather than picking one arbitrarily: external_auth_providers is stored sorted by ID, so HCL declaration order is unavailable and no principled tie-break exists. Requests supplying an explicit provider ID are unchanged. Refs #23718
Documentation CheckThis PR is primarily a server-side bug fix (Refs #23718 / PLAT-190), but it changes the user-observable behavior of a documented, user-facing feature: external auth provider selection for HTTPS git operations. Documentation for that behavior change landed in commit Updates Needed
A cross-reference was also added to All previously flagged documentation needs are addressed. Automated review via Coder Agents |
geokat
left a comment
There was a problem hiding this comment.
Overall LGTM! 👍
One potential bug (found with the help of an agent):
P2: stale template provider declarations can fall back to a different provider’s token. If a template declares github-read, then an administrator removes that configured provider while github-write still matches github.com, the new helper produces no declared candidate. The caller then performs the deployment-wide fallback and can return or prompt for github-write’s token.
That defeats the branch’s intended template-specific isolation, especially when the remaining token has broader privileges. The existing template-auth requirement path treats a declared but missing provider as an error, so this resolver should do the same. The fallback should apply only when the declared, configured providers simply do not match the Git host.
Add a regression case for: declared provider absent from deployment config + another same-host provider present.
|
Oh, and I agree with the doc review comments above 👍 |
|
Just a heads-up that this fix could potentially be a breaking change for deployments that rely on the old behavior. |
A template declaring a provider the deployment no longer configures produced no declared candidate, so hostname-only resolution fell through to the deployment-wide scan and could return a different same-host provider's token, chosen by config order. Report declared-but-unconfigured provider IDs separately and refuse with a 404 naming them, rather than substituting. The fallback now applies only when every declared provider is configured and none of them match the hostname. A stale declaration for one host still leaves a declared, configured provider for another host resolving normally. Use 404 for both this and the ambiguity error so cli/gitaskpass.go warns and defers to git's own credential handling instead of failing with a raw error, which also keeps behavior correct for agent binaries that predate this change.
Thank you @geokat for the review. This is a good catch—this was a real issue. I reproduced it end to end before fixing it: The template declares github-dotfiles. Fixed in 5547015. The helper now reports declared-but-unconfigured provider IDs separately from matched providers, and the handler refuses the request instead of falling back. The rule is the conjunction in your second sentence: fallback is allowed if and only if every declared provider is configured and none of them match the hostname. With the same workspace, configuration, and command after the fix: |
The external auth docs said Coder picks a provider for HTTPS git operations "based on the repository URL", which no longer describes the behavior. Hostname-only GIT_ASKPASS requests now resolve against the providers the workspace's template declares, falling back to a deployment-wide match only when every declared provider is configured and none of them match the host. Document the resolution order, and the two cases where Coder refuses rather than guess: several declared providers matching one host, and a declared provider the deployment no longer configures. Both include the remedy. Add the template-author-facing half to the extending-templates page so template authors learn that what they declare determines which token native git receives.
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
Agreed, and worth being precise about the scope. Two cases turn a previously-succeeding git operation into a failure, both on the hostname-only (
|
geokat
left a comment
There was a problem hiding this comment.
All good, just one optional non-blocking documentation nit 👍
The prose said a template declaring a provider the deployment no longer configures causes Coder to refuse, without the precondition. A single declared and configured provider matching the host is selected before the missing declaration is considered, so the refusal only applies when none of the declared providers match the host. State the precondition on both pages, and note that a missing declaration leaves hosts served by the template's other declared providers unaffected.
TL;DR
Problem. A template can declare which external auth provider it wants via
data "coder_external_auth" { id = "..." }, and that declaration is honored at every stage of the build. It was ignored at runtime. Any git operation going throughGIT_ASKPASSsupplies only a hostname, never a provider ID, and the handler scanned every provider configured on the deployment and returned whichever matched the hostname last in config order, with no reference to what the requesting workspace's own template declared. ReorderingCODER_EXTERNAL_AUTH_<N>_*silently redirected a plaingit clonefrom one OAuth client's token to a completely different one.Fix. For hostname-only requests, resolve the calling agent's workspace and build before selecting a provider, then narrow candidates to the providers declared by that build's template version. Exactly one match wins regardless of config order. No matching declared provider falls back to today's deployment-wide scan, so a template that declares only a GitHub provider can still clone an unrelated host. Two or more matching declared providers return
409naming them, rather than picking one arbitrarily:external_auth_providersis stored sorted by ID, so HCL declaration order is already unavailable and no principled tie-break exists.Requests supplying an explicit provider ID are untouched. Server-side only: no wire protocol, proto, manifest, or database schema change, so already-running agents get the corrected behavior on their next askpass call with no restart.
Refs #23718
Call flow
flowchart TD subgraph Push["1. Template import: coder templates push"] A1["Terraform extracts coder_external_auth id/optional attrs"] A2["CompleteJob(TemplateImport) validates each id<br/>against deployment config"] A4["template_versions.external_auth_providers persisted"] A1 --> A2 --> A4 end subgraph PreBuild["2. Pre-build and workspace build (unaffected)"] B1["User authenticates declared provider(s), exact-ID lookup"] B2["Build resolves token by exact ID<br/>(provisionerdserver.go)"] A4 --> B1 --> B2 end subgraph Runtime["3. Workspace running: a credential is needed"] B2 --> C0{"Caller supplies id or match?"} C0 -->|"id (explicit)"| D1["Exact-ID match<br/>UNCHANGED, already deterministic<br/>(coder external-auth access-token)"] C0 -->|"match only (GIT_ASKPASS)"| C1["git needs credentials for a hostname<br/>GIT_ASKPASS invoked, unchanged"] C1 --> C2["coder gitaskpass sends ExternalAuthRequest{Match: host}<br/>unchanged (cli/gitaskpass.go)"] C2 --> C3["workspaceAgentsExternalAuth<br/>(coderd/workspaceagents.go)"] C3 --> C4["CHANGED:<br/>1. resolve workspace/build BEFORE matching<br/>2. read that build's declared provider IDs<br/>3. filter: declared AND regex matches host"] C4 --> C5{"how many candidates?"} C5 -->|"exactly 1"| C6["use it, regardless of config order"] C5 -->|"0"| C7["fall back to deployment-wide scan<br/>(unchanged legacy behavior)"] C5 -->|"2 or more"| C8["409 naming every matching ID"] end D1 --> E1["Token returned"] C6 --> E1 C7 --> E1 style C4 fill:#1f4d2e,stroke:#4caf50,color:#fff style C6 fill:#1f4d2e,stroke:#4caf50,color:#fff style C8 fill:#1f4d2e,stroke:#4caf50,color:#fff style D1 fill:#333,stroke:#888,color:#fffVerification
Two test functions were added in
coderd/workspaceagents_test.go, and the behavior no unit test can reach was verified against a local dev cluster with two real GitHub OAuth Apps whose regexes both matchgithub.com.409The last three are properties a unit test cannot express: they involve swapping the server binary underneath a live agent, removing deployment configuration, and rebuilding a workspace against a new template version.
Unit test detail
TestWorkspaceAgentsExternalAuthTemplateScopedbuilds a deployment with two providers sharing a regex, a template declaring one of them, and a seeded token for every provider, so a mis-selection returns a valid token with the wrong identity rather than an error. Subtests:DeclaredProviderLast/DeclaredProviderFirst: the declared provider wins in both config orders. Only theFirstarm is discriminating, since the pre-change loop had nobreakand returned the last regex match, which theLastarm happens to agree with.NoDeclaredProvidersFallsBackToFullScan: a template declaring nothing keeps today's behavior exactly, pinning the legacy last-match rule.UnrelatedHostStillResolvesViaFallback: a template declaring only a GitHub provider still resolves a GitLab host.AmbiguousDeclaredSetReturnsError:409whose message names both colliding provider IDs.OptionalUnauthenticatedDeclaredProviderReturnsAuthURL: returns the auth URL for the declared provider, not for an unrelated one the user happens to hold a token for.TestWorkspaceAgentsExternalAuthMultipleTemplatesruns two workspaces from two templates, each declaring a different provider, issuing requests concurrently. Each resolves to its own template's provider.Manual verification detail
Local dev cluster, two GitHub OAuth Apps both defaulting to
^(https?://)?github\.com(/.*)?$, both authorized by the workspace owner so a wrong selection yields a usable token rather than an error. Workspace built from a template declaring onlygithub-dotfiles. Tokens redacted.Order independence. Same workspace, never rebuilt, config order reversed between runs:
[github-broad, github-dotfiles]gho_<dotfiles>[github-dotfiles, github-broad]gho_<dotfiles>A/B against the pre-fix binary. Everything held constant except the coderd build, with
/api/v2/buildinfochecked on both sides so the comparison rests on verified binary identity. The workspace was never stopped, rebuilt, or re-authorized:v2.35.3-devel+11e03cfb3agho_<broad>v2.35.3-devel+e8b87d0333gho_<dotfiles>This doubles as the demonstration that a coderd-only upgrade corrects behavior on a live agent's next askpass call.
Declared provider removed from config.
github-dotfilesdeleted from deployment configuration while the workspace's template still declared it. Result:HTTP/2 200withgho_<broad>via the fallback. No500, no fail-closed404. The orphanedexternal_auth_linkrow remained in the database throughout and correctly had no effect.Recomputation after a template update.
github-dotfilesgho_<dotfiles>coder updategithub-broadgho_<broad>The pair is what makes it conclusive: the first rules out following the template's newest version, the second rules out a cached value.
Explicit-ID path.
coder external-auth access-token github-broadreturned that provider's result even though the template declared onlygithub-dotfiles, and did not substitute the declared provider's already-valid token.Raw traces were captured with
GIT_CURL_VERBOSE=1 git -c credential.helper="" ls-remote <private repo>, reading the unredacted== Info: Server auth using Basic with user '<token>'line. A private repo is required, since a public one never triggers a401and therefore never invokesGIT_ASKPASS.