feat(aibridge): resolve bedrock inference profiles on first request - #28996
Closed
evgeniy-scherbina wants to merge 18 commits into
Closed
feat(aibridge): resolve bedrock inference profiles on first request#28996evgeniy-scherbina wants to merge 18 commits into
evgeniy-scherbina wants to merge 18 commits into
Conversation
Application inference profile ARNs are opaque, so Bedrock capability detection matched nothing and adaptive-thinking conversion shaped requests for the wrong model. Resolve the ARN through GetInferenceProfile and use the underlying model ID for capabilities, usage records, pricing, and metrics, while still invoking the profile so AWS attributes spend. Resolution only runs for application inference profile ARNs, so deployments configured with plain model IDs need no extra permission.
…solved Drop the empty-value fallback so the runtime is always constructed with resolved model IDs, and name the accessors after what they return rather than how they are used.
…dpoint Drop the injected resolver in favor of the mock-endpoint pattern the Bedrock credential tests already use, so the AWS client, response decoding, and error wrapping are exercised. Building the client from the loaded AWS config also honors custom control-plane endpoints.
…edrockCredentials Inference profile resolution reused the credentials but reloaded the AWS config and overwrote its credentials. Return the config that was already loaded so the control-plane client shares one identity and one set of environment-derived settings.
…ctly The injected resolver existed only for tests, which now drive resolution through a mock Bedrock endpoint. Asserting that no request reaches Bedrock is also a stronger statement than asserting a stub was unused.
…terceptor constructors
Contributor
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
Base automatically changed from
yevhenii/aigov-488-support-bedrock-application-inference-profile-arns-with
to
main
September 8, 2026 01:01
Contributor
Author
|
Closing unmerged: superseded by the write-time resolution design. Resolution is moving to coderd, at provider create/update, with the result persisted and passed to the gateway in the provider payload. That removes the AWS call from provider construction entirely, so it fixes CRF-11 and CRF-12 at the root rather than by moving the call to the request path, and it keeps resolution in one place instead of two. Created by Coder Agents on behalf of @evgeniy-scherbina. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #28877, stacked on its branch. Review that PR first.
#28877 resolves Bedrock application inference profile (AIP) ARNs while constructing a provider. Review found two consequences of putting an AWS call on that path:
GetInferenceProfileor STS failure drops the provider from the rebuilt snapshot. Reloads are event driven only, so the provider stays gone until a provider CRUD event or a restart, and requests get404 route not supportedrather than the 503 disabled sentinel.coder serverstartup undercontext.Background(), so boot blocks up to 30s per AIP-configured provider on unreachable egress, and SIGINT does not interrupt it.Both follow from resolution being a construction-time step. This moves it to the first request that needs it.
NewAnthropicnow makes no network call: it keeps the configured Bedrock settings, the loadedaws.Config, and a sharedInferenceProfileCache.CreateInterceptorresolves through that cache using the request context, then builds the per-requestBedrockRuntime.Model()stays infallible and the sharedInterceptorinterface is unchanged, because resolution completes before the interceptor exists.The cache holds successes for the process lifetime, keyed by ARN. Bedrock has no
UpdateInferenceProfile, so a profile's underlying model is fixed at creation and repointing requires a new ARN, which means no invalidation logic. Failures are not cached, so a transient one is retried by the next request.singleflightcollapses a burst of concurrent first requests into one AWS lookup. The cache is created once per reloader and passed into provider construction, so it survivesReplaceProvidersinstead of being discarded on every reload.Net effect: boot never waits on AWS, reloads never call AWS, a deployment resolves each profile once per process, and a resolution failure costs one request instead of the provider.
The trade is that a permission misconfiguration now surfaces as a failed request rather than at startup, and
provider_info{status="error"}no longer flags it.Implementation plan
Why change the current design
The PR resolves application inference profile (AIP) ARNs during provider
construction. Review found two P1 consequences:
GetInferenceProfileor STS failure drops the Bedrockprovider from the snapshot. Reloads are event driven only, so the provider
stays gone until a provider CRUD event or a restart, and requests get
404 route not supportedrather than the 503 disabled sentinel.coder serverstartup withcontext.Background(), so boot blocks up to 30s per AIP provider onunreachable egress and SIGINT does not interrupt it.
A process cache alone fixes neither: the cache is cold at boot, and it only
helps after a first success.
Resolving on the first request removes AWS from both the boot path and the
reload path. Construction becomes pure again, so a resolution failure can only
affect the request that triggers it, and the next request retries.
Design
Resolution seam
Anthropic.CreateInterceptor(w, r, tracer) (intercept.Interceptor, error)isper request, already returns an error, and has
r.Context(). Resolutionhappens there, before the interceptor is built, so:
interceptionBase.Model()stays infallible.intercept.Interceptorinterface is unchanged.bridge.goalready maps aCreateInterceptorerror to a logged 500.Cache
Resolve(ctx, awsCfg, identifier) (string, error).Bedrock has no
UpdateInferenceProfile: an AIP's underlying model is fixed atcreation, and repointing requires a new ARN.
singleflightcollapses concurrent cold-cache resolutions of the same ARNinto one AWS call.
The cache must outlive provider instances, since
ReplaceProvidersswaps themon every reload. It is created once in
poolRPCReloaderand passed throughBuildProvidersFromProtoandbuildProviderintoNewAnthropic, rather thanbeing a package-level global, so tests get their own.
Timeout and context
Resolution derives from
r.Context(), so it is cancelable and bounded by therequest deadline, with
inferenceProfileResolutionTimeoutas an upper bound.Boot no longer waits on AWS at all, so the uncancelable
context.Background()problem disappears from this path.
Trade-offs accepted
This reverses the earlier "fail at construction" decision, which review showed
to be silent deletion plus a 404 rather than a loud failure.
provider_info{status="error"}no longer signals AIP problems. Resolutionfailures are logged per request; a dedicated metric is a possible follow-up.
Out of scope
Relates to https://linear.app/codercom/issue/AIGOV-488
Created by Coder Agents on behalf of @evgeniy-scherbina.