Codestin Search App

pawbana · 2026-06-23T10:48:13Z

AI Tools were used to produce this PR

This PR adds coder ai-gateway start command that runs the AI Gateway as an independent process.

Standalone process doesn't have access to DB. Uses DRPC services under /api/v2/ai-gateway/servefor auth, recording and provider initialization.
It only handles LLM traffic, other endpoints (eg. /sessions) are only available though coderd.
The standalone gateway reuses applicable flags from AI Gateway deployment options. Provider-seeding and coderd-only options are excluded.
Only added to fat build, the slim build stub rejects the command.

Some wiring used by this new command is added.

NewWebsocketDialer - implements the standalone gateway's connection to coderd's /api/v2/ai-gateway/serve endpoint. It upgrades to a WebSocket, multiplexes with yamux, and wires all DRPC services.

AIGatewayDataPlaneMiddleware - extracts the per-request middleware chain (concurrency limiting, rate limiting, BYOK gating) into a shared function used by both the embedded route and the standalone gateway.

RootCmd.ResolveClientConnection - resolve the deployment URL and builds an HTTP transport without requiring a session token. Used in ai-gateway startcommand as it authenticates using different credential type.

linear-code · 2026-06-23T10:48:17Z

AIGOV-315

pawbana · 2026-06-23T10:48:27Z

feat: add ai-gateway start command #26605 : 3 dependent PRs (#26746 , #26797 , #26988 ) 👈 (View in Graphite)
Revert "chore: hide AI Gateway key management UI/CLI/API (#26879)" #26913
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2026-06-23T10:48:28Z

Docs preview

📖 View docs preview for docs/reference/cli/ai-gateway.md

coderagents · 2026-07-06T07:51:35Z

Documentation Check

This PR adds coder ai-gateway start, a new standalone deployment mode for AI Gateway. The auto-generated CLI reference (docs/reference/cli/ai-gateway_start.md) is included, but the hand-written docs need updates to reflect this new deployment option.

Updates Needed

docs/ai-coder/ai-gateway/setup.md - The opening paragraph states AI Gateway "runs inside the Coder control plane (coderd), requiring no separate compute to deploy or scale." This is no longer the only option. Add a section documenting standalone mode setup: the coder ai-gateway start command, required flags (--key/CODER_AI_GATEWAY_KEY, --url/CODER_URL), optional TLS configuration, and the listening address.
docs/ai-coder/ai-gateway/reference.md - The Implementation Details section says "In future releases we will support running external instances for higher throughput and complete memory isolation from coderd." This future is now delivered; update to document the standalone architecture (WebSocket DRPC connection to coderd, separate HTTP listener for LLM traffic).
docs/ai-coder/ai-gateway/auth.md - Document AI Gateway key authentication for standalone mode. Currently only covers user-facing Coder API tokens. The standalone gateway authenticates to coderd using AI Gateway keys (--key/CODER_AI_GATEWAY_KEY), which is a distinct auth flow from user tokens.
docs/admin/infrastructure/architecture.md - If it references AI Gateway topology, update to mention the standalone replica option.

Automated review via Coder Agents

ssncferreira · 2026-07-06T17:20:31Z

+			if errors.As(err, &sdkErr) {
+				switch sdkErr.StatusCode() {
+				// These statuses are returned by the /api/v2/ai-gateway/serve (wrong Gateway key or incompatible API versions)
+				// or FeatureAIBridge check.


So these status codes can only happen when calling the /api/v2/ai-gateway/serve endpoint (including the feature check), right?. What happens if the URL is valid, but incorrect, we would get a 404 here, that would be considered a transient error and retried forever, right? 🤔

Yes, those sdkErr is only returned by /serve endpoint. Changed comment so it is clearer.
Yes, if URL is unreachable or returns 404 standalone will retry forever to connect.

ssncferreira · 2026-07-06T17:35:23Z

 	resp, err := client.IsAuthorized(ctx, authReq)
 	if err != nil {
-		logger.Warn(ctx, "key authorization check failed", slog.Error(err))
+		logger.Warn(ctx, "key authorization check failed", slog.F("error", err.Error()))


Why this change? 👀

I added this to shorten the logs emitted. Each time wrong coder token is used by client connecting to AI Gateway error + stack will be logged:

2026-07-07 12:57:15.230 [warn] aibridged: key authorization check failed method=POST path=/openai/v1/responses source="" auth_mode=centralized auth_delegated=false ... error= invalid key storj.io/drpc/drpcwire.UnmarshalError:26 storj.io/drpc/drpcstream.(*Stream).HandlePacket:230 storj.io/drpc/drpcmanager.(*Manager).manageReader:250

With this change only 1 line is logged:

2026-07-07 12:56:02.513 [warn] aibridged: key authorization check failed method=POST path=/openai/v1/responses source="" auth_mode=centralized auth_delegated=false error="invalid key"

Reverted to original to keep pattern used in other places but maybe some throttling would be nice here.

ssncferreira · 2026-07-06T17:52:48Z

+				}
+			}
+
+			shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), shutdownTimeout)


Is the shutdownTimeout=15s enough? 🤔 IIUC this means that if for instance, ai-gateway receives a SIGTERM, it stops accepting new requests, waits up to 15 seconds for in-flight requests to finish. Any request still running at 15s gets its connection cut and the client sees a broken stream mid-response.

15 seconds feels short for some LLM requests, would it make sense to increase this?

Changed to 5 min. Maybe a bit much but for standalone maybe it is fine to have long timeout.

NewWebsocketDialer connects "coder ai-gateway start" to coderd's /api/v2/ai-gateway/serve endpoint over a yamux-multiplexed WebSocket. It lives with its sole consumer (aigatewaystart.go) on this branch.

The standalone gateway no longer builds providers from CODER_AI_GATEWAY_* env config; it fetches the provider set from coderd over DRPC (GetAIProviders). Update the command doc comment and --help text to match, and regenerate the help golden and CLI reference docs.

Both the standalone gateway dialer (coderd/aibridged/dialer.go) and the serve handler (enterprise/coderd/aibridgeserve.go) hardcoded the "version" query parameter. Extract it to aibridgedproto.VersionQueryParam so the two ends of the AI Gateway serve handshake share a single definition.

…erve response

This was referenced Jun 23, 2026

feat(coderd/database): add AI Gateway key auth lookup and last-used queries #26505

Merged

feat: add /api/v2/aibridge/serve endpoint #26506

Merged

pawbana mentioned this pull request Jun 23, 2026

feat(enterprise/coderd): add /api/v2/aibridge/serve endpoint #26507

Closed

github-actions Bot assigned pawbana Jun 23, 2026

pawbana force-pushed the pawel/aigov-308-implement-apiv2aibridgeserve-endpoint-3 branch from a66f1fe to c768ef1 Compare June 23, 2026 11:35

pawbana force-pushed the pawel/aigov-315-implement-basic-coder-aibridge-start-sub-command branch from d2902a8 to a0e84e0 Compare June 23, 2026 11:35

pawbana force-pushed the pawel/aigov-308-implement-apiv2aibridgeserve-endpoint-3 branch from c768ef1 to 8a51ad7 Compare June 23, 2026 13:46

pawbana force-pushed the pawel/aigov-315-implement-basic-coder-aibridge-start-sub-command branch from a0e84e0 to 0814404 Compare June 23, 2026 13:46

pawbana force-pushed the pawel/aigov-308-implement-apiv2aibridgeserve-endpoint-3 branch from 8a51ad7 to 94f6f0d Compare June 23, 2026 14:11

pawbana force-pushed the pawel/aigov-315-implement-basic-coder-aibridge-start-sub-command branch from 0814404 to 06972b6 Compare June 23, 2026 14:11

pawbana force-pushed the pawel/aigov-308-implement-apiv2aibridgeserve-endpoint-3 branch from 94f6f0d to 0d518d8 Compare June 23, 2026 17:00

pawbana force-pushed the pawel/aigov-315-implement-basic-coder-aibridge-start-sub-command branch 2 times, most recently from ce1531b to cfecff6 Compare June 23, 2026 17:01

pawbana force-pushed the pawel/aigov-308-implement-apiv2aibridgeserve-endpoint-3 branch from 0d518d8 to 3ca7c1b Compare June 23, 2026 17:01

pawbana changed the base branch from pawel/aigov-308-implement-apiv2aibridgeserve-endpoint-3 to graphite-base/26605 June 23, 2026 17:19

pawbana changed the base branch from graphite-base/26605 to pawel/aigov-308-implement-apiv2aibridgeserve-endpoint-2 June 23, 2026 17:19

pawbana force-pushed the pawel/aigov-308-implement-apiv2aibridgeserve-endpoint-2 branch from 3ca7c1b to ea4601e Compare June 24, 2026 17:38

pawbana force-pushed the pawel/aigov-315-implement-basic-coder-aibridge-start-sub-command branch from cfecff6 to d54cd8d Compare June 24, 2026 17:38

pawbana force-pushed the pawel/aigov-308-implement-apiv2aibridgeserve-endpoint-2 branch from ea4601e to d1181b3 Compare June 24, 2026 17:44

pawbana force-pushed the pawel/aigov-315-implement-basic-coder-aibridge-start-sub-command branch 2 times, most recently from 334315a to 9973338 Compare June 25, 2026 09:44

pawbana force-pushed the pawel/aigov-308-implement-apiv2aibridgeserve-endpoint-2 branch from 4350b16 to 2138d0d Compare June 25, 2026 12:16

pawbana force-pushed the pawel/aigov-315-implement-basic-coder-aibridge-start-sub-command branch 2 times, most recently from 06af6f1 to 83efd04 Compare June 25, 2026 12:22

pawbana force-pushed the pawel/aigov-308-implement-apiv2aibridgeserve-endpoint-2 branch from 2138d0d to 754b2bd Compare June 25, 2026 12:22

dannykopping changed the base branch from pawel/aigov-308-implement-apiv2aibridgeserve-endpoint-2 to graphite-base/26605 June 25, 2026 12:34

dannykopping force-pushed the graphite-base/26605 branch from 754b2bd to 1ae96fc Compare June 25, 2026 12:34

pawbana marked this pull request as ready for review July 6, 2026 07:48

pawbana mentioned this pull request Jul 6, 2026

feat: add /healtz and /readyz endpoints to standalone Gateway #26988

Merged

ssncferreira reviewed Jul 6, 2026

View reviewed changes

dannykopping reviewed Jul 7, 2026

View reviewed changes

Comment thread coderd/aibridged/aibridged.go

Comment thread coderd/aibridged/aibridged.go Outdated

Comment thread enterprise/cli/aigatewaystart.go Outdated

dannykopping and others added 20 commits July 7, 2026 14:39

feat(enterprise/cli): add 'coder ai-gateway start' standalone gateway

405f3de

feat(coderd/aibridged): add websocket dialer for the standalone gateway

f785e69

NewWebsocketDialer connects "coder ai-gateway start" to coderd's /api/v2/ai-gateway/serve endpoint over a yamux-multiplexed WebSocket. It lives with its sole consumer (aigatewaystart.go) on this branch.

dialer + /serve endpoint test changes

15d666b

remove unused options

526b69e

Add TestResolveClientConnection test

6f0b310

agentic review 1: fixed [CRF-1]

926fe97

agentic review 1: [CRF-4] + [CRF-11]

5e3929c

agentic review: fix [CRF-2]

7b18c55

agentic review 1: fix [CRF-3]

2b4bd56

agentic review 1: [CRF-5]

ebe6f80

agentic review 1: rest

2df573a

flake fix

5c1c433

test cleanup

1708f0f

agentic review 2: fix [CRF-15]

9e42925

agentic review 2: rest

a9a4a5d

cosmetic changes

92e91a4

review 1: small stuff, test cases, logs

99a3ca8

review 1: add --key-file flag + remove 'try logging in' error from /s…

2d2bb0c

…erve response

pawbana force-pushed the pawel/aigov-315-implement-basic-coder-aibridge-start-sub-command branch from a76de94 to 2d2bb0c Compare July 7, 2026 14:42

pawbana requested review from dannykopping and ssncferreira July 7, 2026 14:49

pawbana mentioned this pull request Jul 7, 2026

feat: setup logging, tracing and metrics in standalone AI Gateway #27068

Merged

dannykopping approved these changes Jul 7, 2026

View reviewed changes

Uh oh!

Conversation

pawbana commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linear-code Bot commented Jun 23, 2026

Uh oh!

pawbana commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 23, 2026

Docs preview

Uh oh!

coderagents Bot commented Jul 6, 2026

Documentation Check

Updates Needed

Uh oh!

Uh oh!

Uh oh!

ssncferreira Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

pawbana Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ssncferreira Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

pawbana Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

ssncferreira Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

pawbana Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pawbana commented Jun 23, 2026 •

edited

Loading

pawbana commented Jun 23, 2026 •

edited

Loading

pawbana Jul 7, 2026 •

edited

Loading