From 31cbaf410e042e52de6ebac8d149ffed0cced13b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Banaszewski?= Date: Wed, 19 Aug 2026 15:47:29 +0000 Subject: [PATCH 1/9] docs: address AI Gateway documentation review feedback Apply the documentation review feedback from #27794, which was merged as a backport before the comments could be addressed. --- docs/admin/infrastructure/architecture.md | 13 +-- .../validated-architectures/index.md | 11 +- .../integrations/multiple-kube-clusters.md | 2 +- docs/admin/integrations/prometheus.md | 9 +- .../ai-gateway/ai-gateway-proxy/setup.md | 20 ++-- docs/ai-coder/ai-gateway/auth.md | 53 ++++------ docs/ai-coder/ai-gateway/clients/copilot.md | 12 +-- docs/ai-coder/ai-gateway/clients/index.md | 10 +- docs/ai-coder/ai-gateway/index.md | 4 +- docs/ai-coder/ai-gateway/mcp.md | 10 +- docs/ai-coder/ai-gateway/monitoring.md | 41 +++---- docs/ai-coder/ai-gateway/providers.md | 25 ++--- docs/ai-coder/ai-gateway/reference.md | 38 +++---- docs/ai-coder/ai-gateway/setup.md | 11 +- docs/ai-coder/ai-gateway/standalone.md | 100 +++++++++--------- docs/install/kubernetes.md | 12 +-- docs/install/upgrade.md | 2 +- 17 files changed, 179 insertions(+), 194 deletions(-) diff --git a/docs/admin/infrastructure/architecture.md b/docs/admin/infrastructure/architecture.md index 79b709a6e91cd..25e94c4fdf012 100644 --- a/docs/admin/infrastructure/architecture.md +++ b/docs/admin/infrastructure/architecture.md @@ -133,17 +133,18 @@ AI-powered development within Coder workspaces. ### AI Gateway -AI Gateway is a centralized gateway that sits between coding agents and LLM providers such -as OpenAI and Anthropic. Users authenticate through Coder instead of managing separate -provider API keys. All prompts, token usage, and tool invocations are recorded -for compliance and cost tracking. +AI Gateway is a centralized feature that sits between coding agents and LLM providers such as OpenAI and Anthropic. +Users authenticate through Coder instead of managing separate provider API keys. +All prompts, token usage, and tool invocations are recorded for compliance and cost tracking. AI Gateway supports 2 deployment topologies: - **Embedded:** `coderd` runs the AI Gateway data plane in the same process. -- **Standalone:** AI Gateway runs outside `coderd`, as replicas that serve AI traffic and send requests directly to upstream providers. +- **Standalone:** gateway replicas run outside `coderd`, serve AI traffic, and send requests directly to upstream providers. -Standalone replicas hold no durable state. `coderd` is the source of truth and the only component that writes AI Gateway state to the database. +Standalone replicas have no authoritative database state. +They keep ephemeral provider snapshots, caches, and metrics in memory, and can optionally write API dumps to local disk. +`coderd` is the source of truth and the only component that writes AI Gateway state to the database. Each replica maintains a control connection to `coderd` for Coder API key validation, provider configuration, and AI session recording, and becomes unready when that connection is unavailable. Refer to [AI Gateway](../../ai-coder/ai-gateway/index.md) and [standalone deployment](../../ai-coder/ai-gateway/standalone.md) for configuration and operational guidance. diff --git a/docs/admin/infrastructure/validated-architectures/index.md b/docs/admin/infrastructure/validated-architectures/index.md index 8c037a643376d..a9e6d241ce100 100644 --- a/docs/admin/infrastructure/validated-architectures/index.md +++ b/docs/admin/infrastructure/validated-architectures/index.md @@ -125,13 +125,10 @@ offline use. ### AI Gateway -[AI Gateway](../../../ai-coder/ai-gateway/index.md) proxies AI provider traffic -and records each AI session. It runs inside `coderd` by default, and can also -run as a [standalone deployment](../../../ai-coder/ai-gateway/standalone.md) -that scales independently of the control plane. Size replicas from your own AI -request volume and `CODER_AI_GATEWAY_MAX_CONCURRENCY`. For the chart's resource -requests and autoscaling defaults, refer to the -[AI Gateway Helm chart README](https://github.com/coder/coder/blob/main/helm/ai-gateway/README.md). +[AI Gateway](../../../ai-coder/ai-gateway/index.md) proxies AI provider traffic and records each AI session. +It runs inside `coderd` by default and can also run as a [standalone deployment](../../../ai-coder/ai-gateway/standalone.md) that scales independently of the control plane. +Size replicas based on your AI request volume and `CODER_AI_GATEWAY_MAX_CONCURRENCY`. +For the chart's resource requests and autoscaling defaults, refer to the [AI Gateway Helm chart README](../../../../helm/ai-gateway/README.md). ## Kubernetes Infrastructure diff --git a/docs/admin/integrations/multiple-kube-clusters.md b/docs/admin/integrations/multiple-kube-clusters.md index 3bbea893f9639..e177d1b5c16b4 100644 --- a/docs/admin/integrations/multiple-kube-clusters.md +++ b/docs/admin/integrations/multiple-kube-clusters.md @@ -46,7 +46,7 @@ coder: readOnly: true ``` -[Upgrade Coder](../../install/kubernetes.md#upgrading-coder-via-helm) with these +[Upgrade Coder](../../install/kubernetes.md#upgrade-coder-via-helm) with these new values. ### VM control plane diff --git a/docs/admin/integrations/prometheus.md b/docs/admin/integrations/prometheus.md index bb03362fc4689..d057e6254f762 100644 --- a/docs/admin/integrations/prometheus.md +++ b/docs/admin/integrations/prometheus.md @@ -72,12 +72,9 @@ scrape_configs: apps: "coder" ``` -If you run a [standalone AI Gateway](../../ai-coder/ai-gateway/standalone.md), -each replica exports its own metrics on its own listener. Its Helm chart uses the -same `0.0.0.0:2112` default as the `coder` chart, but sets up no scrape -discovery. Refer to -[AI Gateway monitoring](../../ai-coder/ai-gateway/monitoring.md#kubernetes-discovery) -for more details. +If you run a [standalone AI gateway](../../ai-coder/ai-gateway/standalone.md), each replica exports its own metrics on its own listener. +Its Helm chart uses the same `0.0.0.0:2112` default as the `coder` chart, but sets up no scrape discovery. +Refer to [AI gateway monitoring](../../ai-coder/ai-gateway/monitoring.md#kubernetes-discovery) for more details. To use the Kubernetes Prometheus operator to scrape metrics, you will need to create a `ServiceMonitor` in your Coder deployment namespace. The following is diff --git a/docs/ai-coder/ai-gateway/ai-gateway-proxy/setup.md b/docs/ai-coder/ai-gateway/ai-gateway-proxy/setup.md index 0fd17985c5530..b32282499affc 100644 --- a/docs/ai-coder/ai-gateway/ai-gateway-proxy/setup.md +++ b/docs/ai-coder/ai-gateway/ai-gateway-proxy/setup.md @@ -55,7 +55,7 @@ Intercepted requests are forwarded to the AI Gateway, configured via [`CODER_AI_ By default, this is the embedded AI Gateway at `/api/v2/ai-gateway`, and no configuration is needed. AI Gateway Proxy remains part of the `coder server` process when you [deploy AI Gateway as a standalone service](../standalone.md). -To forward intercepted requests to the standalone Gateway, set: +To forward intercepted requests to the standalone gateway, set the following: ```sh CODER_AI_GATEWAY_PROXY_TARGET=https://ai-gateway.example.com/ @@ -73,7 +73,7 @@ For additional configuration options, see the [Coder server configuration](../.. > The AI Gateway Proxy should only be accessible within a trusted network and **must not** be directly exposed to the public internet. > Without proper network restrictions, unauthorized users could route traffic through the proxy or intercept credentials. -### Encrypting client connections +### Encrypt client connections By default, AI tools send the Coder session token in the proxy credentials over unencrypted HTTP. This only applies to the initial connection between the client and the proxy. @@ -89,7 +89,7 @@ See [Proxy TLS Configuration](#proxy-tls-configuration) for configuration steps. * Internal network only: If the proxy and all clients are on the same trusted network, credentials are not exposed to external attackers. * TLS-terminating load balancer: Place a TLS-terminating load balancer in front of the proxy that terminates TLS and forwards requests over HTTP. -### Restricting proxy access +### Restrict proxy access Requests to non-allowlisted domains are tunneled through the proxy, but connections to private and reserved IP ranges are blocked by default. The IP validation and TCP connect happen atomically, preventing DNS rebinding attacks where the resolved address could change between the check and the connection. @@ -151,7 +151,7 @@ CODER_AI_GATEWAY_PROXY_CERT_FILE=/path/to/intermediate-ca.crt CODER_AI_GATEWAY_PROXY_KEY_FILE=/path/to/intermediate-ca.key ``` -### Securing the private key +### Secure the private key > [!WARNING] > The CA private key is used to sign certificates for MITM interception. @@ -162,7 +162,7 @@ Best practices: * Restrict file permissions so only the Coder process can read the key. * Use a secrets manager to store the key where possible. -### Distributing the certificate +### Distribute the certificate AI tools need to trust the CA certificate before connecting through the proxy. @@ -235,7 +235,7 @@ The certificate must include a SAN matching the proxy's hostname or IP address. If clients already trust your organization's root CA, no additional certificate configuration is needed for the TLS connection to the proxy. -### Trusting the TLS certificate +### Trust the TLS certificate For **self-signed certificates**, AI tools must be configured to trust the TLS certificate. @@ -290,7 +290,7 @@ To use AI Gateway Proxy, AI tools must be configured to: 1. Route traffic through the proxy 1. Trust the proxy's CA certificate -### Configuring the proxy +### Configure the proxy The preferred approach is to configure the proxy directly in the AI tool's settings, as this avoids routing unnecessary traffic through the proxy. Consult the tool's documentation for specific instructions. @@ -311,7 +311,7 @@ Note: if [TLS is not enabled](#proxy-tls-configuration) on the proxy, replace `h In order for AI tools that communicate with AI Gateway Proxy to authenticate with Coder via AI Gateway, the Coder session token needs to be passed in the proxy credentials as the password field. -### Trusting the CA certificate +### Trust the CA certificate The preferred approach is to configure the CA certificate directly in the AI tool's settings, as this limits the scope of the trusted certificate to that specific application. Consult the tool's documentation for specific instructions. @@ -418,7 +418,7 @@ x509: certificate signed by unknown authority ``` it has not been configured to trust the proxy's -MITM CA certificate. See [Trusting the CA certificate](#trusting-the-ca-certificate). If +MITM CA certificate. See [Trust the CA certificate](#trust-the-ca-certificate). If [TLS is enabled on the listener](#proxy-tls-configuration), the tool must trust that certificate as well. ### Requests are not being intercepted @@ -465,4 +465,4 @@ WARN blocking connection to private/reserved IP hostname=... port=... resolved ``` To allow specific internal networks, set -[`CODER_AI_GATEWAY_PROXY_ALLOWED_PRIVATE_CIDRS`](#restricting-proxy-access). +[`CODER_AI_GATEWAY_PROXY_ALLOWED_PRIVATE_CIDRS`](#restrict-proxy-access). diff --git a/docs/ai-coder/ai-gateway/auth.md b/docs/ai-coder/ai-gateway/auth.md index 1cab40e8aabb5..d8bec4cfa65e4 100644 --- a/docs/ai-coder/ai-gateway/auth.md +++ b/docs/ai-coder/ai-gateway/auth.md @@ -7,28 +7,24 @@ AI Gateway uses different credentials for different kinds of connections: - AI clients use a Coder API token to authenticate with AI Gateway as a user. -- Standalone Gateway replicas use AI Gateway keys to connect to the Coder control plane. +- Standalone gateway replicas use AI Gateway keys to connect to the Coder control plane. - AI Gateway uses provider credentials configured by an administrator to authenticate to upstream AI providers. -- In Bring Your Own Key (BYOK) mode, a user also supplies a personal provider credential or subscription token. +- In [Bring Your Own Key (BYOK)](#bring-your-own-key-byok) mode, a user also supplies a personal provider credential or subscription token. These credentials are not interchangeable. -A Gateway key does not authenticate an AI client, and a Coder API token does not authenticate a standalone replica. +A gateway key does not authenticate an AI client, and a Coder API token does not authenticate a standalone replica. ## Authenticate AI clients -AI Gateway authenticates clients with the same Coder API token -that a user uses for the rest of the Coder API. +AI Gateway authenticates clients with the same Coder API token that a user uses for the rest of the Coder API. No separate AI Gateway login is required for client traffic. For token creation, expiration, and revocation, refer to [Sessions and API tokens](../../admin/users/sessions-tokens.md). -Authenticating with a Coder token avoids distributing centralized provider API keys, -such as OpenAI or Anthropic keys, to individual users. -AI Gateway handles upstream credentials centrally and -forwards each request to the configured provider on the user's behalf. +Authenticating with a Coder token avoids distributing centralized provider API keys, such as OpenAI or Anthropic keys, to individual users. +AI Gateway handles upstream credentials centrally and forwards each request to the configured provider on the user's behalf. The exact environment variable or setting name differs between tools. -Refer to the list of [supported clients](./clients/index.md) and -your tool's documentation for details. +Refer to the list of [supported clients](./clients/index.md) and your tool's documentation for details. ### Create a Coder API token @@ -54,8 +50,7 @@ Use short lifetimes for automation and CI to limit the blast radius if a token l ### Retrieve your session token -If you're logged in with the Coder CLI, retrieve your current session token -with [`coder login token`](../../reference/cli/login_token.md): +If you're logged in with the Coder CLI, retrieve your current session token with [`coder login token`](../../reference/cli/login_token.md): ```sh export ANTHROPIC_API_KEY=$(coder login token) @@ -75,11 +70,11 @@ export HTTPS_PROXY="https://coder:$(coder login token)@:8888" The client machine also needs to trust the proxy's CA certificate. For full setup, refer to [AI Gateway Proxy setup](./ai-gateway-proxy/setup.md). -## Authenticate standalone Gateway replicas +## Authenticate standalone gateway replicas AI Gateway keys are scoped to the Coder deployment. -A [standalone AI Gateway](./standalone.md) uses one of these keys to connect to `coderd`. -Only the built-in Owner role can create, list, and delete these keys. +A [standalone AI gateway](./standalone.md) uses one of these keys to connect to `coderd`. +Only the built-in **owner** role can create, list, and delete these keys. Coder custom roles are organization-scoped and cannot grant the site-level `ai_gateway_key` permissions. Create a key with a descriptive name: @@ -89,7 +84,7 @@ coder ai-gateway keys create standalone-production ``` The command displays the plaintext key once. -Save it immediately in your secret manager because Coder cannot retrieve it later. +Save it immediately in your secret manager, because you cannot retrieve it later. Coder stores only a short prefix of the key for display and a SHA-256 hash for authentication, never the full secret. Names must be unique, 64 characters or fewer, and use only lowercase letters, numbers, and hyphens. @@ -102,7 +97,7 @@ Configure the standalone process with either of the following options, but not b - `CODER_AI_GATEWAY_KEY_FILE` or `--key-file` reads the key from a file. A user login and `CODER_SESSION_TOKEN` are not used by `coder ai-gateway start`. -The same Gateway key can authenticate multiple replicas. +The same gateway key can authenticate multiple replicas. Separate keys make it easier to rotate or revoke each deployment independently. List keys and the most recent heartbeat for each: @@ -115,14 +110,14 @@ A replica records a heartbeat when its control connection is established, then r The heartbeat reports control-connection liveness rather than client request volume. Coder stores one timestamp per key, so replicas that share a key cannot be distinguished. -For usage and flags, refer to the generated CLI reference for [creating](../../reference/cli/ai-gateway_keys_create.md), [listing](../../reference/cli/ai-gateway_keys_list.md), and [deleting](../../reference/cli/ai-gateway_keys_delete.md) Gateway keys. +For usage and flags, refer to the generated CLI reference for [creating](../../reference/cli/ai-gateway_keys_create.md), [listing](../../reference/cli/ai-gateway_keys_list.md), and [deleting](../../reference/cli/ai-gateway_keys_delete.md) gateway keys. -### Rotate a Gateway key +### Rotate a gateway key Rotate a key with a rolling restart. Run more than 1 replica behind a load balancer so client traffic continues during the rollout: -1. Create a new Gateway key. +1. Create a new gateway key. 1. Update the Kubernetes Secret, environment variable, or key file used by every replica. 1. Restart or roll out the standalone deployment so every replica uses the new key. 1. Verify readiness and confirm that the new key has a recent heartbeat. @@ -160,8 +155,7 @@ while AI Gateway continues to provide observability and governance. In BYOK mode, users need two credentials: - A Coder API token to authenticate with AI Gateway. -- Their own LLM credential, such as a personal API key or subscription token, - which AI Gateway forwards to the upstream provider. +- Their own LLM credential, such as a personal API key or subscription token, which AI Gateway forwards to the upstream provider. BYOK and centralized modes can be used together. When a user provides their own credential, AI Gateway forwards it directly. @@ -173,26 +167,23 @@ while allowing individual users to bring their own key. > When a BYOK credential is present, [key failover](./providers.md#key-failover) > is skipped. -Coder Agents requests routed through AI Gateway are in-process control plane -requests, not external client requests that send their own AI Gateway bearer token. +Coder Agents requests routed through AI Gateway are in-process control plane requests, not external client requests that send their own AI Gateway bearer token. Coder Agents use the same global BYOK setting. -When BYOK is enabled, users can save personal API keys for any enabled AI provider -from the Agents settings page. -Refer to [Agents credential selection](../agents/models.md#credential-selection) -for the Agents-specific behavior. +When BYOK is enabled, users can save personal API keys for any enabled AI provider from the Agents settings page. +Refer to [Agents credential selection](../agents/models.md#credential-selection) for the Agents-specific behavior. Visit individual [client pages](./clients/index.md) for configuration details. ### Enable or disable BYOK BYOK is enabled by default. -Administrators can disable it for the embedded Gateway with `--ai-gateway-allow-byok=false` or `CODER_AI_GATEWAY_ALLOW_BYOK=false`: +Administrators can disable it for the embedded gateway with `--ai-gateway-allow-byok=false` or `CODER_AI_GATEWAY_ALLOW_BYOK=false`: ```sh coder server --ai-gateway-allow-byok=false ``` -For a standalone Gateway, set the option on each replica: +For a standalone gateway, set the option on each replica: ```sh CODER_AI_GATEWAY_ALLOW_BYOK=false coder ai-gateway start diff --git a/docs/ai-coder/ai-gateway/clients/copilot.md b/docs/ai-coder/ai-gateway/clients/copilot.md index 27304fb5ab402..521389c015138 100644 --- a/docs/ai-coder/ai-gateway/clients/copilot.md +++ b/docs/ai-coder/ai-gateway/clients/copilot.md @@ -42,7 +42,7 @@ Copilot CLI is built on Node.js and uses the `NODE_EXTRA_CA_CERTS` environment v export NODE_EXTRA_CA_CERTS="/path/to/coder-ai-gateway-proxy-ca.pem" ``` -See [Client Configuration CA certificate trust](../ai-gateway-proxy/setup.md#trusting-the-ca-certificate) for details on how to obtain the certificate file. +See [Client Configuration CA certificate trust](../ai-gateway-proxy/setup.md#trust-the-ca-certificate) for details on how to obtain the certificate file. When [TLS is enabled](../ai-gateway-proxy/setup.md#proxy-tls-configuration) on the proxy, combine the MITM CA certificate and the TLS certificate into a single file: @@ -62,7 +62,7 @@ For installation instructions, see [Installing the GitHub Copilot extension in V ### Proxy configuration You can configure the proxy using environment variables or VS Code settings. -For environment variables, see [AI Gateway Proxy client configuration](../ai-gateway-proxy/setup.md#configuring-the-proxy). +For environment variables, see [AI Gateway Proxy client configuration](../ai-gateway-proxy/setup.md#configure-the-proxy). Alternatively, you can configure the proxy directly in VS Code settings: @@ -92,11 +92,11 @@ For more details, see [Configuring proxy settings for Copilot](https://docs.gith Add the AI Gateway Proxy CA certificate to your operating system's trust store. By default, VS Code loads system certificates, controlled by the `http.systemCertificates` setting. -See [Client Configuration CA certificate trust](../ai-gateway-proxy/setup.md#trusting-the-ca-certificate) for details on how to obtain the certificate file. +See [Client Configuration CA certificate trust](../ai-gateway-proxy/setup.md#trust-the-ca-certificate) for details on how to obtain the certificate file. When [TLS is enabled](../ai-gateway-proxy/setup.md#proxy-tls-configuration) on the proxy, add the TLS certificate to the system trust store as well. -### Using Coder Remote extension +### Coder Remote extension When connecting to a Coder workspace with the [Coder extension](https://marketplace.visualstudio.com/items?itemName=coder.coder-remote), the Copilot extension runs inside the Coder workspace and not on your local machine. This means proxy and certificate configuration must be done in the Coder workspace environment. @@ -119,7 +119,7 @@ Replace `` with your AI Gateway Proxy hostname and ` [!IMPORTANT] -> The embedded Gateway metric prefix changed from `coder_aibridged_*` to `coder_ai_gateway_*`, and the proxy prefix changed from `coder_aibridgeproxyd_*` to `coder_ai_gateway_proxy_*`. -> The embedded Gateway and AI Gateway Proxy emit the legacy names with identical values during the v2.35 and v2.36 deprecation window, and the legacy names are planned for removal in v2.37. +> The embedded gateway metric prefix changed from `coder_aibridged_*` to `coder_ai_gateway_*`, and the proxy prefix changed from `coder_aibridgeproxyd_*` to `coder_ai_gateway_proxy_*`. +> The embedded gateway and AI Gateway Proxy emit the legacy names with identical values during the v2.35 and v2.36 deprecation window, and the legacy names are planned for removal in v2.37. > The cost control metrics were added after the rename and have no legacy alias. -> The standalone Gateway emits only the current `coder_ai_gateway_*` names. +> The standalone gateway emits only the current `coder_ai_gateway_*` names. > Migrate dashboards and alerts to the new names. > Do not relabel new names back to old names while both are emitted because this creates duplicate legacy series in the same scrape. > After the legacy names are removed, use `metric_relabel_configs` only if you need a temporary compatibility bridge: @@ -123,7 +123,7 @@ Alert when the provider reload loop is firing but failing to refresh the pool fo Use the `coder_ai_gateway_proxy_*` metrics when you alert on AI Gateway Proxy. -## Standalone Gateway monitoring +## Standalone gateway monitoring ### Metrics listener @@ -137,7 +137,7 @@ In addition to the common `coder_ai_gateway_*` metrics, the standalone listener The [AI Gateway Helm chart](../../../helm/ai-gateway/README.md#metrics) enables metrics and binds the listener to `0.0.0.0:2112` by default. The chart exposes a named `metrics` container port, but it does not include this port in the data-plane Service or create monitoring discovery resources. -For Prometheus pod-based discovery, add scrape annotations to each Gateway pod: +For Prometheus pod-based discovery, add scrape annotations to each gateway pod: ```yaml coder: @@ -153,14 +153,15 @@ Refer to the [Helm chart metrics configuration](../../../helm/ai-gateway/README. ### Health and readiness -A standalone AI Gateway exposes health endpoints on its data-plane listener: +A standalone AI gateway exposes health endpoints on its data-plane listener: -| Endpoint | Success condition | -|------------|-----------------------------------------------------------------------------------------------| -| `/healthz` | The HTTP listener is serving. | -| `/readyz` | The control connection to `coderd` is active and provider configuration has been initialized. | +| Endpoint | Success condition | +|------------|----------------------------------------------------------------------------------------------------------| +| `/healthz` | The HTTP listener is serving. | +| `/readyz` | The control connection to `coderd` is active and the initial provider configuration fetch has completed. | -`/readyz` returns HTTP 503 until provider configuration is initialized and whenever the control connection to `coderd` is unavailable. +`/readyz` returns HTTP 503 until the initial provider configuration fetch completes and whenever the control connection to `coderd` is unavailable. +The initial fetch can return an empty provider set. A `200 OK` response from `/healthz` only means the HTTP listener is accepting connections. It returns `200 OK` even when the control connection is down. Both endpoints are unauthenticated, bypass the concurrency, rate limiting, and BYOK middleware, and do not create trace spans. @@ -178,7 +179,7 @@ Refer to the [`coder ai-gateway start` logging options](../../reference/cli/ai-g ### Structured interception logs AI Gateway can emit a structured log for every interception record to an external SIEM or observability platform. -The `CODER_AI_GATEWAY_STRUCTURED_LOGGING` setting belongs to `coderd`, standalone Gateway does not consume it. +The `CODER_AI_GATEWAY_STRUCTURED_LOGGING` setting belongs to `coderd`, and the standalone gateway does not consume it. Standalone replicas send interception records to `coderd`, which writes the structured logs to the Coder server log output. Refer to [structured logging](./setup.md#structured-logging) for configuration and record types. @@ -243,7 +244,7 @@ in the AI Gateway setup guide. ## Tracing AI Gateway supports tracing through [OpenTelemetry](https://opentelemetry.io/) for request processing, upstream API calls, and MCP server interactions. -Embedded Gateway spans are emitted by the `coder server` process. +Embedded gateway spans are emitted by the `coder server` process. Standalone spans are emitted independently by every replica with the service name `coder-ai-gateway`. ### Enable tracing @@ -253,8 +254,8 @@ The exporter always dials without TLS, so an `https://` endpoint is still contac `CODER_TRACE_HONEYCOMB_API_KEY` adds a Honeycomb exporter and works with or without `CODER_TRACE_ENABLE`. Set only the Honeycomb key to export to Honeycomb alone, or set both to export to Honeycomb and an OTLP collector. -The embedded and standalone Gateways share the same tracing options. -Refer to the [`coder server` tracing options](../../reference/cli/server.md#--trace) for the embedded Gateway and the [`coder ai-gateway start` tracing options](../../reference/cli/ai-gateway_start.md#--trace) for standalone replicas. +The embedded and standalone gateways support the same tracing configuration options, but each standalone replica must be configured separately. +Refer to the [`coder server` tracing options](../../reference/cli/server.md#--trace) for the embedded gateway and the [`coder ai-gateway start` tracing options](../../reference/cli/ai-gateway_start.md#--trace) for standalone replicas. Configure tracing on every standalone process or through `coder.env` in the AI Gateway Helm chart. The following minimal configuration enables tracing and exports spans over OTLP/gRPC: @@ -264,7 +265,7 @@ export CODER_TRACE_ENABLE=true export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://otel-collector:4317 ``` -In both deployment modes, each request to the Gateway's LLM API endpoint creates an HTTP request span, including requests that are passed through or rejected instead of intercepted. +In both deployment modes, each request to the gateway's LLM API endpoint creates an HTTP request span, including requests that are passed through or rejected instead of intercepted. ### Traced operations diff --git a/docs/ai-coder/ai-gateway/providers.md b/docs/ai-coder/ai-gateway/providers.md index 8e2535879d369..881587aaf6e88 100644 --- a/docs/ai-coder/ai-gateway/providers.md +++ b/docs/ai-coder/ai-gateway/providers.md @@ -38,9 +38,8 @@ After seeding, manage providers through the dashboard or API. A provider that has been edited or removed there is not recreated or overwritten from the environment on the next restart. -Seeding is a `coderd` operation. A [standalone Gateway](./standalone.md) -ignores the deprecated provider variables and fetches provider -configuration from `coderd`. +Seeding is a `coderd` operation. +A [standalone gateway](./standalone.md) ignores the deprecated provider variables and fetches provider configuration from `coderd`. ## Provider types @@ -146,9 +145,9 @@ AI Gateway resolves AWS credentials one of three ways: that role before calling Bedrock, signing requests with the resulting temporary credentials. This works on top of either of the above base identities and supports cross-account Bedrock access. See - [Assuming an IAM role](#assuming-an-iam-role). + [IAM role assumption](#iam-role-assumption). -#### Obtaining static Bedrock credentials +#### Static Bedrock credentials When you cannot use the default credential chain, create a dedicated IAM user and generate a static access key: @@ -175,7 +174,7 @@ user and generate a static access key: [AI Providers API](../../reference/api/aiproviders.md), along with the region (or base URL) and model identifiers. -#### Assuming an IAM role +#### IAM role assumption Set the optional **Role ARN** field to have the gateway assume an IAM role before calling Bedrock. The base identity (static credentials or the @@ -303,14 +302,12 @@ attempt and each successful reload, exposed as Prometheus metrics: If you run the [external proxy](./ai-gateway-proxy/index.md), it exposes the same pair under the `coder_ai_gateway_proxy_` prefix. -Each [standalone Gateway](./standalone.md) replica reloads providers -independently. - -A growing gap between the attempt and success timestamps means reloads -are firing but failing to apply. Alert on that gap rather than on a -single failure, which may resolve on the next change. See -[Monitoring](./monitoring.md#prometheus-metrics) for the full metric list -and sample alert queries. +Each [standalone gateway](./standalone.md) replica reloads providers independently and replaces only its own provider snapshot. +If a reload fails, that replica retains its previous provider snapshot and continues serving from it. + +A growing gap between the attempt and success timestamps means reloads are firing but failing to apply. +Alert on that gap rather than on a single failure, which may resolve on the next change. +Refer to [Monitoring](./monitoring.md#prometheus-metrics) for the full metric list and sample alert queries. ## Key failover diff --git a/docs/ai-coder/ai-gateway/reference.md b/docs/ai-coder/ai-gateway/reference.md index 68836e01ec61d..19ac1673c9a56 100644 --- a/docs/ai-coder/ai-gateway/reference.md +++ b/docs/ai-coder/ai-gateway/reference.md @@ -7,26 +7,26 @@ ## Deployment topologies AI Gateway can run inside `coderd` or as a standalone data-plane service. -Both topologies run the same Gateway request handling and keep `coderd` as the source of truth for Coder API key validation, provider configuration, and AI session records. -They differ in how requests are routed to the Gateway. +Both topologies run the same gateway request handling and keep `coderd` as the source of truth for Coder API key validation, provider configuration, and AI session records. +They differ in how requests are routed to the gateway. -### Embedded Gateway +### Embedded gateway -By default, `coder server` runs an in-memory Gateway instance in the `coderd` process. +By default, `coder server` runs an in-memory gateway instance in the `coderd` process. AI clients send requests to `/api/v2/ai-gateway//`. -The embedded Gateway uses the same control RPC as a standalone deployment, over an in-process transport rather than a network connection. -It does not use a Gateway key and does not negotiate an API version. +The embedded gateway uses the same control RPC as a standalone deployment, over an in-process transport rather than a network connection. +It does not use a gateway key and does not negotiate an AI Gateway API version. The following diagram shows the embedded topology: ![AI Gateway implementation details](../../images/aibridge/aibridge-implementation-details.png) -### Standalone Gateway +### Standalone gateway A [standalone deployment](./standalone.md) runs the AI traffic data plane outside the `coderd` process. -Each replica accepts client traffic, sends AI requests directly to upstream providers, and maintains a control connection to `coderd` using a [Gateway key](./standalone.md#create-a-gateway-key). +Each replica accepts client traffic, sends AI requests directly to upstream providers, and maintains a control connection to `coderd` by using a [gateway key](./standalone.md#create-a-gateway-key). -The control connection carries: +The control connection carries the following: - Coder API key validation, which resolves each request to an active Coder user. - AI budget checks, which reject requests from users over their spend limit. @@ -34,25 +34,25 @@ The control connection carries: - AI session records. - **Deprecated**: the configuration and access tokens used by [injected MCP](./mcp.md). -Standalone replicas do not own authoritative database state. +Standalone replicas have no authoritative database state. They keep ephemeral provider snapshots, request caches, provider key pools, and metrics in memory, and emit their own logs and traces. -Each replica writes its own [API dumps](./setup.md#api-dumps) to its own local disk when dumps are enabled. +When API dumps are enabled, each replica writes its own [API dumps](./setup.md#api-dumps) to its own local disk. `coderd` remains required for standalone operation. A replica becomes unready when its control connection is unavailable, even if its HTTP listener remains healthy. -AI Gateway Proxy remains part of `coder server` and can forward its intercepted traffic to either the embedded Gateway or a standalone endpoint. +AI Gateway Proxy remains part of `coder server` and can forward its intercepted traffic to either the embedded gateway or a standalone endpoint. ## Version compatibility -The control connection between a standalone replica and `coderd` is versioned. -The current version is defined in [`coderd/aibridged/proto/version.go`](https://github.com/coder/coder/blob/main/coderd/aibridged/proto/version.go). +The control connection between a standalone replica and `coderd` uses a versioned AI Gateway API. +The current AI Gateway API version is defined in [`coderd/aibridged/proto/version.go`](../../../coderd/aibridged/proto/version.go). -`coderd` validates the version that a standalone replica advertises before it accepts the control connection. -Compatibility follows these rules: +`coderd` validates the AI Gateway API version that a standalone replica advertises before it accepts the control connection. +AI Gateway API compatibility follows these rules: -- The Gateway and `coderd` major versions must match. -- The Gateway minor version must be less than or equal to the `coderd` minor version. -- `coderd` rejects a standalone Gateway that advertises a newer minor version. +- The gateway and `coderd` AI Gateway API major versions must match. +- The gateway AI Gateway API minor version must be less than or equal to the `coderd` AI Gateway API minor version. +- `coderd` rejects a standalone gateway that advertises a newer AI Gateway API minor version. A rejected replica receives an HTTP 400 response that reports the `client_api_version` and `server_api_version` values. Coder build versions are not the compatibility criterion. diff --git a/docs/ai-coder/ai-gateway/setup.md b/docs/ai-coder/ai-gateway/setup.md index 91f48c120edec..940f2f8acba9b 100644 --- a/docs/ai-coder/ai-gateway/setup.md +++ b/docs/ai-coder/ai-gateway/setup.md @@ -1,7 +1,7 @@ # Setup By default, AI Gateway runs inside the Coder control plane (`coderd`) and requires no separate compute. -In embedded mode, `coderd` runs the Gateway in memory and brokers traffic to your configured AI providers on behalf of authenticated users. +In embedded mode, `coderd` runs the gateway in memory and brokers traffic to your configured AI providers on behalf of authenticated users. If AI traffic needs dedicated compute, independent scaling, or a separate network endpoint, you can [deploy AI Gateway as a standalone service](./standalone.md). @@ -15,7 +15,7 @@ If AI traffic needs dedicated compute, independent scaling, or a separate networ ## Activation The AI Gateway feature must be enabled in the Coder deployment configuration before -embedded or standalone Gateway instances can serve authenticated traffic. +embedded or standalone gateway instances can serve authenticated traffic. _AI Gateway is enabled by default as of v2.34._ @@ -28,7 +28,7 @@ coder server --ai-gateway-enabled=true A standalone process does not read `CODER_AI_GATEWAY_ENABLED` from its own environment. However, this setting must remain enabled on `coderd`. -It is required for Gateway key management endpoints to work and for standalone replicas to connect to the control plane. +It is required for gateway key management endpoints to work and for standalone replicas to connect to the control plane. ## Configure Providers @@ -91,8 +91,7 @@ with `/var/lib/coder/ai-gateway-dumps` configured writes to Sensitive headers are redacted before dumps are written. Leave the value empty to disable dumping. -Each [standalone Gateway](./standalone.md) replica accepts the same API dump -settings and writes dumps to its own local disk. +Each [standalone gateway](./standalone.md) replica accepts the same API dump settings and writes optional dumps to its own local disk. > [!WARNING] > API dumps are intended for short diagnostic sessions only. Dump files contain @@ -149,7 +148,7 @@ ingestion, set `--log-json` to a file path or `/dev/stderr` so that records are emitted as JSON. This setting belongs to `coderd`. -A [standalone Gateway](./standalone.md) does not consume it. +A [standalone gateway](./standalone.md) does not consume it. Filter for AI Gateway records in your logging pipeline by matching on the `"interception log"` message. Each log line includes a `record_type` field that diff --git a/docs/ai-coder/ai-gateway/standalone.md b/docs/ai-coder/ai-gateway/standalone.md index 22d9283bb0c01..c23b30a6d175d 100644 --- a/docs/ai-coder/ai-gateway/standalone.md +++ b/docs/ai-coder/ai-gateway/standalone.md @@ -6,12 +6,12 @@ When AI traffic needs dedicated compute, independent scaling, or a separate network endpoint, you can deploy AI Gateway separately from the Coder control plane (`coderd`). -A standalone AI Gateway serves client traffic on its own listener and maintains a control connection to `coderd`. +A standalone AI gateway serves client traffic on its own listener and maintains a control connection to `coderd`. `coderd` continues to manage authentication, authorization, provider configuration, and AI session records. ## Before you begin -Standalone AI Gateway requires: +A standalone AI gateway requires the following: - Coder v2.36.0 or later. - A [Premium license with AI Governance](../ai-governance.md). @@ -19,15 +19,15 @@ Standalone AI Gateway requires: - The full Coder image or a Coder binary that includes the `coder ai-gateway start` command. `coder ai-gateway start` does not read `CODER_AI_GATEWAY_ENABLED` from the standalone process. -However, this setting must remain enabled on `coderd` for Gateway key management and standalone control connections. +However, this setting must remain enabled on `coderd` for gateway key management and standalone control connections. -## Create a Gateway key +## Create a gateway key -Each standalone replica uses a Gateway key to authenticate and establish its control connection to `coderd`. -Gateway key management requires site-level `ai_gateway_key` permissions, which only the built-in Owner role includes. +Each standalone replica uses a gateway key to authenticate and establish its control connection to `coderd`. +Gateway key management requires site-level `ai_gateway_key` permissions, which only the built-in **owner** role includes. Coder custom roles are organization-scoped and cannot grant site-level permissions. -Log in to the Coder CLI as an Owner or another user with these permissions, then create a dedicated key for the standalone deployment: +Log in to the Coder CLI as an **owner**, then create a dedicated key for the standalone deployment: ```sh coder login https://coder.example.com @@ -41,24 +41,24 @@ For independent rotation and revocation, use a separate key for each standalone ## Start a standalone process -Set the Coder URL, Gateway key, and listener address, then start the Gateway: +Set the Coder URL, gateway key, and listener address, then start the gateway: ```sh export CODER_URL=https://coder.example.com -export CODER_AI_GATEWAY_KEY='' +export CODER_AI_GATEWAY_KEY='' export CODER_AI_GATEWAY_HTTP_ADDRESS=0.0.0.0:4001 coder ai-gateway start ``` Use `CODER_AI_GATEWAY_KEY_FILE` instead of `CODER_AI_GATEWAY_KEY` to read the key from a file. -The standalone process does not require a user login or `CODER_SESSION_TOKEN` after you provide the Gateway key. +The standalone process does not require a user login or `CODER_SESSION_TOKEN` after you provide the gateway key. The listener defaults to `127.0.0.1:4001`, which accepts connections only from the local host. -Set `CODER_AI_GATEWAY_HTTP_ADDRESS` to a routable address, as shown above, before other hosts or pods can reach the Gateway. +Set `CODER_AI_GATEWAY_HTTP_ADDRESS` to a routable address, as shown above, before other hosts or pods can reach the gateway. -The standalone Gateway fetches provider configuration from `coderd`. -Configure at least one [AI provider](./providers.md) in Coder before sending provider traffic through the Gateway. -The standalone Gateway does not use the deprecated [provider seed variables](./providers.md#database-management-of-providers). +The standalone gateway fetches provider configuration from `coderd`. +Configure at least one [AI provider](./providers.md) in Coder before sending provider traffic through the gateway. +The standalone gateway does not use the deprecated [provider seed variables](./providers.md#database-management-of-providers). The listener uses HTTP by default. Set both `CODER_AI_GATEWAY_TLS_CERT_FILE` and `CODER_AI_GATEWAY_TLS_KEY_FILE` to terminate TLS in the process. @@ -66,20 +66,20 @@ For all command options, refer to [`coder ai-gateway start`](../../reference/cli ## Run AI Gateway in Kubernetes -To run the standalone Gateway as a Kubernetes workload, provide the same environment variables and network access to the Coder URL. +To run the standalone gateway as a Kubernetes workload, provide the same environment variables and network access to the Coder URL. You can manage the workload with your own Kubernetes manifests or use the provided Helm chart. The chart configures the Deployment, probes, and Service, plus an optional Ingress or `HTTPRoute`. To use the Kubernetes examples, install `kubectl` and configure access to the cluster where AI Gateway will run. Install Helm if you use the chart. -Create a namespace and store the Gateway key in a Kubernetes Secret: +Create a namespace and store the gateway key in a Kubernetes Secret: ```sh kubectl create namespace coder-ai-gateway kubectl create secret generic coder-ai-gateway-key \ --namespace coder-ai-gateway \ - --from-literal=key='' + --from-literal=key='' ``` ### Configure the Helm chart @@ -97,7 +97,7 @@ aigateway: name: coder-ai-gateway-key ``` -The Gateway must be able to reach the Coder URL from every replica. +The gateway must be able to reach the Coder URL from every replica. For an HTTPS URL signed by a private CA, set `aigateway.coderTLS.caSecret.name` to the Secret holding the CA bundle. If Coder requires client mTLS, also set `aigateway.coderTLS.clientSecret.name`. @@ -123,7 +123,7 @@ Chart versions omit the leading `v`, so use `2.36.0` rather than `v2.36.0`. When you install the chart directly from a Git checkout, set `coder.image.tag` explicitly and install `./helm/ai-gateway`. Released chart packages select the matching Coder image by default. -If you need to run different `coderd` and Gateway versions, refer to [Version compatibility](#version-compatibility). +If you need to run different `coderd` and gateway versions, refer to [Version compatibility](#version-compatibility). ### Validate the Helm deployment @@ -138,7 +138,7 @@ kubectl rollout status deployment/coder-ai-gateway \ The chart configures the following probes by default: - The liveness probe requests `/healthz` and verifies that the HTTP listener is serving. -- The readiness probe requests `/readyz` and verifies that the Gateway is connected to `coderd` and has completed its initial provider load. +- The readiness probe requests `/readyz` and verifies that the gateway is connected to `coderd` and has completed its initial provider load. - The startup probe is disabled by default. `/healthz` can return HTTP 200 while `/readyz` returns HTTP 503. @@ -165,7 +165,7 @@ curl --fail http://127.0.0.1:4001/readyz Both endpoints return HTTP 200 with an empty body when the replica is serving and ready. -List Gateway keys and verify that the key has a recent heartbeat: +List gateway keys and verify that the key has a recent heartbeat: ```sh coder ai-gateway keys list @@ -177,7 +177,7 @@ An active control connection updates the timestamp every 60 seconds. Coder stores one timestamp per key, so a recent heartbeat on a shared key does not confirm that every replica is connected. Check `/readyz` on each replica to verify individual health. -## Route traffic to the standalone Gateway +## Route traffic to the standalone gateway The standalone deployment does not move traffic automatically. Configure AI Gateway Proxy and direct AI clients to send requests to the standalone endpoint. @@ -205,28 +205,27 @@ The proxy appends the provider name and request path to the target, so do not in ### Direct AI clients -Clients that use `/api/v2/ai-gateway//` continue to reach the embedded Gateway. -To route these clients to the standalone Gateway, replace the Coder access URL and embedded route prefix with the standalone endpoint while keeping the provider path. +Clients that use `/api/v2/ai-gateway//` continue to reach the embedded gateway. +To route these clients to the standalone gateway, replace the Coder access URL and the embedded `/api/v2/ai-gateway` path prefix with the standalone endpoint while keeping the provider path. -For example, if the Gateway is reachable at `https://ai-gateway.example.com` and the providers are configured, configure Claude Code with: +For example, if the gateway is reachable at `https://ai-gateway.example.com` and the providers are configured, configure Claude Code with the following: ```sh ANTHROPIC_BASE_URL=https://ai-gateway.example.com/anthropic ``` -Configure an OpenAI-compatible client with: +Configure an OpenAI-compatible client with the following: ```sh OPENAI_BASE_URL=https://ai-gateway.example.com/openai/v1 ``` -The standalone listener also accepts the equivalent `/api/v2/ai-gateway//` paths for compatibility. For full per-client configuration examples, refer to [Client Configuration](./clients/index.md). -### Expose the standalone AI Gateway +### Expose the standalone AI gateway -If clients cannot access the in-cluster Service, expose the Gateway through an Ingress, an `HTTPRoute`, or a load balancer Service. -In the provided Helm chart, set: +If clients cannot access the in-cluster Service, expose the gateway through an Ingress, an `HTTPRoute`, or a load balancer Service. +In the provided Helm chart, set the following: ```yaml service: @@ -235,16 +234,16 @@ service: Use TLS to protect credentials and AI traffic whenever they cross an untrusted network. Prefer TLS termination at the Ingress or Kubernetes Gateway when your platform already manages certificates there. -To terminate TLS in the AI Gateway process, set `aigateway.listenerTLS.name` to an existing TLS Secret. +To terminate TLS in the gateway process, set `aigateway.listenerTLS.name` to an existing TLS Secret. If `CODER_AI_GATEWAY_PROXY_TARGET` uses HTTPS with a private CA, add that CA to the trust store of the `coderd` pods before changing the proxy target. For the Coder Helm chart, you can mount a CA bundle with `coder.certs.secrets`. -This trust configuration is separate from `aigateway.coderTLS.caSecret`, which configures the connection from the standalone Gateway to `coderd`. +This trust configuration is separate from `aigateway.coderTLS.caSecret`, which configures the connection from the standalone gateway to `coderd`. ## Scale replicas Run multiple replicas behind a Service or load balancer. -In the provided Helm chart, set: +In the provided Helm chart, set the following: ```yaml coder: @@ -261,39 +260,39 @@ Sticky load balancing can improve cache efficiency, but it is not required for c When you enable [API dumps](./setup.md#api-dumps), each replica writes dumps to its own local disk. Use persistent storage if you need those files to survive pod replacement. -The chart requests 1 CPU and 1 GiB of memory per replica and does not set resource limits. +The chart requests 1 CPU and 1 GiB of memory per replica and does not set resource limits. Measure production traffic before changing requests, limits, or `CODER_AI_GATEWAY_MAX_CONCURRENCY`. The chart does not create a Horizontal Pod Autoscaler or PodDisruptionBudget. -## Migrate from the embedded Gateway +## Migrate from the embedded gateway Use a gradual cutover so that you can validate the standalone data plane before moving production traffic: 1. Upgrade `coderd` to the target Coder version. 1. Keep `CODER_AI_GATEWAY_ENABLED=true` on `coderd`. -1. Create a dedicated Gateway key. +1. Create a dedicated gateway key. 1. Deploy one standalone replica without changing client or proxy routing. 1. Verify `/readyz`, the key heartbeat, metrics, logs, and traces. -1. Send a test request directly to the standalone Gateway and confirm that the AI session appears in Coder. +1. Send a test request directly to the standalone gateway and confirm that the AI session appears in Coder. 1. Set `CODER_AI_GATEWAY_PROXY_TARGET` to the standalone endpoint. 1. Update direct client base URLs that should use the standalone endpoint. 1. Scale the standalone deployment after the canary path is stable. Keep `CODER_AI_GATEWAY_ENABLED=true` on `coderd` after the cutover. -The setting is required for standalone control connections and Coder features that use the in-process Gateway. +The setting is required for standalone control connections and Coder features that use the in-process gateway. -## Roll back to the embedded Gateway +## Roll back to the embedded gateway -To return traffic to the embedded Gateway: +To return traffic to the embedded gateway: 1. Remove `CODER_AI_GATEWAY_PROXY_TARGET`, or set it to `/api/v2/ai-gateway`. 1. Restart or upgrade `coderd` so the proxy target change takes effect. 1. Restore direct client base URLs to `/api/v2/ai-gateway//`. 1. Verify requests and AI session records through the embedded route. 1. Scale down or uninstall the standalone deployment. -1. Delete the standalone Gateway key after all replicas have stopped. +1. Delete the standalone gateway key after all replicas have stopped. -Delete the key with: +Delete the key with the following command: ```sh coder ai-gateway keys delete standalone-production @@ -306,20 +305,21 @@ Stop the deployment before deleting its key. ## Version compatibility -The standalone Gateway image does not need to match the `coderd` image version exactly. -The components can connect when their AI Gateway API versions are compatible, and `coderd` checks compatibility whenever a Gateway replica connects. +The standalone gateway image does not need to match the `coderd` image version exactly. +The components can connect when their AI Gateway API versions are compatible, and `coderd` checks AI Gateway API compatibility whenever a gateway replica connects. -A replica may run at the same API version as `coderd` or an earlier minor version of the same major version, but never a newer one. -The Gateway treats certain handshake failures from `/api/v2/ai-gateway/serve` as fatal and exits rather than retrying: +A replica may use the same AI Gateway API version as `coderd` or an earlier minor version of the same major version, but never a newer one. +The gateway treats certain handshake failures from `/api/v2/ai-gateway/serve` as fatal and exits rather than retrying: - HTTP 400: incompatible API version. -- HTTP 401: invalid Gateway key. +- HTTP 401: invalid gateway key. - HTTP 403: missing entitlement. -- HTTP 404: endpoint not found. `coderd` may be too old and not expose `/serve` endpoint. +- HTTP 404: endpoint not found. + `coderd` may be too old and not expose the `/serve` endpoint. Sequence changes that move both components: -- To upgrade, upgrade `coderd` first, then roll out the standalone Gateway release. -- To roll back, roll back the standalone Gateway first, then roll back `coderd`. +- To upgrade, upgrade `coderd` first, then roll out the standalone gateway release. +- To roll back, roll back the standalone gateway first, then roll back `coderd`. Running the same Coder release on `coderd` and every standalone replica is the simplest way to stay compatible, but it is not required. diff --git a/docs/install/kubernetes.md b/docs/install/kubernetes.md index c75a60f54fed5..38252c05f2186 100644 --- a/docs/install/kubernetes.md +++ b/docs/install/kubernetes.md @@ -185,7 +185,7 @@ If you do not have a domain, you should set `CODER_ACCESS_URL` to this URL in the Helm chart and upgrade Coder (see below). This allows workspaces to connect to the proper Coder URL. -## Upgrading Coder via Helm +## Upgrade Coder via Helm To upgrade Coder in the future or change values, you can run the following command: @@ -197,13 +197,11 @@ helm upgrade coder coder-v2/coder \ -f values.yaml ``` -## Standalone AI Gateway Chart +## Standalone AI gateway chart -Coder also publishes a chart, `oci://ghcr.io/coder/chart/coder-ai-gateway`, -that runs [AI Gateway](../ai-coder/ai-gateway/index.md) as its own Deployment -alongside the control plane. Use it when you want to scale AI traffic -independently of `coderd`. For installation and configuration, visit -[Standalone AI Gateway](../ai-coder/ai-gateway/standalone.md). +Coder also publishes a chart, `oci://ghcr.io/coder/chart/coder-ai-gateway`, that runs [AI Gateway](../ai-coder/ai-gateway/index.md) as its own Deployment alongside the control plane. +Use it when you want to scale AI traffic independently of `coderd`. +For installation and configuration, refer to [Deploy AI Gateway as a standalone service](../ai-coder/ai-gateway/standalone.md) and the [AI Gateway Helm chart README](../../helm/ai-gateway/README.md). ## Coder Observability Chart diff --git a/docs/install/upgrade.md b/docs/install/upgrade.md index df17ba0ec25c5..d2ee5294b0ddf 100644 --- a/docs/install/upgrade.md +++ b/docs/install/upgrade.md @@ -46,7 +46,7 @@ docker-compose pull coder && docker-compose up -d coder ### Kubernetes See -[Upgrading Coder via Helm](../install/kubernetes.md#upgrading-coder-via-helm). +[Upgrade Coder via Helm](../install/kubernetes.md#upgrade-coder-via-helm). ### Coder AMI on AWS From 8976b6eaa02afb1eaffc3f3316b2b56dd73c682a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Banaszewski?= Date: Sun, 23 Aug 2026 21:37:18 +0200 Subject: [PATCH 2/9] Update docs/ai-coder/ai-gateway/ai-gateway-proxy/setup.md Co-authored-by: Nick Vigilante --- docs/ai-coder/ai-gateway/ai-gateway-proxy/setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ai-coder/ai-gateway/ai-gateway-proxy/setup.md b/docs/ai-coder/ai-gateway/ai-gateway-proxy/setup.md index b32282499affc..8c6b2a3d0f7d1 100644 --- a/docs/ai-coder/ai-gateway/ai-gateway-proxy/setup.md +++ b/docs/ai-coder/ai-gateway/ai-gateway-proxy/setup.md @@ -418,7 +418,7 @@ x509: certificate signed by unknown authority ``` it has not been configured to trust the proxy's -MITM CA certificate. See [Trust the CA certificate](#trust-the-ca-certificate). If +MITM CA certificate. Visit [Trust the CA certificate](#trust-the-ca-certificate). If [TLS is enabled on the listener](#proxy-tls-configuration), the tool must trust that certificate as well. ### Requests are not being intercepted From 6e41fca020fe8f911e85f0abf99c60dd2f72fff8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Banaszewski?= Date: Sun, 23 Aug 2026 21:37:26 +0200 Subject: [PATCH 3/9] Update docs/ai-coder/ai-gateway/clients/copilot.md Co-authored-by: Nick Vigilante --- docs/ai-coder/ai-gateway/clients/copilot.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ai-coder/ai-gateway/clients/copilot.md b/docs/ai-coder/ai-gateway/clients/copilot.md index 521389c015138..e1f6a41e6b00b 100644 --- a/docs/ai-coder/ai-gateway/clients/copilot.md +++ b/docs/ai-coder/ai-gateway/clients/copilot.md @@ -42,7 +42,7 @@ Copilot CLI is built on Node.js and uses the `NODE_EXTRA_CA_CERTS` environment v export NODE_EXTRA_CA_CERTS="/path/to/coder-ai-gateway-proxy-ca.pem" ``` -See [Client Configuration CA certificate trust](../ai-gateway-proxy/setup.md#trust-the-ca-certificate) for details on how to obtain the certificate file. +Refer to [Client Configuration CA certificate trust](../ai-gateway-proxy/setup.md#trust-the-ca-certificate) for details on how to obtain the certificate file. When [TLS is enabled](../ai-gateway-proxy/setup.md#proxy-tls-configuration) on the proxy, combine the MITM CA certificate and the TLS certificate into a single file: From e5409824afabf64d066853bcd5d63a62b5910b5a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Banaszewski?= Date: Sun, 23 Aug 2026 21:37:33 +0200 Subject: [PATCH 4/9] Update docs/ai-coder/ai-gateway/clients/copilot.md Co-authored-by: Nick Vigilante --- docs/ai-coder/ai-gateway/clients/copilot.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ai-coder/ai-gateway/clients/copilot.md b/docs/ai-coder/ai-gateway/clients/copilot.md index e1f6a41e6b00b..7853863bb307e 100644 --- a/docs/ai-coder/ai-gateway/clients/copilot.md +++ b/docs/ai-coder/ai-gateway/clients/copilot.md @@ -62,7 +62,7 @@ For installation instructions, see [Installing the GitHub Copilot extension in V ### Proxy configuration You can configure the proxy using environment variables or VS Code settings. -For environment variables, see [AI Gateway Proxy client configuration](../ai-gateway-proxy/setup.md#configure-the-proxy). +For environment variables, visit [AI Gateway Proxy client configuration](../ai-gateway-proxy/setup.md#configure-the-proxy). Alternatively, you can configure the proxy directly in VS Code settings: From cec1348bf101648b99f3825bfb320cf82f306aa0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Banaszewski?= Date: Sun, 23 Aug 2026 21:37:44 +0200 Subject: [PATCH 5/9] Update docs/ai-coder/ai-gateway/clients/copilot.md Co-authored-by: Nick Vigilante --- docs/ai-coder/ai-gateway/clients/copilot.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ai-coder/ai-gateway/clients/copilot.md b/docs/ai-coder/ai-gateway/clients/copilot.md index 7853863bb307e..7735e126063c0 100644 --- a/docs/ai-coder/ai-gateway/clients/copilot.md +++ b/docs/ai-coder/ai-gateway/clients/copilot.md @@ -92,7 +92,7 @@ For more details, see [Configuring proxy settings for Copilot](https://docs.gith Add the AI Gateway Proxy CA certificate to your operating system's trust store. By default, VS Code loads system certificates, controlled by the `http.systemCertificates` setting. -See [Client Configuration CA certificate trust](../ai-gateway-proxy/setup.md#trust-the-ca-certificate) for details on how to obtain the certificate file. +Check out [Client Configuration CA certificate trust](../ai-gateway-proxy/setup.md#trust-the-ca-certificate) for details on how to obtain the certificate file. When [TLS is enabled](../ai-gateway-proxy/setup.md#proxy-tls-configuration) on the proxy, add the TLS certificate to the system trust store as well. From 34af859fc0e430e9479d95ebb6e8801ed9a797c8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Banaszewski?= Date: Sun, 23 Aug 2026 21:38:01 +0200 Subject: [PATCH 6/9] Update docs/ai-coder/ai-gateway/reference.md Co-authored-by: Nick Vigilante --- docs/ai-coder/ai-gateway/reference.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ai-coder/ai-gateway/reference.md b/docs/ai-coder/ai-gateway/reference.md index 19ac1673c9a56..119beea7b3b13 100644 --- a/docs/ai-coder/ai-gateway/reference.md +++ b/docs/ai-coder/ai-gateway/reference.md @@ -51,7 +51,7 @@ The current AI Gateway API version is defined in [`coderd/aibridged/proto/versio AI Gateway API compatibility follows these rules: - The gateway and `coderd` AI Gateway API major versions must match. -- The gateway AI Gateway API minor version must be less than or equal to the `coderd` AI Gateway API minor version. +- The gateway's AI Gateway API minor version must be less than or equal to the `coderd` instance's AI Gateway API minor version. - `coderd` rejects a standalone gateway that advertises a newer AI Gateway API minor version. A rejected replica receives an HTTP 400 response that reports the `client_api_version` and `server_api_version` values. From d3da65d9f3f39c99cbd0f395d7a33b5058924036 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Banaszewski?= Date: Sun, 23 Aug 2026 21:38:16 +0200 Subject: [PATCH 7/9] Update docs/ai-coder/ai-gateway/standalone.md Co-authored-by: Nick Vigilante --- docs/ai-coder/ai-gateway/standalone.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ai-coder/ai-gateway/standalone.md b/docs/ai-coder/ai-gateway/standalone.md index c23b30a6d175d..f61dd6255c28f 100644 --- a/docs/ai-coder/ai-gateway/standalone.md +++ b/docs/ai-coder/ai-gateway/standalone.md @@ -45,7 +45,7 @@ Set the Coder URL, gateway key, and listener address, then start the gateway: ```sh export CODER_URL=https://coder.example.com -export CODER_AI_GATEWAY_KEY='' +export CODER_AI_GATEWAY_KEY='' export CODER_AI_GATEWAY_HTTP_ADDRESS=0.0.0.0:4001 coder ai-gateway start ``` From cf831c00b47ea393a4fe88392a4765e91161185a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Banaszewski?= Date: Sun, 23 Aug 2026 21:38:34 +0200 Subject: [PATCH 8/9] Update docs/ai-coder/ai-gateway/standalone.md Co-authored-by: Nick Vigilante --- docs/ai-coder/ai-gateway/standalone.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ai-coder/ai-gateway/standalone.md b/docs/ai-coder/ai-gateway/standalone.md index f61dd6255c28f..0b0038278e87d 100644 --- a/docs/ai-coder/ai-gateway/standalone.md +++ b/docs/ai-coder/ai-gateway/standalone.md @@ -54,7 +54,7 @@ Use `CODER_AI_GATEWAY_KEY_FILE` instead of `CODER_AI_GATEWAY_KEY` to read the ke The standalone process does not require a user login or `CODER_SESSION_TOKEN` after you provide the gateway key. The listener defaults to `127.0.0.1:4001`, which accepts connections only from the local host. -Set `CODER_AI_GATEWAY_HTTP_ADDRESS` to a routable address, as shown above, before other hosts or pods can reach the gateway. +Set `CODER_AI_GATEWAY_HTTP_ADDRESS` to a routable address, as shown previously, before other hosts or pods can reach the gateway. The standalone gateway fetches provider configuration from `coderd`. Configure at least one [AI provider](./providers.md) in Coder before sending provider traffic through the gateway. From e24267c83180260b0961950f7e99e047d6cc3194 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Banaszewski?= Date: Sun, 23 Aug 2026 21:38:41 +0200 Subject: [PATCH 9/9] Update docs/ai-coder/ai-gateway/standalone.md Co-authored-by: Nick Vigilante --- docs/ai-coder/ai-gateway/standalone.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ai-coder/ai-gateway/standalone.md b/docs/ai-coder/ai-gateway/standalone.md index 0b0038278e87d..853485985000b 100644 --- a/docs/ai-coder/ai-gateway/standalone.md +++ b/docs/ai-coder/ai-gateway/standalone.md @@ -79,7 +79,7 @@ Create a namespace and store the gateway key in a Kubernetes Secret: kubectl create namespace coder-ai-gateway kubectl create secret generic coder-ai-gateway-key \ --namespace coder-ai-gateway \ - --from-literal=key='' + --from-literal=key='' ``` ### Configure the Helm chart