Thanks to visit codestin.com
Credit goes to github.com

Skip to content

docs: address AI Gateway documentation review feedback #28309

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pawbana merged 9 commits into from
Aug 23, 2026
Merged

docs: address AI Gateway documentation review feedback #28309

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
31cbaf4
docs: address AI Gateway documentation review feedback
pawbana Aug 19, 2026
8976b6e
Update docs/ai-coder/ai-gateway/ai-gateway-proxy/setup.md
pawbana Aug 23, 2026
6e41fca
Update docs/ai-coder/ai-gateway/clients/copilot.md
pawbana Aug 23, 2026
e540982
Update docs/ai-coder/ai-gateway/clients/copilot.md
pawbana Aug 23, 2026
cec1348
Update docs/ai-coder/ai-gateway/clients/copilot.md
pawbana Aug 23, 2026
34af859
Update docs/ai-coder/ai-gateway/reference.md
pawbana Aug 23, 2026
d3da65d
Update docs/ai-coder/ai-gateway/standalone.md
pawbana Aug 23, 2026
cf831c0
Update docs/ai-coder/ai-gateway/standalone.md
pawbana Aug 23, 2026
e24267c
Update docs/ai-coder/ai-gateway/standalone.md
pawbana Aug 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions docs/admin/infrastructure/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,17 +133,18 @@ AI-powered development within Coder workspaces.

### AI Gateway

AI Gateway is a centralized gateway that sits between coding agents and LLM providers such
as OpenAI and Anthropic. Users authenticate through Coder instead of managing separate
provider API keys. All prompts, token usage, and tool invocations are recorded
for compliance and cost tracking.
AI Gateway is a centralized feature that sits between coding agents and LLM providers such as OpenAI and Anthropic.
Users authenticate through Coder instead of managing separate provider API keys.
All prompts, token usage, and tool invocations are recorded for compliance and cost tracking.

AI Gateway supports 2 deployment topologies:

- **Embedded:** `coderd` runs the AI Gateway data plane in the same process.
- **Standalone:** AI Gateway runs outside `coderd`, as replicas that serve AI traffic and send requests directly to upstream providers.
- **Standalone:** gateway replicas run outside `coderd`, serve AI traffic, and send requests directly to upstream providers.

Standalone replicas hold no durable state. `coderd` is the source of truth and the only component that writes AI Gateway state to the database.
Standalone replicas have no authoritative database state.
They keep ephemeral provider snapshots, caches, and metrics in memory, and can optionally write API dumps to local disk.
`coderd` is the source of truth and the only component that writes AI Gateway state to the database.
Each replica maintains a control connection to `coderd` for Coder API key validation, provider configuration, and AI session recording, and becomes unready when that connection is unavailable.

Refer to [AI Gateway](../../ai-coder/ai-gateway/index.md) and [standalone deployment](../../ai-coder/ai-gateway/standalone.md) for configuration and operational guidance.
Expand Down
11 changes: 4 additions & 7 deletions docs/admin/infrastructure/validated-architectures/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,13 +125,10 @@ offline use.

### AI Gateway

[AI Gateway](../../../ai-coder/ai-gateway/index.md) proxies AI provider traffic
and records each AI session. It runs inside `coderd` by default, and can also
run as a [standalone deployment](../../../ai-coder/ai-gateway/standalone.md)
that scales independently of the control plane. Size replicas from your own AI
request volume and `CODER_AI_GATEWAY_MAX_CONCURRENCY`. For the chart's resource
requests and autoscaling defaults, refer to the
[AI Gateway Helm chart README](https://github.com/coder/coder/blob/main/helm/ai-gateway/README.md).
[AI Gateway](../../../ai-coder/ai-gateway/index.md) proxies AI provider traffic and records each AI session.
It runs inside `coderd` by default and can also run as a [standalone deployment](../../../ai-coder/ai-gateway/standalone.md) that scales independently of the control plane.
Size replicas based on your AI request volume and `CODER_AI_GATEWAY_MAX_CONCURRENCY`.
For the chart's resource requests and autoscaling defaults, refer to the [AI Gateway Helm chart README](../../../../helm/ai-gateway/README.md).

## Kubernetes Infrastructure

Expand Down
2 changes: 1 addition & 1 deletion docs/admin/integrations/multiple-kube-clusters.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ coder:
readOnly: true
```

[Upgrade Coder](../../install/kubernetes.md#upgrading-coder-via-helm) with these
[Upgrade Coder](../../install/kubernetes.md#upgrade-coder-via-helm) with these
new values.

### VM control plane
Expand Down
9 changes: 3 additions & 6 deletions docs/admin/integrations/prometheus.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,12 +72,9 @@ scrape_configs:
apps: "coder"
```

If you run a [standalone AI Gateway](../../ai-coder/ai-gateway/standalone.md),
each replica exports its own metrics on its own listener. Its Helm chart uses the
same `0.0.0.0:2112` default as the `coder` chart, but sets up no scrape
discovery. Refer to
[AI Gateway monitoring](../../ai-coder/ai-gateway/monitoring.md#kubernetes-discovery)
for more details.
If you run a [standalone AI gateway](../../ai-coder/ai-gateway/standalone.md), each replica exports its own metrics on its own listener.
Its Helm chart uses the same `0.0.0.0:2112` default as the `coder` chart, but sets up no scrape discovery.
Refer to [AI gateway monitoring](../../ai-coder/ai-gateway/monitoring.md#kubernetes-discovery) for more details.

To use the Kubernetes Prometheus operator to scrape metrics, you will need to
create a `ServiceMonitor` in your Coder deployment namespace. The following is
Expand Down
20 changes: 10 additions & 10 deletions docs/ai-coder/ai-gateway/ai-gateway-proxy/setup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ Intercepted requests are forwarded to the AI Gateway, configured via [`CODER_AI_
By default, this is the embedded AI Gateway at `<coderd-access-url>/api/v2/ai-gateway`, and no configuration is needed.

AI Gateway Proxy remains part of the `coder server` process when you [deploy AI Gateway as a standalone service](../standalone.md).
To forward intercepted requests to the standalone Gateway, set:
To forward intercepted requests to the standalone gateway, set the following:

```sh
CODER_AI_GATEWAY_PROXY_TARGET=https://ai-gateway.example.com/
Expand All @@ -73,7 +73,7 @@ For additional configuration options, see the [Coder server configuration](../..
> The AI Gateway Proxy should only be accessible within a trusted network and **must not** be directly exposed to the public internet.
> Without proper network restrictions, unauthorized users could route traffic through the proxy or intercept credentials.

### Encrypting client connections
### Encrypt client connections

By default, AI tools send the Coder session token in the proxy credentials over unencrypted HTTP.
This only applies to the initial connection between the client and the proxy.
Expand All @@ -89,7 +89,7 @@ See [Proxy TLS Configuration](#proxy-tls-configuration) for configuration steps.
* Internal network only: If the proxy and all clients are on the same trusted network, credentials are not exposed to external attackers.
* TLS-terminating load balancer: Place a TLS-terminating load balancer in front of the proxy that terminates TLS and forwards requests over HTTP.

### Restricting proxy access
### Restrict proxy access

Requests to non-allowlisted domains are tunneled through the proxy, but connections to private and reserved IP ranges are blocked by default.
The IP validation and TCP connect happen atomically, preventing DNS rebinding attacks where the resolved address could change between the check and the connection.
Expand Down Expand Up @@ -151,7 +151,7 @@ CODER_AI_GATEWAY_PROXY_CERT_FILE=/path/to/intermediate-ca.crt
CODER_AI_GATEWAY_PROXY_KEY_FILE=/path/to/intermediate-ca.key
```

### Securing the private key
### Secure the private key

> [!WARNING]
> The CA private key is used to sign certificates for MITM interception.
Expand All @@ -162,7 +162,7 @@ Best practices:
* Restrict file permissions so only the Coder process can read the key.
* Use a secrets manager to store the key where possible.

### Distributing the certificate
### Distribute the certificate

AI tools need to trust the CA certificate before connecting through the proxy.

Expand Down Expand Up @@ -235,7 +235,7 @@ The certificate must include a SAN matching the proxy's hostname or IP address.

If clients already trust your organization's root CA, no additional certificate configuration is needed for the TLS connection to the proxy.

### Trusting the TLS certificate
### Trust the TLS certificate

For **self-signed certificates**, AI tools must be configured to trust the TLS certificate.

Expand Down Expand Up @@ -290,7 +290,7 @@ To use AI Gateway Proxy, AI tools must be configured to:
1. Route traffic through the proxy
1. Trust the proxy's CA certificate

### Configuring the proxy
### Configure the proxy

The preferred approach is to configure the proxy directly in the AI tool's settings, as this avoids routing unnecessary traffic through the proxy.
Consult the tool's documentation for specific instructions.
Expand All @@ -311,7 +311,7 @@ Note: if [TLS is not enabled](#proxy-tls-configuration) on the proxy, replace `h

In order for AI tools that communicate with AI Gateway Proxy to authenticate with Coder via AI Gateway, the Coder session token needs to be passed in the proxy credentials as the password field.

### Trusting the CA certificate
### Trust the CA certificate

The preferred approach is to configure the CA certificate directly in the AI tool's settings, as this limits the scope of the trusted certificate to that specific application.
Consult the tool's documentation for specific instructions.
Expand Down Expand Up @@ -418,7 +418,7 @@ x509: certificate signed by unknown authority
```

it has not been configured to trust the proxy's
MITM CA certificate. See [Trusting the CA certificate](#trusting-the-ca-certificate). If
MITM CA certificate. Visit [Trust the CA certificate](#trust-the-ca-certificate). If
[TLS is enabled on the listener](#proxy-tls-configuration), the tool must trust that certificate as well.

### Requests are not being intercepted
Expand Down Expand Up @@ -465,4 +465,4 @@ WARN blocking connection to private/reserved IP hostname=... port=... resolved
```

To allow specific internal networks, set
[`CODER_AI_GATEWAY_PROXY_ALLOWED_PRIVATE_CIDRS`](#restricting-proxy-access).
[`CODER_AI_GATEWAY_PROXY_ALLOWED_PRIVATE_CIDRS`](#restrict-proxy-access).
53 changes: 22 additions & 31 deletions docs/ai-coder/ai-gateway/auth.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,28 +7,24 @@
AI Gateway uses different credentials for different kinds of connections:

- AI clients use a Coder API token to authenticate with AI Gateway as a user.
- Standalone Gateway replicas use AI Gateway keys to connect to the Coder control plane.
- Standalone gateway replicas use AI Gateway keys to connect to the Coder control plane.
- AI Gateway uses provider credentials configured by an administrator to authenticate to upstream AI providers.
- In Bring Your Own Key (BYOK) mode, a user also supplies a personal provider credential or subscription token.
- In [Bring Your Own Key (BYOK)](#bring-your-own-key-byok) mode, a user also supplies a personal provider credential or subscription token.

These credentials are not interchangeable.
A Gateway key does not authenticate an AI client, and a Coder API token does not authenticate a standalone replica.
A gateway key does not authenticate an AI client, and a Coder API token does not authenticate a standalone replica.

## Authenticate AI clients

AI Gateway authenticates clients with the same Coder API token
that a user uses for the rest of the Coder API.
AI Gateway authenticates clients with the same Coder API token that a user uses for the rest of the Coder API.
No separate AI Gateway login is required for client traffic.
For token creation, expiration, and revocation, refer to [Sessions and API tokens](../../admin/users/sessions-tokens.md).

Authenticating with a Coder token avoids distributing centralized provider API keys,
such as OpenAI or Anthropic keys, to individual users.
AI Gateway handles upstream credentials centrally and
forwards each request to the configured provider on the user's behalf.
Authenticating with a Coder token avoids distributing centralized provider API keys, such as OpenAI or Anthropic keys, to individual users.
AI Gateway handles upstream credentials centrally and forwards each request to the configured provider on the user's behalf.

The exact environment variable or setting name differs between tools.
Refer to the list of [supported clients](./clients/index.md) and
your tool's documentation for details.
Refer to the list of [supported clients](./clients/index.md) and your tool's documentation for details.

### Create a Coder API token

Expand All @@ -54,8 +50,7 @@ Use short lifetimes for automation and CI to limit the blast radius if a token l

### Retrieve your session token

If you're logged in with the Coder CLI, retrieve your current session token
with [`coder login token`](../../reference/cli/login_token.md):
If you're logged in with the Coder CLI, retrieve your current session token with [`coder login token`](../../reference/cli/login_token.md):

```sh
export ANTHROPIC_API_KEY=$(coder login token)
Expand All @@ -75,11 +70,11 @@ export HTTPS_PROXY="https://coder:$(coder login token)@<proxy-host>:8888"
The client machine also needs to trust the proxy's CA certificate.
For full setup, refer to [AI Gateway Proxy setup](./ai-gateway-proxy/setup.md).

## Authenticate standalone Gateway replicas
## Authenticate standalone gateway replicas

AI Gateway keys are scoped to the Coder deployment.
A [standalone AI Gateway](./standalone.md) uses one of these keys to connect to `coderd`.
Only the built-in Owner role can create, list, and delete these keys.
A [standalone AI gateway](./standalone.md) uses one of these keys to connect to `coderd`.
Only the built-in **owner** role can create, list, and delete these keys.
Coder custom roles are organization-scoped and cannot grant the site-level `ai_gateway_key` permissions.

Create a key with a descriptive name:
Expand All @@ -89,7 +84,7 @@ coder ai-gateway keys create standalone-production
```

The command displays the plaintext key once.
Save it immediately in your secret manager because Coder cannot retrieve it later.
Save it immediately in your secret manager, because you cannot retrieve it later.
Coder stores only a short prefix of the key for display and a SHA-256 hash for authentication, never the full secret.

Names must be unique, 64 characters or fewer, and use only lowercase letters, numbers, and hyphens.
Expand All @@ -102,7 +97,7 @@ Configure the standalone process with either of the following options, but not b
- `CODER_AI_GATEWAY_KEY_FILE` or `--key-file` reads the key from a file.

A user login and `CODER_SESSION_TOKEN` are not used by `coder ai-gateway start`.
The same Gateway key can authenticate multiple replicas.
The same gateway key can authenticate multiple replicas.
Separate keys make it easier to rotate or revoke each deployment independently.

List keys and the most recent heartbeat for each:
Expand All @@ -115,14 +110,14 @@ A replica records a heartbeat when its control connection is established, then r
The heartbeat reports control-connection liveness rather than client request volume.
Coder stores one timestamp per key, so replicas that share a key cannot be distinguished.

For usage and flags, refer to the generated CLI reference for [creating](../../reference/cli/ai-gateway_keys_create.md), [listing](../../reference/cli/ai-gateway_keys_list.md), and [deleting](../../reference/cli/ai-gateway_keys_delete.md) Gateway keys.
For usage and flags, refer to the generated CLI reference for [creating](../../reference/cli/ai-gateway_keys_create.md), [listing](../../reference/cli/ai-gateway_keys_list.md), and [deleting](../../reference/cli/ai-gateway_keys_delete.md) gateway keys.

### Rotate a Gateway key
### Rotate a gateway key

Rotate a key with a rolling restart.
Run more than 1 replica behind a load balancer so client traffic continues during the rollout:

1. Create a new Gateway key.
1. Create a new gateway key.
1. Update the Kubernetes Secret, environment variable, or key file used by every replica.
1. Restart or roll out the standalone deployment so every replica uses the new key.
1. Verify readiness and confirm that the new key has a recent heartbeat.
Expand Down Expand Up @@ -160,8 +155,7 @@ while AI Gateway continues to provide observability and governance.
In BYOK mode, users need two credentials:

- A Coder API token to authenticate with AI Gateway.
- Their own LLM credential, such as a personal API key or subscription token,
which AI Gateway forwards to the upstream provider.
- Their own LLM credential, such as a personal API key or subscription token, which AI Gateway forwards to the upstream provider.

BYOK and centralized modes can be used together.
When a user provides their own credential, AI Gateway forwards it directly.
Expand All @@ -173,26 +167,23 @@ while allowing individual users to bring their own key.
> When a BYOK credential is present, [key failover](./providers.md#key-failover)
> is skipped.

Coder Agents requests routed through AI Gateway are in-process control plane
requests, not external client requests that send their own AI Gateway bearer token.
Coder Agents requests routed through AI Gateway are in-process control plane requests, not external client requests that send their own AI Gateway bearer token.
Coder Agents use the same global BYOK setting.
When BYOK is enabled, users can save personal API keys for any enabled AI provider
from the Agents settings page.
Refer to [Agents credential selection](../agents/models.md#credential-selection)
for the Agents-specific behavior.
When BYOK is enabled, users can save personal API keys for any enabled AI provider from the Agents settings page.
Refer to [Agents credential selection](../agents/models.md#credential-selection) for the Agents-specific behavior.

Visit individual [client pages](./clients/index.md) for configuration details.

### Enable or disable BYOK

BYOK is enabled by default.
Administrators can disable it for the embedded Gateway with `--ai-gateway-allow-byok=false` or `CODER_AI_GATEWAY_ALLOW_BYOK=false`:
Administrators can disable it for the embedded gateway with `--ai-gateway-allow-byok=false` or `CODER_AI_GATEWAY_ALLOW_BYOK=false`:

```sh
coder server --ai-gateway-allow-byok=false
```

For a standalone Gateway, set the option on each replica:
For a standalone gateway, set the option on each replica:

```sh
CODER_AI_GATEWAY_ALLOW_BYOK=false coder ai-gateway start
Expand Down
Loading
Loading