Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit eeb2624

Browse files
ethanndicksonethanndickson
authored
fix: pin workspace agent API client to intended agent (#26600)
## Summary The control-plane HTTP client used to talk to workspace agents followed HTTP redirects and trusted the redirected host, letting a malicious workspace agent bounce a coderd request onto a different agent on the shared tailnet. Because the agent HTTP API on port 4 is unauthenticated (it relies on tailnet reachability plus control-plane authorization), this allowed cross-tenant file read/write and remote code execution. This PR refuses redirects and pins every dial to the intended agent. Closes CODAGT-668. ## Problem `agentConn.apiClient` in `codersdk/workspacesdk/agentconn.go` constructed an `http.Client` with no `CheckRedirect`, so Go's default policy followed up to 10 redirects. Its custom `Transport.DialContext` parsed the host from the (post-redirect) request URL and dialed that IP over the shared tailnet, validating only that the port was `AgentHTTPAPIServerPort` (4). It never pinned the connection to the intended `AgentID` / `agentAddress()`. A workspace owner (any regular org member, not just admins) controls their own agent and can make its port-4 handler return a `3xx` `Location` pointing at a victim agent's tailnet IP. When a control-plane action (for example a chat tool or the HTTP MCP server) sends an agent API request to the attacker's agent, coderd acts as a confused deputy and replays the request against the victim: - `301/302/303` rewrite POST to GET, but `307/308` preserve method and body when the body is replayable. The real callers pass replayable bodies, so a redirected `POST /api/v0/write-file` writes attacker-controlled content into the victim workspace and a redirected `POST /api/v0/processes/start` executes it, giving RCE on the victim agent. The dangerous callers run server-side on coderd's single deployment-wide `ServerTailnet`, which is authorized to tunnel to any agent, so the blast radius is cross-tenant / cross-organization (limited in practice to victim agents coderd currently has a live tunnel to). ## Fix In `agentConn.apiClient`: - Set `CheckRedirect: http.ErrUseLastResponse` so the client never follows a redirect. A `3xx` is surfaced to the caller as the response (which the existing `ReadBodyAsError` path turns into an error) instead of being replayed against another host. - Capture the intended agent address once from `AgentID` (`agentAddr := netip.AddrPortFrom(c.agentAddress(), AgentHTTPAPIServerPort)`), reject any dial whose host or port does not match it, and always dial that pinned address rather than the URL-derived host. In `coderd/aitasks.go`, the task app proxy client (`taskAppHTTPClient`) also now sets `CheckRedirect: http.ErrUseLastResponse`. This client dials through `agentConn.DialContext`, which already pins the host to the originating workspace's agent (it takes only the port from the dial address), so it was never cross-agent. The change is hardening for parity so a malicious app cannot bounce the request to a different port on the same agent. ## Hardening and defense in depth The two layers are independent. `CheckRedirect` removes the redirect-following behavior entirely, and the dial pinning guarantees that even a request constructed with a foreign host can only ever reach the intended agent. Removing either one in the future cannot, on its own, reintroduce the cross-agent vector. ## Tests - `codersdk/workspacesdk/agentconn_redirect_test.go` builds a three-peer tailnet (client, attacker, victim). The attacker agent redirects to the victim's port-4 URL, and the test asserts that `GET` `302`, `POST` `307`, and `POST` `308` all return an error and that the victim is never contacted. - `coderd/aitasks_internal_test.go` adds `TestTaskAppHTTPClient_RejectsRedirect`, which verifies the task app client surfaces a `307` instead of following it to a stand-in victim. ## Why this closes the whole vulnerability class `apiClient` is the only HTTP chokepoint to the agent port-4 API, so fixing it covers every server-side caller: - Every agent HTTP API method in `agentConn` funnels through `apiClient`, either via `apiRequest`, a direct `apiClient(ctx).Do(...)` (`ExecuteDesktopAction`), or as the websocket `HTTPClient` (`WatchContainers`, `WatchGit`, `ConnectDesktopVNC`). The websocket handshake matters here: `coder/websocket` follows `3xx` during the handshake by default and only requires `101` on the final hop, but it honors the underlying client's `CheckRedirect`, so reusing `apiClient` closes the websocket paths too. - The HTTP MCP server coderd hosts at `/api/experimental/mcp/http` registers tools (`coder_workspace_bash`, `_write_file`, `_read_file`, `_edit_files`, etc.) that reach the agent through `workspacesdk.AgentConn` methods, so they go through `apiClient` and are covered. The same is true for agent-hosted MCP, which coderd reaches only via `agentConn.CallMCPTool` / `ListMCPTools`. coderd never opens an MCP client connection directly to an agent over the tailnet. - Raw-TCP agent services (reconnecting PTY, SSH, speedtest, generic `DialContext`) speak non-HTTP protocols and have no redirect surface. The workspace apps reverse proxy targets user app ports, not port 4, forwards `3xx` to the browser rather than following them, and pins its transport to the request's agent. - `provisionerd` does not talk to the agent HTTP API at all. No other server-side client follows redirects to an agent-controllable tailnet host, so no further redirect changes are required for this class.
1 parent 77f1731 commit eeb2624

7 files changed

Lines changed: 253 additions & 42 deletions

File tree

‎coderd/aitasks.go‎

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@ import (
66
"encoding/json"
77
"errors"
88
"fmt"
9-
"net"
109
"net/http"
1110
"net/url"
1211
"slices"
@@ -1086,13 +1085,7 @@ func (api *API) authAndDoWithTaskAppClient(
10861085
}
10871086
defer release()
10881087

1089-
client := &http.Client{
1090-
Transport: &http.Transport{
1091-
DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
1092-
return agentConn.DialContext(ctx, network, addr)
1093-
},
1094-
},
1095-
}
1088+
client := agentConn.AppHTTPClient()
10961089
return do(ctx, client, parsedURL)
10971090
}
10981091

‎coderd/tailnet.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -298,6 +298,7 @@ func (s *ServerTailnet) AgentConn(ctx context.Context, agentID uuid.UUID) (works
298298
conn = workspacesdk.NewAgentConn(s.conn, workspacesdk.AgentConnOptions{
299299
AgentID: agentID,
300300
CloseFunc: func() error { return workspacesdk.ErrSkipClose },
301+
Logger: s.logger,
301302
})
302303

303304
// Since we now have an open conn, be careful to close it if we error

‎codersdk/workspacesdk/agentconn.go‎

Lines changed: 35 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,7 @@ type AgentConn interface {
102102
DebugMagicsock(ctx context.Context) ([]byte, error)
103103
DebugManifest(ctx context.Context) ([]byte, error)
104104
DialContext(ctx context.Context, network string, addr string) (net.Conn, error)
105+
AppHTTPClient() *http.Client
105106
GetPeerDiagnostics() tailnet.PeerDiagnostics
106107
ListContainers(ctx context.Context) (codersdk.WorkspaceAgentListContainersResponse, error)
107108
ListProcesses(ctx context.Context) (ListProcessesResponse, error)
@@ -157,6 +158,7 @@ func (c *agentConn) SetExtraHeaders(h http.Header) {
157158
type AgentConnOptions struct {
158159
AgentID uuid.UUID
159160
CloseFunc func() error
161+
Logger slog.Logger
160162
}
161163

162164
func (c *agentConn) agentAddress() netip.Addr {
@@ -369,6 +371,24 @@ func (c *agentConn) DialContext(ctx context.Context, network string, addr string
369371
}
370372
}
371373

374+
// AppHTTPClient returns an HTTP client for reaching HTTP apps served by this
375+
// workspace agent. Redirects are blocked to prevent misuse.
376+
func (c *agentConn) AppHTTPClient() *http.Client {
377+
return &http.Client{
378+
CheckRedirect: func(*http.Request, []*http.Request) error {
379+
return http.ErrUseLastResponse
380+
},
381+
Transport: &http.Transport{
382+
// Disable keep-alives so these short-lived clients don't leave
383+
// idle connections (and their goroutines) lingering after they're
384+
// discarded.
385+
DisableKeepAlives: true,
386+
// Host locked to agent, port from URL.
387+
DialContext: c.DialContext,
388+
},
389+
}
390+
}
391+
372392
// ListeningPorts lists the ports that are currently in use by the workspace.
373393
func (c *agentConn) ListeningPorts(ctx context.Context) (codersdk.WorkspaceAgentListeningPortsResponse, error) {
374394
ctx, span := tracing.StartSpan(ctx)
@@ -1362,7 +1382,12 @@ func (c *agentConn) apiRequest(ctx context.Context, method, path string, body in
13621382
// scoped to a single request: its transport cancels in-flight dials
13631383
// once reqCtx ends.
13641384
func (c *agentConn) apiClient(reqCtx context.Context) *http.Client {
1385+
agentAddr := netip.AddrPortFrom(c.agentAddress(), AgentHTTPAPIServerPort)
13651386
return &http.Client{
1387+
// Redirects are blocked to prevent misuse.
1388+
CheckRedirect: func(*http.Request, []*http.Request) error {
1389+
return http.ErrUseLastResponse
1390+
},
13661391
Transport: &http.Transport{
13671392
// Disable keep alives as we're usually only making a single
13681393
// request, and this triggers goleak in tests
@@ -1376,11 +1401,17 @@ func (c *agentConn) apiClient(reqCtx context.Context) *http.Client {
13761401
if err != nil {
13771402
return nil, xerrors.Errorf("split host port %q: %w", addr, err)
13781403
}
1379-
1380-
// Verify that the port is TailnetStatisticsPort.
13811404
if port != strconv.Itoa(AgentHTTPAPIServerPort) {
13821405
return nil, xerrors.Errorf("request %q does not appear to be for http api", addr)
13831406
}
1407+
if reqAddr, err := netip.ParseAddr(host); err != nil || reqAddr != agentAddr.Addr() {
1408+
c.opts.Logger.Warn(ctx, "blocked workspace agent API request to unintended host",
1409+
slog.F("agent_id", c.opts.AgentID),
1410+
slog.F("request_host", host),
1411+
slog.F("intended_agent_addr", agentAddr.Addr()),
1412+
)
1413+
return nil, xerrors.Errorf("request host %q does not match intended agent %q", host, agentAddr.Addr())
1414+
}
13841415

13851416
// http.Transport detaches ctx from the request context so
13861417
// a pending dial can outlive its request and serve future
@@ -1398,12 +1429,8 @@ func (c *agentConn) apiClient(reqCtx context.Context) *http.Client {
13981429
return nil, xerrors.Errorf("workspace agent not reachable in time: %v", ctx.Err())
13991430
}
14001431

1401-
ipAddr, err := netip.ParseAddr(host)
1402-
if err != nil {
1403-
return nil, xerrors.Errorf("parse host addr: %w", err)
1404-
}
1405-
1406-
conn, err := c.Conn.DialContextTCP(ctx, netip.AddrPortFrom(ipAddr, AgentHTTPAPIServerPort))
1432+
// Always dial the pinned agent address, never the request host.
1433+
conn, err := c.Conn.DialContextTCP(ctx, agentAddr)
14071434
if err != nil {
14081435
return nil, xerrors.Errorf("dial http api: %w", err)
14091436
}

‎codersdk/workspacesdk/agentconn_test.go‎

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,25 @@ package workspacesdk_test
22

33
import (
44
"context"
5+
"errors"
6+
"fmt"
7+
"net"
8+
"net/http"
59
"net/netip"
10+
"strings"
11+
"sync/atomic"
612
"testing"
713

814
"github.com/google/uuid"
15+
"github.com/stretchr/testify/assert"
916
"github.com/stretchr/testify/require"
1017
"go.uber.org/goleak"
18+
"tailscale.com/tailcfg"
1119

1220
"github.com/coder/coder/v2/codersdk/workspacesdk"
1321
"github.com/coder/coder/v2/tailnet"
22+
"github.com/coder/coder/v2/tailnet/proto"
23+
"github.com/coder/coder/v2/tailnet/tailnettest"
1424
"github.com/coder/coder/v2/testutil"
1525
)
1626

@@ -64,3 +74,191 @@ func TestAgentConn_DialBoundedByRequestContext(t *testing.T) {
6474

6575
goleak.VerifyNone(t, ignoreCurrent)
6676
}
77+
78+
func TestAgentConnRejectsCrossAgentRedirects(t *testing.T) {
79+
t.Parallel()
80+
81+
derpMap, _ := tailnettest.RunDERPAndSTUN(t)
82+
cases := []struct {
83+
name string
84+
status int
85+
invoke func(context.Context, workspacesdk.AgentConn) error
86+
}{
87+
{
88+
name: "get 302",
89+
status: http.StatusFound,
90+
invoke: func(ctx context.Context, conn workspacesdk.AgentConn) error {
91+
_, err := conn.ListeningPorts(ctx)
92+
return err
93+
},
94+
},
95+
{
96+
name: "post 307",
97+
status: http.StatusTemporaryRedirect,
98+
invoke: func(ctx context.Context, conn workspacesdk.AgentConn) error {
99+
return conn.WriteFile(ctx, "/tmp/attacker", strings.NewReader("redirect-body"))
100+
},
101+
},
102+
{
103+
name: "post 308",
104+
status: http.StatusPermanentRedirect,
105+
invoke: func(ctx context.Context, conn workspacesdk.AgentConn) error {
106+
return conn.WriteFile(ctx, "/tmp/attacker", strings.NewReader("redirect-body"))
107+
},
108+
},
109+
}
110+
111+
for _, tc := range cases {
112+
t.Run(tc.name, func(t *testing.T) {
113+
t.Parallel()
114+
115+
ctx := testutil.Context(t, testutil.WaitMedium)
116+
117+
clientID := uuid.New()
118+
attackerID := uuid.New()
119+
victimID := uuid.New()
120+
clientConn, _ := newTailnetConn(t, derpMap, clientID, "client")
121+
attackerConn, attackerIP := newTailnetConn(t, derpMap, attackerID, "attacker")
122+
victimConn, victimIP := newTailnetConn(t, derpMap, victimID, "victim")
123+
stitchTailnet(t, map[uuid.UUID]*tailnet.Conn{
124+
clientID: clientConn,
125+
attackerID: attackerConn,
126+
victimID: victimConn,
127+
})
128+
129+
var victimHit atomic.Bool
130+
victimRouter := http.NewServeMux()
131+
victimRouter.HandleFunc("/api/v0/listening-ports", func(rw http.ResponseWriter, _ *http.Request) {
132+
victimHit.Store(true)
133+
rw.Header().Set("Content-Type", "application/json")
134+
_, _ = rw.Write([]byte(`{"ports":[]}`))
135+
})
136+
victimRouter.HandleFunc("/api/v0/write-file", func(rw http.ResponseWriter, _ *http.Request) {
137+
victimHit.Store(true)
138+
rw.WriteHeader(http.StatusOK)
139+
})
140+
serveTailnetHTTP(t, victimConn, victimRouter)
141+
142+
victimBaseURL := fmt.Sprintf("http://[%s]:%d", victimIP, workspacesdk.AgentHTTPAPIServerPort)
143+
attackerRouter := http.NewServeMux()
144+
attackerRouter.HandleFunc("/", func(rw http.ResponseWriter, r *http.Request) {
145+
http.Redirect(rw, r, victimBaseURL+r.URL.RequestURI(), tc.status)
146+
})
147+
serveTailnetHTTP(t, attackerConn, attackerRouter)
148+
149+
require.True(t, clientConn.AwaitReachable(ctx, attackerIP))
150+
require.True(t, clientConn.AwaitReachable(ctx, victimIP))
151+
152+
conn := workspacesdk.NewAgentConn(clientConn, workspacesdk.AgentConnOptions{
153+
AgentID: attackerID,
154+
})
155+
156+
err := tc.invoke(ctx, conn)
157+
require.Error(t, err)
158+
require.False(t, victimHit.Load())
159+
})
160+
}
161+
}
162+
163+
// TestAgentConnAppHTTPClientRefusesRedirects verifies the app HTTP client does
164+
// not follow redirects.
165+
func TestAgentConnAppHTTPClientRefusesRedirects(t *testing.T) {
166+
t.Parallel()
167+
168+
tailnetConn, err := tailnet.NewConn(&tailnet.Options{
169+
Addresses: []netip.Prefix{tailnet.TailscaleServicePrefix.RandomPrefix()},
170+
Logger: testutil.Logger(t),
171+
})
172+
require.NoError(t, err)
173+
t.Cleanup(func() {
174+
_ = tailnetConn.Close()
175+
})
176+
177+
conn := workspacesdk.NewAgentConn(tailnetConn, workspacesdk.AgentConnOptions{
178+
AgentID: uuid.New(),
179+
})
180+
181+
client := conn.AppHTTPClient()
182+
require.NotNil(t, client.CheckRedirect)
183+
req, err := http.NewRequestWithContext(context.Background(), http.MethodGet, "http://example.invalid/", nil)
184+
require.NoError(t, err)
185+
require.ErrorIs(t, client.CheckRedirect(req, nil), http.ErrUseLastResponse)
186+
}
187+
188+
func newTailnetConn(t *testing.T, derpMap *tailcfg.DERPMap, id uuid.UUID, name string) (*tailnet.Conn, netip.Addr) {
189+
t.Helper()
190+
191+
addr := tailnet.TailscaleServicePrefix.AddrFromUUID(id)
192+
conn, err := tailnet.NewConn(&tailnet.Options{
193+
ID: id,
194+
Addresses: []netip.Prefix{netip.PrefixFrom(addr, 128)},
195+
Logger: testutil.Logger(t).Named(name),
196+
DERPMap: derpMap,
197+
})
198+
require.NoError(t, err)
199+
t.Cleanup(func() {
200+
assert.NoError(t, conn.Close())
201+
})
202+
203+
return conn, addr
204+
}
205+
206+
func serveTailnetHTTP(t *testing.T, conn *tailnet.Conn, handler http.Handler) {
207+
t.Helper()
208+
209+
ln, err := conn.Listen("tcp", fmt.Sprintf(":%d", workspacesdk.AgentHTTPAPIServerPort))
210+
require.NoError(t, err)
211+
212+
server := &http.Server{Handler: handler, ReadHeaderTimeout: testutil.WaitShort}
213+
t.Cleanup(func() {
214+
assert.NoError(t, server.Close())
215+
assert.NoError(t, ln.Close())
216+
})
217+
218+
go func() {
219+
err := server.Serve(ln)
220+
if err != nil && !errors.Is(err, net.ErrClosed) && !errors.Is(err, http.ErrServerClosed) {
221+
assert.NoError(t, err)
222+
}
223+
}()
224+
}
225+
226+
// stitchTailnet cross-programs every conn's node into every other conn, the
227+
// N-peer analog of tailnet's stitch test helper, so the peers can reach each
228+
// other without a coordinator.
229+
func stitchTailnet(t *testing.T, conns map[uuid.UUID]*tailnet.Conn) {
230+
t.Helper()
231+
232+
sendNode := func(srcID uuid.UUID, node *tailnet.Node) {
233+
protoNode, err := tailnet.NodeToProto(node)
234+
if !assert.NoError(t, err) {
235+
return
236+
}
237+
for dstID, dst := range conns {
238+
if dstID == srcID {
239+
continue
240+
}
241+
err = dst.UpdatePeers([]*proto.CoordinateResponse_PeerUpdate{{
242+
Id: srcID[:],
243+
Node: protoNode,
244+
Kind: proto.CoordinateResponse_PeerUpdate_NODE,
245+
}})
246+
assert.NoError(t, err)
247+
}
248+
}
249+
250+
for srcID, src := range conns {
251+
src.SetNodeCallback(func(node *tailnet.Node) {
252+
sendNode(srcID, node)
253+
})
254+
if node := src.Node(); node != nil {
255+
sendNode(srcID, node)
256+
}
257+
}
258+
259+
t.Cleanup(func() {
260+
for _, conn := range conns {
261+
conn.SetNodeCallback(nil)
262+
}
263+
})
264+
}

‎codersdk/workspacesdk/agentconnmock/agentconnmock.go‎

Lines changed: 14 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎codersdk/workspacesdk/workspacesdk.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -300,6 +300,7 @@ func (c *Client) DialAgent(dialCtx context.Context, agentID uuid.UUID, options *
300300
<-controller.Closed()
301301
return conn.Close()
302302
},
303+
Logger: options.Logger,
303304
})
304305

305306
if !agentConn.AwaitReachable(dialCtx) {

0 commit comments

Comments
 (0)