Memory spike on GET /api/v2/workspaces?limit=0 (unbounded convertWorkspaceBuild map allocations) #27205

Summary

Hitting GET /api/v2/workspaces?limit=0 on a deployment with many workspaces triggers a large transient memory spike. A live repro was observed on dev.coder.com at approximately 19:47:10 GMT on 2026-07-13, correlated with Grafana memory metrics.

Root cause (suspected)

Request path:

GET /api/v2/workspaces?limit=0 → api.workspaces → api.workspaceData → api.convertWorkspaceBuilds → api.convertWorkspaceBuild (called once per build).

1. limit=0 is explicitly interpreted as no limit in pagination.go:

   // A limit of 0 should be interpreted by the SQL query as "null" or
   // "no limit". Do not make this value anything besides 0.
   Limit:  int(parser.PositiveInt32(queryParams, 0, "limit")),

   So GetAuthorizedWorkspaces returns every workspace the caller can read. On a large deployment this is a lot.

2. In convertWorkspaceBuild (called for each build inside convertWorkspaceBuilds), the same fan-out maps are rebuilt from the global slices on every iteration:

   resourcesByJobID     := map[uuid.UUID][]database.WorkspaceResource{}
   metadataByResourceID := map[uuid.UUID][]database.WorkspaceResourceMetadatum{}
   agentsByResourceID   := map[uuid.UUID][]database.WorkspaceAgent{}
   appsByAgentID        := map[uuid.UUID][]database.WorkspaceApp{}
   scriptsByAgentID     := map[uuid.UUID][]database.GetWorkspaceAgentScriptsByAgentIDsRow{}
   logSourcesByAgentID  := map[uuid.UUID][]database.WorkspaceAgentLogSource{}
   statusesByAgentID    := map[uuid.UUID][]database.WorkspaceAppStatus{}

   All seven maps are allocated N times (once per build) even though the input slices are identical across the loop. Per-call cost is roughly O(R + M + A + App + Script + LogSrc + Status).

   With an unbounded workspace count and non-trivial resources per build this is easily hundreds of MB of transient allocation, matching a spike-then-GC-recovery shape rather than a steady leak.

Reproduction

• Deployment: dev.coder.com
• Request: GET /api/v2/workspaces?limit=0 (any caller with read access to a large number of workspaces)
• Observed: memory spike at ~19:47:10 GMT, 2026-07-13 (confirmed against Grafana dashboards in the originating thread).

Proposed fixes

1. Hoist map construction out of convertWorkspaceBuild into convertWorkspaceBuilds so the seven maps are built once and reused across all builds. Small, safe change; should eliminate most of the spike.
2. Clamp limit on /workspaces server-side (e.g. cap limit=0 at 1000 for this endpoint) so a single request cannot fan out unbounded object graphs.
3. Longer-term: push the fan-out into SQL / return pre-joined rows so we don't materialize the entire object graph in memory per request.

Related

• Coder pods running out of memory #14881 – "Coder pods running out of memory" (may share root cause). Most likely not related; that seems to be a slower leak instead of a sharp spike.

Created on behalf of @johnstcn.