fix: bound request body size on JSON API endpoints - #28168
Conversation
httpapi.Read decoded r.Body with no ceiling, so one request could allocate memory without limit. The exposure is pre-authentication: login, OTP, first-user creation, OAuth2 dynamic registration, and SCIM provisioning all decode a body before any authorization decision is reached. Rate limiting bounds request rate, not the memory a single admitted request may consume. Split Read into Read and ReadLimit. ReadLimit wraps r.Body in an http.MaxBytesReader and holds the existing decode and validate logic; Read delegates to it with a new DefaultMaxRequestBodyBytes of 4 MiB. That single wrap site covers the 124 remaining non-test callers at once. http.MaxBytesReader composes as tightest-wins, so an unconditional wrap inside Read would have overridden the handlers that pre-wrapped their own bodies. That is silent where the caller's limit is larger: the bulk secrets import at 8x MaxSecretsFileBytes would have halved to the default. Those handlers pass their limits to ReadLimit instead, leaving the effective limit at each byte for byte unchanged. TestMaxBytesReaderNesting pins the composition behavior the requirement rests on. The limit is a constant rather than a deployment option. An operator raising it to unblock something would reopen the vulnerability as configuration, where a security scan will not find it. A legitimate 413 is answered with a targeted ReadLimit on that endpoint.
A 413 for an oversized body named no limit, so an operator could not tell which of the ceilings in the tree produced it, and a limit set too tight for a legitimate payload was indistinguishable from a client that hung up. RecordRequestBodyLimit puts the limit on the request's existing log line, rather than a line of its own: a caller can produce 413s at will, so a dedicated line is attacker-controlled log volume. It also marks the request through a tracker carried on the context, which the Prometheus middleware reads to tell a body size rejection from the other reasons coderd answers 413. The tracker is installed in a later commit; a call without one is a no-op, which is what lets sites adopt this before the middleware exists. Every site that answers 413 because a request body exceeded a limit calls this, and a site answering 413 for any other reason must not. The sites keep their own error shapes, which is why what they share is this call rather than a response writer.
Nothing counted body size rejections apart from the other reasons coderd answers 413, so a deliberate exhaustion attempt and a limit set too tight for a legitimate payload were both invisible. coderd_api_requests_too_large_total carries method, path, and a reason label fed by the tracker this middleware installs and the recording sites mark. reason="request_body" is the alertable series; everything else, such as the agent log storage overflow at workspaceagents.go, lands under reason="other". Series exist only for routes that have actually rejected a body, which is what makes this readable at a glance where filtering requests_processed_total by code is not. Counting is keyed on the response status rather than the point of rejection, which is what reaches the endpoints that bound their own bodies to keep their own error shapes. The metric name and its existing labels are unchanged, so a query that ignores labels keeps working, but anything matching an exact label set will need reason added.
POST /api/v2/files installed a 100 MiB bound and then reported the rejection as a 400 read failure, leaking the stdlib "http: request body too large" string through Detail. It is the largest limit in the tree, so the metric under-counted precisely where a legitimate payload is most likely to be refused. An oversized body is a size failure and is now reported as one, with the limit recorded. The separate 413 for an oversized expanded archive is about the expanded bytes, which are not reached until this read succeeds, and is unchanged. The swagger annotation listed only the success responses, so the published reference did not mention the status the endpoint could already return.
postWorkspaceAgentTaskLogSnapshot reimplemented ReadLimit: its own MaxBytesReader wrap, its own decode, its own 400 for an oversized body. It calls ReadLimit instead, which removes the duplication and records the limit it was missing. Validate is a no-op on the payload type, which carries no validate tags, so validation behavior is unchanged. Behavior change: the endpoint answers 413 rather than 400 once its existing 64 KiB cap is exceeded, and its decode failure message becomes "Request body must be valid JSON.", which is what every other endpoint answers. Its test is updated to match both.
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
Every endpoint that reads a JSON body can now answer 413, but the reference listed that status on four of them. Annotate the rest, plus the endpoints that answered 413 for their own reasons and never said so: chat file uploads and agent log storage overflow.
The 4 MiB ceiling applies to every endpoint that decodes a JSON body, so a caller that trips it has no single endpoint page to learn it from. State it once on the reference index, with the 413 it produces and the fact that no deployment option raises it.
This reverts commit cd704a0. The 108 annotations expanded into roughly 1,800 lines of regenerated swagger and markdown, which buries the security fix this branch is for. The reference documents the request body limit once on its index page instead.
|
/coder-agents-review |
|
Chat: Review posted | View chat Review historydeep-review v0.9.0 | Round 2 | Last posted: Round 2, 17 findings (2 P2, 4 P3, 10 Nit, 1 Note), COMMENT. Review Finding inventoryFinding inventory - PR #28168Findings
Contested and acknowledgedCRF-1 (P2, enterprise/coderd/aimodelprices.go:72) - missing RecordRequestBodyLimit on enterprise 413 site
CRF-2 (P3, coderd/files.go:68) - no e2e test for body-too-large 413 flip
Round logRound 1Netero-only. 1 P2, 1 P3. LOC 545 effective (< 1000), Law skipped. P2 gates the panel per the Netero decision gate; first-pass posted so the author can address before the full panel spends time. Reviewed against b5d18bb..30515fb. Round 2Churn guard PROCEED. No PR-authored commits between R1 and R2 heads; only two About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
First-pass review from Netero only. The full panel has not looked at this yet; the panel will spin up after these mechanical findings are addressed.
The change is well-scoped and the tests do work: TestReadLimit/RecordsLimitOnRequestLog pins the log field, TestPrometheus/RequestTooLarge pins both the emission of reason=request_body and the abstention on non-body 413s, and TestMaxBytesReaderNesting pins the tightest-wins composition that the whole "pre-wrapped handler keeps its ceiling" story rests on. TestImportUserSecretsBodyLargerThanDefaultLimit is exactly the regression guard the description promises for the 8 * MaxSecretsFileBytes case. As Netero put it: "pins the composition claim, not a decoration."
Severity count: 1 P2, 1 P3.
One sibling site was missed. The invariant that RecordRequestBodyLimit's doc pins ("every site that answers 413 because a request body exceeded a limit must call this") holds inside coderd/, but the enterprise upsert-model-prices handler answers 413 in exactly the codersdk.Response shape the PR claims as its scope and never calls the recorder. See CRF-1. This flips its metric label to reason=other and drops max_request_body_bytes from its log line, defeating the observability the metric exists to provide on that one route.
CRF-2 is a testing gap on one of the two user-visible status flips called out in the Behavior-change section.
enterprise/coderd/aimodelprices.go:72
P2 [CRF-1] upsertAIModelPrices answers 413 for a body-size rejection without calling httpapi.RecordRequestBodyLimit. (Netero)
The invariant introduced at
+coderd/httpapi/requestbodylimit.go:41says "Every site that answers 413 because a request body exceeded a limit must call this," and the PR patches the three coderd sites that satisfy that predicate (csp.go,files.go,exp_chats.go'spostChatFile).enterprise/coderd/aimodelprices.go:68wrapsr.Bodyinhttp.MaxBytesReader(codersdk.MaxAIModelPricesBytes),:72catches*http.MaxBytesErrorand answerscodersdk.Response{"Request body too large."}, and the route mounts under/api/experimental/ai/model-pricesinside the AGPL router that installsprometheusMW. It answers incodersdk.Responseshape, which is exactly the scope this PR claims ("the endpoints that answer incodersdk.Responseshape").Consequence: a request rejected for exceeding
MaxAIModelPricesBytesincrementscoderd_api_requests_too_large_total{path="/api/experimental/ai/model-prices",reason="other"}instead ofreason="request_body", and the request's log line does not carrymax_request_body_bytes. That is exactly the observability the metric exists to provide, silently degraded on one route. An operator alerting onrate(coderd_api_requests_too_large_total{reason="request_body"}[5m])to catch a limit set too tight would miss this endpoint.
Verified: the route is registered on api.AGPL.ExperimentalHandler (enterprise/coderd/coderd.go:338), which is mounted at /api/experimental on the router that installs prometheusMW at coderd/coderd.go:1156, so the tracker set by RecordRequestBodyLimit would be read. The AGPL sites all call the recorder before writing 413 (csp.go:44, files.go:69, exp_chats.go:5856). This one enterprise site is the only miss under this PR's stated scope, and every future enterprise 413 site will need the same call, so pinning the convention now avoids a repeat. Match the AGPL sites' Detail while you are here (fmt.Sprintf("Maximum request body size is %d bytes.", codersdk.MaxAIModelPricesBytes)) so it names the limit instead of returning the stdlib string.
🤖
🤖 This review was automatically generated with Coder Agents.
|
[CRF-1] Addressed in #28181 ( It lands there rather than here because the same commit adds the For the record on the other half of the question: |
|
/coder-agents-review |
There was a problem hiding this comment.
Panel round. The security half is done and it holds up: Read installs the 4 MiB default before any pre-auth decoder runs, TestUserLogin/BodyTooLarge proves the cut-off lands before authentication, and TestMaxBytesReaderNesting pins the tightest-wins composition the whole ceiling rests on. TestImportUserSecretsBodyLargerThanDefaultLimit is the regression guard the description promises. The observability half wires the metric and log field into every AGPL body-size 413 site through one function. As Pariston put it after walking the causal chain: "The alternative framings would either match this design's cost with less flexibility (B, C) or require a broader helper than the codebase actually needs (D). The one framing this PR does not fully embrace (E, mechanical enforcement) is what the follow-up PR ships."
R2 counts: 1 P2, 3 P3, 10 Nits, 1 Note. R1 CRF-1 stays deferred to #28181; the fix is verified on that branch head. R1 CRF-2 is re-raised because the author's "no cheaper option" claim did not hold: two panel reviewers surfaced structural alternatives (see the reply on that thread). CI: 22 passed, 1 pending, 8 skipped at HEAD 670f27b.
What matters:
CRF-3 (P2). ReadLimit writes its own limit parameter into the 413 Detail and into RecordRequestBodyLimit. Nested MaxBytesReaders propagate the inner error unchanged, so the reported number can be 64x larger than what actually tripped. CRF-4 is the one deployed caller that exercises this today (agent/agentfiles/bundlefiles.go:52 pre-wraps at 64 KiB then calls httpapi.Read), and the ReadLimit docstring explicitly names the pattern it violates. Fix the class at ReadLimit by taking the number from the *http.MaxBytesError and migrate the one caller; the two together give both belt and suspenders. TestMaxBytesReaderNesting currently asserts bytes-read and error-type only; extending it to require maxErr.Limit == tight pins the invariant the fix rests on.
CRF-5 (P3). The ruleguard rule in #28181 scopes its allowlist to the coderd HTTP API and excludes aibridge by name. aibridge/bridge.go:267 and aibridge/passthrough.go:59 both write 413 for *http.MaxBytesError at 32 MiB without calling RecordRequestBodyLimit, and they mount under api.AGPL.APIHandler, so coderd's Prometheus middleware runs on them. Rejection at the AI gateway shows up as reason="other" next to agent log storage overflow. That is not a deferral covered by the follow-up; it is a drop.
CRF-6 (P3). docs/admin/integrations/prometheus.md:231 explains reason="request_body" and leaves reason="other" undefined. In this tree other covers at least three unrelated failure modes (agent log storage overflow, expanded archive too large on /api/v2/files, and the CRF-1 route until #28181 lands). A platform engineer alerting on other reads the source to interpret their own dashboard.
Process note. Commit graph on this branch shows cd704a068c adding an exhaustive @Failure 413 pass and 2447570c90 revert: cutting it two commits later with the exact reasoning "buries the security fix this branch is for." That is proportional-output self-correction visible in the graph, not just claimed. The rest of the diff behaves.
agent/agentfiles/bundlefiles.go:52
P3 [CRF-4] HandleBundleFiles pre-wraps r.Body at bundleFilesRequestMaxBytes (64 KiB) and then calls httpapi.Read, the pattern the new ReadLimit docstring at coderd/httpapi/httpapi.go:251 says callers must not use. (Melody)
The rest of the migration walks the same pairing site by site and replaces the pre-wrap with
ReadLimit:coderd/exp_chats.gofive callers,coderd/usersecrets.goone,coderd/userskills.gotwo,coderd/aitasks.goone. This one caller was missed.Tightest-wins composition still trips the 64 KiB inner wrap first, so the cap holds. But the 413 the client sees names 4 MiB instead of 65536, and any request logger installed on the agent side receives
max_request_body_bytes=4194304. A client that exceeds 64 KiB reads that they were allowed 4 MiB, which is the observability regression this PR is meant to prevent.
This is the one deployed instance of CRF-3's class-level bug. Even with CRF-3's fix (which would surface the correct 64 KiB in the response), the caller still violates the ReadLimit docstring's explicit invariant, so both fixes should land. Replace the two lines with if !httpapi.ReadLimit(r.Context(), w, r, bundleFilesRequestMaxBytes, &req) { return }.
🤖
aibridge/bridge.go:267
P3 [CRF-5] Aibridge answers 413 for *http.MaxBytesError at 32 MiB without calling RecordRequestBodyLimit, so coderd_api_requests_too_large_total{reason="request_body"} never counts an AI Gateway overflow. Same class at aibridge/passthrough.go:59 in the reverse-proxy ErrorHandler. (Hisoka)
RequestBridge.ServeHTTPataibridge/bridge.go:423wrapsr.Bodywithhttp.MaxBytesReaderatmaxRequestBodyBytes = 32 << 20(aibridge/bridge.go:53). The handler at line 265 catches*http.MaxBytesErrorand callswriteRequestBodyTooLarge(w), which ishttp.Error(..., http.StatusRequestEntityTooLarge)at line 386. Nohttpapi.RecordRequestBodyLimit.The routes reach coderd's metric middleware.
enterprise/coderd/coderd.go:298,307mount/aibridgeand/ai-gatewayonapi.AGPL.APIHandler, which runsprometheusMWatcoderd/coderd.go:1156outside every route.StatusWriter.Statusbecomes 413; the tracker was allocated and never flipped, because no call toRecordRequestBodyLimitreached the request. The 413 is written asreason="other", sitting next toworkspaceagents.go:221where the log storage overflowed for a reason that has nothing to do with the body the client sent.
The ruleguard rule in #28181 (2aeda6ad80 chore: reject unbounded r.Body reads in coderd) explicitly excludes aibridge by name: "The agent, aibridge, and the load-test mocks serve their own ingress with their own error shapes and their own limits, so httpapi.Read is not the remedy there." That rule enforces the bound-your-body invariant; it says nothing about the record-the-limit invariant that keeps the new metric honest. The three follow-up PRs enumerated in the description (codersdk.Response, RFC 6749/7591, RFC 7644) do not name aibridge. This is not a deferral; there is no follow-up shipping the fix.
Two lines each: httpapi.RecordRequestBodyLimit(r.Context(), maxRequestBodyBytes) immediately before writeRequestBodyTooLarge(w) at bridge.go:267 and before the equivalent write in passthrough.go:59. The tracker rides r.Context() from the outer middleware; no cycle (coderd/httpapi has no reverse dependency on aibridge).
🤖
coderd/exp_chats.go:5872
Note [CRF-16] postChatFile answers 413 with the codersdk-shape "File too large." but its swagger annotation does not declare it. (Leorio)
The revert commit chose "document once on the index page instead" of annotating every endpoint. Two annotations survived: postFile (coderd/files.go) and postWorkspaceAgentTaskLogSnapshot (coderd/aitasks.go). The postFile commit body justifies its annotation with "The swagger annotation listed only the success responses, so the published reference did not mention the status the endpoint could already return." That reasoning fits postChatFile (413 for MaxChatFileSizeBytes, unchanged in this PR) and patchWorkspaceAgentLogs (413 for agent log storage overflow, unchanged) exactly as well. The two exceptions to "index page only" were chosen on one criterion; the population that matches that criterion is not fully covered.
Two coherent resolutions: either annotate postChatFile and patchWorkspaceAgentLogs the same way, or drop postFile's 413 annotation on the grounds that its raw-body 413 is new here and its expanded-archive 413 falls under "documented once on the index page."
🤖
🤖 This review was automatically generated with Coder Agents.
aibridge answers 413 when a request body exceeds its 32 MiB cap, but it never recorded the limit, so the request log did not name the limit that tripped and coderd_api_requests_too_large_total counted the rejection as reason="other". aibridge is mounted on the coderd API handler, which is behind the middleware that reads the tracker, so the mislabelling is reachable in every deployment that enables AI Gateway. Record the limit inside writeRequestBodyTooLarge, the sole path to a body-too-large response from both the interception and passthrough handlers, so the two cannot drift apart.
ReadLimit reported the limit it installed itself, but nested MaxBytesReaders compose as tightest-wins and the error carries the winner. A caller that wrapped r.Body tighter got a 413 naming a cap it never hit, and the same wrong number reached the request log. HandleBundleFiles wraps at 64 KiB before calling Read, so it answered with 4194304, a 64x overstatement of the limit that rejected the request. Read the limit off the *http.MaxBytesError instead of the parameter. That is correct for any caller and any nesting order, so it closes the class rather than the one site. Single-wrap callers are unaffected: the error carries the limit they installed.
HandleBundleFiles wrapped r.Body in its own MaxBytesReader and then called Read, which installs a second one. ReadLimit's docstring forbids exactly this, and every other caller in the tree was migrated to ReadLimit. The cap itself held, since nested readers compose as tightest-wins, but the site was the last remaining nesting in the tree. Cover the 64 KiB cap while here. It had no test, so nothing failed if the limit were dropped.
TestMaxBytesReaderNesting asserted that http.MaxBytesReader composes as tightest-wins. ReadLimit's ReportsLimitThatTripped subtest now asserts the same property through our own code in both nesting orders, so the dedicated test only restated stdlib behaviour. Both halves of the deleted coverage are policed by the survivor: dropping the caller's wrap from that subtest fails it, and reporting the installed limit rather than the one the error carries fails it. TestReadBodyLimit exercises Read, not ReadLimit. Rename it to TestReadDefaultLimit so the pair maps onto the functions it covers. Trim two comments to the part that is not evident from the code.
The help string explained reason="request_body" and said nothing about the other label value, so an operator seeing a spike on it had no path from the docs to a diagnosis. It covers unrelated causes: agent log storage overflow and an archive that only exceeds the limit once expanded. Name both. Swap the rationale comment's example for one that is true at this commit. It cited the SCIM and OAuth2 endpoints as handlers that bound their own bodies, but neither does so yet; those bounds arrive in later PRs. aibridge bounds its own body, writes its own error shape, and is mounted behind this middleware today, so it makes the same point without depending on unmerged work.
Every 413 in the API reference carried the placeholder description "Request Entity Too Large", so a reader had to leave the reference to find out what limit applied. Give each @failure 413 annotation a description naming its limit, and rewrite the index page's request size limits section to say where per-endpoint limits are documented. Three endpoints answer 413 but did not declare it: POST /api/v2/chats, POST /api/v2/chats/{chat}/files, and PATCH /api/v2/workspaceagents/me/logs. Adding those closes the set: every endpoint in the published reference with a non-default request body limit now declares 413. Endpoints marked @x-apidocgen skip, and those with no swagger block, stay out of scope because they do not appear in the reference the index page describes. PATCH /api/v2/workspaceagents/me/logs answers 413 when agent log storage overflows, not when the request body is too large. Its description says so, and the index page notes that a few endpoints answer 413 for a reason other than body size. docs/reference/api/index.md is generated from a string literal in scripts/apidocgen/postprocess/main.go, so the prose change lives there.
ReadLimit's docstring described it as being for endpoints whose payloads exceed DefaultMaxRequestBodyBytes, but eight of its nine non-default callers set a tighter limit, not a looser one. A reader with a sub-default endpoint concluded Read was the right call and either lost the endpoint limit or wrapped the body themselves, which is the bug fixed in 4e7c2ed. Say the limit goes in either direction, and say which direction is the common one. The comment above the agentapi payload read spent two of its three sentences narrating what ReadLimit does, which the declaration already says. Keep the sentence with no other home: that Validate does nothing because agentapisdk.GetMessagesResponse carries no validate tags. RecordRequestBodyLimit takes r.Context() while the Write beneath it takes ctx, which reads like an oversight. Note why the two differ, framed as a warning against standardizing them, since ReadLimit is the only caller where they can be different contexts. DefaultMaxRequestBodyBytes' own docstring carried the same one-direction framing as ReadLimit's and is one line above it, so it gets the same correction.
…wo lines The comment explained why the sites answer in their own error shapes rather than through a common response writer. That rationale already lives on RecordRequestBodyLimit's declaration, so the test only needs to name the two things it pins.
Summary
httpapi.Readdecoded request bodies with no size limit, so a single request could allocate memory without bound. This adds a 4 MiB default ceiling, leaves the endpoints that legitimately need more explicitly exempted, and counts the rejections so a limit set too tight is visible.This is the first of three PRs split out of #28048, covering the endpoints that answer in
codersdk.Responseshape. The OAuth2 decode paths (RFC 6749, RFC 7591) and the SCIM ones (RFC 7644) answer in their own error shapes and follow in separate PRs, along with the lint rule that pins the invariant.Closes PLAT-463. Remediates SEC-416 (CWE-770, CVSS 7.5) and SEC-392.
Problem
httpapi.Readcallsjson.NewDecoder(r.Body).Decode(value)with no ceiling, and no middleware in the chain bounds body size. The exposure is pre-authentication: login, OTP, and first-user creation all read a body before any authorization decision is reached. The existing rate limiter bounds request rate, which is orthogonal to the memory a single admitted request may consume.Fix
Readis split intoReadandReadLimit.ReadLimitwrapsr.Bodyin anhttp.MaxBytesReaderand keeps the existing decode and validate logic;Readdelegates to it with a newDefaultMaxRequestBodyBytesof 4 MiB, which covers the 124 remaining non-test callers at a single site.http.MaxBytesReadercomposes as tightest-wins, so the handlers that pre-wrapped their own bodies pass their limit toReadLimitrather than wrapping, and each keeps its previous ceiling byte for byte. That matters most for the bulk secrets import at8 * MaxSecretsFileBytes: an unconditional wrap insideReadwould have silently halved it to the default.TestImportUserSecretsBodyLargerThanDefaultLimitis the regression guard for that specific failure, andTestMaxBytesReaderNestingpins the composition behavior the whole requirement rests on.Every rejection site calls
httpapi.RecordRequestBodyLimit, which names the limit that tripped on the request's existing log line and marks the request socoderd_api_requests_too_large_total{reason="request_body"}counts body rejections apart from the 413s coderd answers for other causes, such as agent log storage overflow. A limit set too tight for a legitimate payload therefore surfaces without waiting for a user report.The limit is a constant rather than a deployment option: an operator raising it to unblock something would reopen the vulnerability as configuration, where a security scan will not find it. A legitimate 413 is answered with a targeted
ReadLimiton that endpoint.Behavior change
POST /api/v2/filesnow answers 413 rather than 400 when a request body exceedsHTTPFileMaxBytes. It installed that bound already but reported the rejection as a read failure, which leaked the stdlibhttp: request body too largestring throughDetailand kept the largest limit in the tree off the metric. The separate 413 for an oversized expanded archive is unchanged.The task log snapshot endpoint now answers 413 rather than 400 when its 64 KiB cap is exceeded. Routing it through
ReadLimitalso changes its decode-failure message from "Failed to decode request payload." to "Request body must be valid JSON.", which is what every other endpoint answers. Its tests are updated to match both.coderd_api_requests_too_large_totalis new, so there is no existing query to migrate. It counts the 413s coderd answers, labeledmethod,path, andreason.reason="request_body"is a rejection by one of the limits above;reason="other"is a 413 that has nothing to do with body size, such as agent log storage overflow.Reading this
The commits are ordered to be read in sequence. Commits 1 and 2 are the security fix; commits 3 to 5 are the observability consequences, and commit 3 is the one that touches dashboards. Commit 7 documents the limit on the REST API reference index. Commits 6 and 8 add and revert an exhaustive
@Failure 413annotation pass, which buried the fix under its regenerated swagger, and cancel out.