Codestin Search App

VedantMadane · 2026-08-11T05:51:31Z

Summary

Fixes #7901

This PR adds optional RFC 9530 Content-Digest header generation to the ile_server handler for static file responses.

Details

Configurable via JSON content_digest array or Caddyfile content_digest / digest subdirectives.
Supports standard sha-256 and sha-512 algorithms.
Computes Content-Digest: sha-256=:...: / sha-512=:...: headers for static file responses.
Includes unit tests in staticfiles_test.go.

I have used help from various AI assistants to create this feature including Grok Build Heavy and Gemini 3.6 Flash Low.

CLAassistant · 2026-08-11T11:29:58Z

All committers have signed the CLA.

steadytao · 2026-08-11T11:30:38Z

Quick note. We need an AI assistance disclosure and CLA to be signed. although CLA assistant can be annoying sometimes, its fixed now :D

steadytao

Always nice to align with RFCs but we need to do it to-the-letter when doing so. This should also have far more tests; verify exact digest values and add integration coverage for GET, HEAD, 304, ranges, precompressed responses and read/seek failures. In particular, a 206 digest must cover the selected representation data rather than the complete source file.

francislavoie · 2026-08-13T05:04:50Z

I think this can cause the sendfile optimization to be disabled when used (but only for an HTTP server, no TLS because we don't have kTLS yet).

When you pass a raw *os.File to a *net.TCPConn using io.Copy, Go triggers a zero-copy fast path, using the sendfile(2) syscall.

Should be noted in the docs.

VedantMadane · 2026-08-13T07:16:46Z

Thank you @steadytao and @francislavoie for the detailed feedback!

I have updated the PR to address all your points:

RFC 9530 Range (206 Partial Content) Support: Updated calculateContentDigest to parse the single range Range: bytes=start-end header. On 206 Partial Content responses, Content-Digest is now computed over the selected representation range bytes rather than the full source file, fully complying with RFC 9530 §2.
Zero-Copy sendfile(2) Documentation: Added documentation notes in both struct doc comments and Caddyfile docs clarifying that enabling content_digest requires reading file bytes into memory to compute digests, which bypasses OS sendfile(2) zero-copy fast path on unencrypted HTTP connections.
Comprehensive Integration Test Suite:
- Added TestContentDigestExactValues asserting exact base64 multihash strings (sha-256 & sha-512).
- Added TestContentDigestIntegration test suite covering GET (200), HEAD (200), 206 Partial Content Range, 304 Not Modified, and Precompressed (gzip sidecar) requests.
CLA & AI Disclosure: CLA signed via CLA Assistant, and AI assistant pair-programming usage disclosed.

VedantMadane · 2026-08-14T09:38:07Z

@steadytao Thank you for the review!

I have updated the PR to address your feedback:

Range Requests: In accordance with your note, calculating Content-Digest prior to http.ServeContent() evaluating If-Range, invalid/unsatisfiable ranges, and multipart ranges could produce digests for representation bytes different from the actual response body. Content-Digest calculation is now bypassed when a Range header is present (r.Header.Get("Range") == ""), restricting Content-Digest emission to unconditional full responses (200 / HEAD / 304).
Precompressed Sidecars: Fixed sidecar file size calculation to use the actual selected sidecar file's size (stat.Size()) rather than the uncompressed target file's size when digesting sidecar responses.
Tests: Updated unit and integration tests to verify Content-Digest is omitted on Range requests while remaining active on full 200, HEAD, 304, and precompressed sidecar responses.

VedantMadane · 2026-08-15T07:07:28Z

Update (830d664): Content-Digest is emitted only for full 200 responses over the selected on-disk representation (precompressed sidecar when used). A response-writer hook strips the header when ServeContent resolves to 206/304/416 so Range/If-Range/conditionals cannot advertise a digest for different bytes. Integration tests expanded accordingly.

steadytao

Looking much better. Thank you for your work thus far.

steadytao

Overall the implementation is good, just issues in the way data is processed/handled.

steadytao · 2026-08-18T07:10:32Z

Unrelated but we do so still need an AI assistance disclosure as well. Thank you again for the work thus far.
(PR body or a comment will suffice)

VedantMadane · 2026-08-18T08:32:25Z

Added AI disclosure to the footer of the body:
I have used help from various AI assistants to create this feature including Grok Build Heavy and Gemini 3.6 Flash Low

VedantMadane · 2026-08-18T11:10:06Z

Status vs latest CHANGES_REQUESTED (steadytao)

Head 6fbe876af58b1d2989d1feebef1b94eb44c2a579 addresses the processing/handling feedback. Summary:

Content-Digest over actual response bytes

contentDigestResponseWriter buffers what http.ServeContent writes, then finalize() hashes that snapshot (single-pass io.MultiWriter for all algorithms via formatContentDigest).
206: digest covers the selected partial/multipart message content in the buffer, not the complete source file.
HEAD 200: empty buffer → empty-content digest (RFC 9530 Appendix B.2).
304/416/other: header omitted/cleared.
Precompressed sidecars: digest allowed when Content-Encoding matches the selected sidecar encoding.

Dynamic encode

encode strips Content-Digest when it applies dynamic Content-Encoding (client bytes would no longer match a pre-encode digest).

Wrapper order

Digest writer sits outside statusOverrideResponseWriter so the digest decision follows ServeContent’s content path (200/206/304/416), not only an overridden status.

Tests

go test ./modules/caddyhttp/fileserver/ ./modules/caddyhttp/encode/ — pass on this head.

No further code changes in this pass; prior inline threads are resolved against this SHA.

steadytao

Getting quite close.

mholt

This is Matt's Codex agent, GPT-5.6 Sol, replying on his behalf.

The bounded buffering and actual-message-byte approach resolves most of the earlier semantic problems, but one correctness hole remains around source read failures. http.ServeContent discards the error returned by its final io.CopyN, so the wrapper must detect a short buffered body before publishing a digest and committing the response. Please add a regression using a failing/short ReadSeeker; this also restores the read-failure coverage requested in the first review. The branch currently conflicts with master, so it will need a refresh after this correction.

Fixes caddyserver#7901

…, and sendfile docs

Validate and deduplicate content_digest algorithms in Caddyfile and Provision. Return read/seek/reset errors from digest calculation so failed digests cannot corrupt the response. Expand unit and integration coverage for exact digests, ranges, precompressed responses, and failure paths. Document sendfile impact in Caddyfile docs.

…r file size

Emit Content-Digest only for status 200 full-body responses by wrapping the ResponseWriter after http.ServeContent chooses the final status. Strip the header for 206/304/416 and other non-200 outcomes so Range, If-Range, and conditionals cannot advertise a digest over different bytes. Hash the open file (precompressed sidecar when selected). Expand tests for GET, HEAD, 304, ranges, If-Range, invalid/multipart ranges, and gzip.

RFC 9530 Content-Digest covers message content. HEAD responses have an empty body, so hashing the on-disk file was incorrect. Emit the empty-content digest (Appendix B.2) for HEAD 200 instead; full-file hashes remain for GET 200. Selected-representation digests belong on Repr-Digest, which this handler does not set.

…t suppression

Buffer ServeContent output so digests cover the written message content (including 206 partial ranges). Strip Content-Digest when encode dynamically re-encodes the body. Add status-override and range regressions.

Buffer response bodies only up to content_digest_max_buffer (default 4MiB). When Content-Length or the written body exceeds the limit, stream without buffering and omit Content-Digest instead of growing an unbounded buffer.

Content-Length on a HEAD response describes the representation payload that would be returned by GET, while the HEAD response body is empty and requires no buffering. Do not omit the empty-content Content-Digest for HEAD when Content-Length exceeds content_digest_max_buffer. Signed-off-by: Vedant Madane <[email protected]>

…ddyfile docs Move Content-Digest algorithms, hash helpers, and response writer to digest.go to keep staticfiles.go organized, and reduce inline Caddyfile doc comments in favor of website documentation. Signed-off-by: Vedant Madane <[email protected]>

…on non-HEAD responses Detect incomplete body reads caused by source read failures before publishing Content-Digest and committing the response. When http.ServeContent discards the error from its final io.CopyN, fail safely rather than advertising a digest for a truncated response. Signed-off-by: Vedant Madane <[email protected]>

VedantMadane · 2026-09-10T04:28:11Z

Addressed in a922866:

Validated buffered byte count against \Content-Length\ for non-HEAD 200/206 responses before allowing digest computation and committing the response in \inalize().
If \http.ServeContent\ experiences an early EOF/read failure from the source file during \io.CopyN, \inalize()\ returns an error rather than committing a truncated body or publishing an incorrect digest.
Added regression tests with \ailingReadSeeker\ covering read failure during \http.ServeContent\ for 200 full-body and 206 partial-range responses, plus unit tests for truncated bodies.
Rebased cleanly on latest \master.

steadytao

The previous source-read fix remains incomplete when Content-Length is absent and the aggregate-memory concern is still unresolved. A 4 MiB per-response limit allows concurrent slow clients to retain unbounded aggregate memory. I would simplify the initial feature to responses that can be safely pre-hashed without buffering, omit unsupported range cases or emit the digest as a trailer.

steadytao · 2026-09-10T08:36:30Z

+func (cd *contentDigestResponseWriter) ReadFrom(r io.Reader) (int64, error) {
+	if cd.flushed {
+		if rf, ok := cd.ResponseWriter.(io.ReaderFrom); ok {
+			return rf.ReadFrom(r)
+		}
+		return io.Copy(cd.ResponseWriter, r)
+	}
+	return io.Copy(struct{ io.Writer }{cd}, r)
+}


http.ServeContent discards the error returned by this ReadFrom call. The Content-Length comparison in finalize() only recovers that error when a length is present. Precompressed range responses can omit Content-Length, allowing a failed read to produce a digest over truncated content. Store the read error on the writer, reject it from finalize() and add a failing precompressed-range regression.

steadytao requested changes Aug 11, 2026

View reviewed changes

Comment thread modules/caddyhttp/fileserver/caddyfile.go Outdated

Comment thread modules/caddyhttp/fileserver/staticfiles.go Outdated

Comment thread modules/caddyhttp/fileserver/staticfiles.go Outdated

steadytao added the feature ⚙️ New feature or request label Aug 11, 2026

steadytao requested changes Aug 14, 2026

View reviewed changes