fileserver: parallelize directory listing to speed up large-directory browsing - #7933

Summary

file_server browse listings were slow for directories with many entries.
directoryListing() called entry.Info() for every entry sequentially - on Unix that's a real Lstat syscall per entry (Windows already gets this for free from FindNextFile), plus a second Stat (and optional Readlink) for every symlink. A directory with thousands of files paid thousands of sequential syscall round-trips.

This splits the work into three passes and parallelizes the expensive one:

1. Filter (sequential) — drop hidden entries, no syscalls involved.
2. Stat (parallel) — a bounded pool of workers (sync.WaitGroup.Go, Go 1.25+) each stat a contiguous chunk of the remaining entries. Each worker owns disjoint indices into a preallocated results slice, so there's no shared mutable state and no locking. Workers bail out on ctx.Err() per item, matching the previous cancellation behavior.
3. Aggregate (sequential) — sum NumDirs/NumFiles/TotalFileSize/TotalFileSizeFollowingSymlinks and build Items from the results. Cheap, no syscalls.

Final Items order doesn't matter here since applySortAndLimit always resorts before any output format (JSON/text/HTML) uses it.

Worker count defaults to 16 (browse.go's defaultDirListingConcurrency), deliberately conservative rather than tied to GOMAXPROCS: this is a syscall/IO-bound workload (Go already scales OS threads for blocked syscalls independent of GOMAXPROCS), and the bound is per request, not global — nothing caps concurrent syscalls across simultaneous browse requests, so a high default would multiply badly under concurrent load. It's now also configurable per-site:

file_server {
    browse {
        concurrency 32
    }
}

(0/unset uses the built-in default.)

RevealSymlinks now also checks whether the fs.FS implements a new readLinkFS interface (ReadLink(name string) (string, error)) before falling back to os.Readlink, so custom filesystem modules can resolve symlinks their own way instead of assuming an OS path.

Benchmark

go test -bench=BenchmarkDirectoryListing -benchmem -run '^$' ./modules/caddyhttp/fileserver/, real OS-backed directories (existing BenchmarkDirectoryListing harness), before vs. after, ns/op:

Allocations rise modestly (e.g. ~56KB → ~80KB at 100 entries) from goroutine/closure overhead — an accepted tradeoff for the wall-clock win, and there's no regression even at small directory sizes since worker count is capped at min(concurrency, len(visible)).

Assistance Disclosure

I used Claude Code to design and implement this change (directory-listing parallelization, the Concurrency config option and its Caddyfile syntax, and the new test), working from my own investigation that large-directory listings were slow. I reviewed the design, benchmarked it, and made additional changes myself on top (the readLinkFS symlink-resolution interface, and the concurrencySet fix in the Caddyfile parser).