chunkedreader: fix parallel reader hanging forever when a stream fails - #9901
Open
diivfx wants to merge 1 commit into
Open
chunkedreader: fix parallel reader hanging forever when a stream fails#9901diivfx wants to merge 1 commit into
diivfx wants to merge 1 commit into
Conversation
When one of the parallel streams failed part way through its chunk - for example because the object was deleted or replaced under an rclone mount while it was being read - stream.read never looked at the stream's error and kept looping on WaitWrite for data that would never arrive. parallel.Read also declared nn, err := inside its loop, so even a surfaced stream error was dropped and the caller received (0, nil) indefinitely. With --vfs-cache-mode full this left the VFS downloader stuck, and a rename of a directory containing such a file hung the mount: the rename waited in Downloaders.Close on the async reader, which waited in stream.read. A stream whose background read has finished now drains what it had buffered and then returns the error it finished with, and parallel.Read passes that error on to the caller. The new test uses an object whose stream fails mid-chunk and hangs without this change. Co-Authored-By: Claude Fable 5.1 <[email protected]>
diivfx
force-pushed
the
fix-parallel-stream-error-hang
branch
from
September 10, 2026 17:13
87487dc to
2dbdcb6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change do?
Fixes the parallel chunked reader (
--vfs-read-chunk-streams > 0) hanging forever when one of its streams fails part way through a chunk. On an S3 mount with--vfs-cache-mode fullthis showed up as a directory rename that never returned while a file in that directory was being read (details and goroutine dump in #9900).Two changes in
fs/chunkedreader/parallel.go:stream.readnow notices when the stream's background read has finished. It drains what was buffered and then returns the error the stream finished with, instead of looping onWaitWritefor data that will never arrive.parallel.Readdeclarednn, err :=inside its loop, so a stream error never reached thereturn n, errafter the loop and the caller got(0, nil)repeatedly. It now assigns toerr.TestParallelStreamErrorDoesNotHanguses an object whose stream fails mid-chunk; it hangs without the change.Tested with
go build,go test -race ./fs/chunkedreader/ -count=3, andmake quicktest(178 packages pass;cmd/nfsmountandfs/loggerfail identically on unpatched master in this environment, no NFS mount available). On the S3 mount from the issue, the rename-while-reading case now completes and the readers get an error instead of hanging; repeated a few dozen times.Linked issue
Fixes #9900
Checklist
test_allpasses for this backend and if submitting a new backend can provide a test account for the integration tester.