chunker: fix single-file operations over union remotes - #9897
Open
git-jxj wants to merge 1 commit into
Open
Conversation
Opening a chunker file over union could select only the upstreams containing its metadata or first chunk, hiding chunks on other upstreams. Directory operations still worked, but individual reads, listings and moves could fail or return metadata instead of the file contents. Detect files through the complete parent filesystem and configure the chunker against the final base, including metadata-free files. Retain the standard filesystem cache lifecycle and the underlying backend's normal creation path when the parent does not confirm a file. Add regression coverage for both metadata modes, configured and nested roots, metadata-only upstreams, distributed chunks and repeated opens. Fixes rclone#7955 Signed-off-by: git-jxj <[email protected]>
Author
|
Update: build run 34450111859, attempt 2, is now |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change do?
Fix single-file listing, copying and moving when chunker wraps a union whose chunks are distributed across upstreams. Opening the union at the metadata file (or first chunk) can restrict it to the upstream containing that file, leaving other chunks inaccessible even though directory reads work.
Detect a file through its cached parent and keep that complete parent as chunker's base. Configure hashing, transaction mode and features from the final base. This also handles metadata-free files and preserves the standard cache lifecycle and path handling. The extra parent-object probe is confined to chunker; ordinary union single-file behavior is unchanged.
Add eight regression cases covering both metadata formats, configured and nested roots, metadata-only upstreams, distributed chunks, repeated file opens and directory reads. All eight fail against the unmodified production code and pass with this change.
This contribution was implemented and tested with OpenAI Codex assistance.
Linked issue
Fixes #7955
The issue includes the maintainer's reproduction and analysis.
For new or changed backends
go build: passed.go test ./backend/chunker ./backend/union/... -count=1: passed.make quicktest: passed as an unprivileged user. FTP loopback tests ran in a private network namespace to avoid host Kubernetes NodePort conflicts.go run ./fstest/test_all -backends chunker,union -tests backend -config <local-test-config> -n 2 -maxtries 1: 4/4 passed, against chunker/union, chunker/local, metadata-free chunker/local and ordinary union, all backed by temporary local directories.test_allmatrix encounters an existing bisync setup failure on chunker/union:sync.CopyDirreturnsobject not foundwhile setting uppath1. This is reproduced on unmodified masteref6968730fc5582aa2a1db028440f3acc48b4e27with the same configuration and unprivileged user usinggo test ./cmd/bisync -run '^TestBisyncRemoteLocal$/^all_changed$' -remote TestChunkerUnion: -count=1 -v. The extra full-matrix attempt was stopped after confirming this baseline failure; a complete full-matrix pass is not claimed.copyto/movetofail. All commands pass after the fix. An independent run with 5 MiB + 137 bytes of random data passes listing, copy, move and renamed-file copy with matching SHA-256 hashes and removal of the old remote file/chunks.golangci-lint run --new-from-rev ef6968730fc5582aa2a1db028440f3acc48b4e27: 0 issues. Full lint reports the same existingrevivewarning atfs/log.go:17on both the unmodified base and this patch.The integration remotes use existing backends and temporary local storage; no cloud account or new backend is involved.
Checklist
test_allpasses for this backend and if submitting a new backend can provide a test account for the integration tester - see CONTRIBUTING.md.