Thanks to visit codestin.com
Credit goes to github.com

Skip to content

chunker: fix single-file operations over union remotes - #9897

Open
git-jxj wants to merge 1 commit into
rclone:masterfrom
git-jxj:git-jxj/fix-chunker-union-single-file
Open

chunker: fix single-file operations over union remotes#9897
git-jxj wants to merge 1 commit into
rclone:masterfrom
git-jxj:git-jxj/fix-chunker-union-single-file

Conversation

@git-jxj

@git-jxj git-jxj commented Sep 10, 2026

Copy link
Copy Markdown

What does this change do?

Fix single-file listing, copying and moving when chunker wraps a union whose chunks are distributed across upstreams. Opening the union at the metadata file (or first chunk) can restrict it to the upstream containing that file, leaving other chunks inaccessible even though directory reads work.

Detect a file through its cached parent and keep that complete parent as chunker's base. Configure hashing, transaction mode and features from the final base. This also handles metadata-free files and preserves the standard cache lifecycle and path handling. The extra parent-object probe is confined to chunker; ordinary union single-file behavior is unchanged.

Add eight regression cases covering both metadata formats, configured and nested roots, metadata-only upstreams, distributed chunks, repeated file opens and directory reads. All eight fail against the unmodified production code and pass with this change.

This contribution was implemented and tested with OpenAI Codex assistance.

Linked issue

Fixes #7955

The issue includes the maintainer's reproduction and analysis.

For new or changed backends

  • go build: passed.
  • go test ./backend/chunker ./backend/union/... -count=1: passed.
  • make quicktest: passed as an unprivileged user. FTP loopback tests ran in a private network namespace to avoid host Kubernetes NodePort conflicts.
  • go run ./fstest/test_all -backends chunker,union -tests backend -config <local-test-config> -n 2 -maxtries 1: 4/4 passed, against chunker/union, chunker/local, metadata-free chunker/local and ordinary union, all backed by temporary local directories.
  • The additional full test_all matrix encounters an existing bisync setup failure on chunker/union: sync.CopyDir returns object not found while setting up path1. This is reproduced on unmodified master ef6968730fc5582aa2a1db028440f3acc48b4e27 with the same configuration and unprivileged user using go test ./cmd/bisync -run '^TestBisyncRemoteLocal$/^all_changed$' -remote TestChunkerUnion: -count=1 -v. The extra full-matrix attempt was stopped after confirming this baseline failure; a complete full-matrix pass is not claimed.
  • CLI reproduction: with a naturally distributed 5 MiB file, directory copying succeeds before the fix while single-file listing is empty and copyto/moveto fail. All commands pass after the fix. An independent run with 5 MiB + 137 bytes of random data passes listing, copy, move and renamed-file copy with matching SHA-256 hashes and removal of the old remote file/chunks.
  • golangci-lint run --new-from-rev ef6968730fc5582aa2a1db028440f3acc48b4e27: 0 issues. Full lint reports the same existing revive warning at fs/log.go:17 on both the unmodified base and this patch.

The integration remotes use existing backends and temporary local storage; no cloud account or new backend is involved.

Checklist

  • This change is trivial OR it has been discussed and agreed in the linked issue.
  • I have read the contribution guidelines.
  • (If I used AI tools to help write this code) I have read and understood the AI-assisted contributions guidance, and I have tested and take ownership of this change myself.
  • I have added tests for all changes in this PR if appropriate.
  • I have added documentation for the changes if appropriate. (No configuration or user-facing documentation change is needed.)
  • All commit messages are in house style.
  • (Backend changes only) test_all passes for this backend and if submitting a new backend can provide a test account for the integration tester - see CONTRIBUTING.md.
  • This Pull Request is ready for review.

Opening a chunker file over union could select only the upstreams
containing its metadata or first chunk, hiding chunks on other upstreams.
Directory operations still worked, but individual reads, listings and
moves could fail or return metadata instead of the file contents.

Detect files through the complete parent filesystem and configure the
chunker against the final base, including metadata-free files. Retain the
standard filesystem cache lifecycle and the underlying backend's normal
creation path when the parent does not confirm a file.

Add regression coverage for both metadata modes, configured and nested
roots, metadata-only upstreams, distributed chunks and repeated opens.

Fixes rclone#7955

Signed-off-by: git-jxj <[email protected]>
@git-jxj

git-jxj commented Sep 11, 2026

Copy link
Copy Markdown
Author

Update: build run 34450111859, attempt 2, is now in_progress for current head ef5e8e917cff. The earlier action_required approval block has been cleared. Waiting for the build and test results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cannot open single file chunked across different union remotes

1 participant