-
-
Notifications
You must be signed in to change notification settings - Fork 4.8k
fs: accept directory names in exclude-if-present #8746
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
ncw
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This would also need to change ListContainsExcludeFile I think.
These could do with tests and docs also, and the help for --exclude-if-present would need adjusting.
I am slightly concerned that this is a backwards incompatible change though.
Perhaps we should require directories in --exclude-if-present to have trailing / so you'd write --exclude-if-present .ignoreFile --exclude-if-present .ignoreDir/ which would fix the backwards incompatibility problem.
What do you think?
I'm not sure of the changes there. I suppose
Makes a lot of sense. Added this.
I'm happy to make those changes after we agree on an implementation :) |
ncw
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks like the right approach now (see notes inline).
Please carry on with docs and tests - thank you :-)
| if exists { | ||
| return true, nil | ||
| } | ||
| if err == fs.ErrorIsDir { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks like the right approach. The logic needs to go like this though otherwise it ignores errors on the exists check
if err != nil {
if errors.Is(err, fs.ErrorIsDir) && strings.HasSuffix(excludeFile, "/") {
return true, nil
}
return false, err
}
if exists {
return true, nil
}There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
above errors out in case exclude file doesn't have a / but name corresponds to a dir.
Eg:
test2
├── node-project
│ └── lol
└── python-project
└── .venv/
rclone ls /Users/neo/tmp/test2 --exclude-if-present '.venv' --exclude-if-present '.git'
2025/08/30 19:05:19 ERROR : error listing: is a directory not a file
2025/08/30 19:05:19 NOTICE: Failed to ls with 2 errors: last error was: is a directory not a file
I wonder if we should even check for / suffix, since, rclone just crashes if it encounters a ignore-file name that exists as a directory in the fs.
Anyone relying on this behaviour is unlikely.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ha, yes you are right. Existing rclone has a bug! I was trying to preserve the existing behaviour, but let's fix the bug
So that makes your original solution nearly right but I think we need to add errors.Is, so
- if err == fs.ErrorIsDir {
+ if errors.Is(err, fs.ErrorIsDir) {
return strings.HasSuffix(excludeFile, "/"), nil
}Can you do that then add tests and docs (in docs/content/filtering.md also, and the help for --exclude-if-present would need adjusting.
Line 36 in 9e20053
| Help: "Exclude directories if filename is present", |
Maybe something like
"Exclude directories if file or dir/ is present"
Thanks
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"Exclude directories if file or dir/ is present"
given it was a bug, do we even require the trailing / for directories ? most of the other commands do not distinguish against file/dir in this manner, (say copy).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the consensus is --exclude-if-present file and --exclude-if-present dir/.
Do you want to finish of the PR assuming that and we can get it merged - thank you
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I agree with the reasoning by @nielash. I'm a little occupied with my day job but I do intend to finish the rest of it soon.
|
I think ListContainsExcludeFile needs to look something like this (untested) // ListContainsExcludeFile checks if exclude file is present in the list.
func (f *Filter) ListContainsExcludeFile(entries fs.DirEntries) bool {
if len(f.Opt.ExcludeFile) == 0 {
return false
}
for _, entry := range entries {
basename := path.Base(entry.Remote())
switch entry.(type) {
case fs.Object:
if slices.Contains(f.Opt.ExcludeFile, basename) {
return true
}
case fs.Directory:
if slices.Contains(f.Opt.ExcludeFile, basename+"/") {
return true
}
}
}
return false
} |
|
The options
What I normally do here is look for prior art to see what other programs do. Here is a selection of programs with an
There isn't a strong consensus for whether files and directories should be treated differently or not. Given the fact that currently if If we choose option 2) if you want files and directories you can always add @albertony @nielash do you have an opinion? |
I lean slightly towards option 2, for a few reasons.
I hear the point about Also, it may or may not be worth revisiting #4523 at the same time. |
I agree - with everything you said there! |
What is the purpose of this change?
Fixes: #8745
Was the change discussed in an issue or in the forum before?
Forum: https://forum.rclone.org/t/accept-directory-names-in-exclude-if-present/52248/1
Issue: Fixes: #8745
Checklist