@monojenkins backport to 2018-06

@monojenkins backport to 2018-08

@luhenry backporting to 2018-06 failed, the patch results in conflicts:

Applying: [w32file] Fix usage of copyfile
Using index info to reconstruct a base tree...
M	mono/metadata/w32file-unix.c
Falling back to patching base and 3-way merge...
Auto-merging mono/metadata/w32file-unix.c
CONFLICT (content): Merge conflict in mono/metadata/w32file-unix.c
error: Failed to merge in the changes.
Patch failed at 0001 [w32file] Fix usage of copyfile

Please backport manually!

@monojenkins backport to 2018-06

@monojenkins backport to 2018-08

@filipnavara could you please have a look at that since you authored #9642

It is definitely an issue, but removing COPYFILE_CLONE kinda defeats the whole purpose of using copyfile in the first place. There's no unit test covering this specific use case, which would be really nice to have. I am fine with reverting my PR or applying your patch for now, but a proper fix would be to detect and handle symlinks and either follow them or drop COPYFILE_CLONE in that specific case. There's already some stat called earlier in the function, so it should not be too hard to detect the case.

Apparently it used to work correctly until Apple "fixed" it in 10.13.2:

This change was intentional as following symlinks allows you to clone directories (which is disallowed due to potential system slowdown in apfs). We will update the man page to reflect this.

This is not a bug in clonefileat(), this was an intentional change in copyfile itself.

I was looking into using clonefile directly to get around all the copyfile quirks. However, the API is only available on newer macOS versions and thus hard to consume if down-level compatibility is desired with the same binaries.

We should be able to have a check for clonefile in our configure.ac, use it if available, fallback to copyfile (without COPYFILE_CLONE) otherwise. I am pretty sure copyfile is still going to be faster than our handrolled read+write just because of the absence of userspace<->kernelspace memory copy.

And looking at the clonefile man page, it seems like it's no silver bullet either since not all volumes (filesystem I assume) support it either. So a fallback to copyfile will be necessary nonetheless.

copyfile does not really help performance-wise compared to read+write. The whole code is open source (https://opensource.apple.com/source/copyfile/copyfile-146.30.2/) and the actual copying is still just read+write. In addition it can do all the nice things like copying extended attributes, but that's not why it was used in the first place.

clonefile is not a silver bullet, but macOS now uses apfs on all system drives, so it will work there. The hope was to avoid the copy overhead during msbuild runs that do a lot of copying. Using configure to check for it is not an option unfortunately. Mono distributes single package for all macOS versions, so it would have to be used through dynamic loading and with proper fallbacks.

I'm working on a PR with clonefile. If everything goes smooth I will submit it later today.

I definitely agree clonefile would be better than copyfile, that's a given. What I meant is clonefile might not be working everywhere, so we still require the fallback to copyfile. I'll take a look at #10020 right now.