Use clonefile instead of copyfile on macOS. #10020

filipnavara · 2018-08-10T10:35:20Z

Alternative to #10008.

Note that copyfile is completely user-mode API implemented in /usr/lib/system/libcopyfile.dylib. As you can see in the source code in the copyfile_data method it does exactly the same read+write loop as Mono and offers no performance benefits by itself.

Someone in Apple decided to break the copyfile API in macOS 10.13.2 by cloning symlinks instead of targets despite the fact that passed in flags do not include COPYFILE_NOFOLLOW_SRC. The reason for the change is also completely bogus - we don't support copying directories, so let's not follow symlinks because they could point to directories. Well, just before calling clonefile they check if the source is a link, so they could have skipped cloning symlinks instead.

Since the broken copyfile is apparently in the wild already the only way to fix it is to drop using it for cloning. And since cloning was the only performance benefit of the API it makes sense to drop its usage entirely.

filipnavara · 2018-08-10T13:00:10Z

I went through the build failures and they seem to be unrelated.

kumpera

I haven't reviewed the large changes to the IO code itself.
There are some bits that require fixing/changing.

kumpera · 2018-08-10T16:02:20Z

mono/metadata/w32file-unix.c

+#endif
+	}
+
+#if __APPLE__


this should use the HOST_ defines

I'm fine with changing it, but there were already other pieces of code using __APPLE__, so I just followed whatever was already there.

HOST_DARWIN?

@filipnavara yes.

kumpera · 2018-08-10T16:03:14Z

mono/metadata/w32file-unix.c

 	finds = g_hash_table_new_full (g_direct_hash, g_direct_equal, NULL, finds_remove);
 	mono_coop_mutex_init (&finds_mutex);

+#if __APPLE__


This is should use HOST_ defines.

kumpera · 2018-08-10T16:04:19Z

mono/metadata/w32file-unix.c

 	mono_coop_mutex_init (&finds_mutex);

+#if __APPLE__
+	clonefile_ptr = (clonefile_fn)dlsym (RTLD_DEFAULT, "clonefile");


AFAICT this won't work on iOS or watchOS. @rolfbjarne can you give us a more authoritative answer here?

We must not use RTLD_DEFAULT as it will cause us all sorts of trouble with embeders and fullAOT scenarios.

I'd appreciate any suggestions on how to fix it. I assumed RTLD_DEFAULT is ok since it's a system API (libc, or libsystem respectively).

Something like

libc_handle = dlopen("/usr/lib/libc.dylib", RTLD_NOW); clonefile_ptr = (clonefile_fn)dlsym (libc_handle, "clonefile");

?
How about iOS? I'd be fine with supporting macOS only if dlsym is a problem.

kumpera · 2018-08-10T16:05:38Z

mono/metadata/w32file-unix.c

 #ifdef HAVE_DIRENT_H
 # include <dirent.h>
 #endif
+#if __APPLE__


Shouldn't this be a HAVE_DLFCN_H define?

Technically yes, but the assumption was that all Apple platforms have it anyway.

luhenry · 2018-08-10T21:15:13Z

mono/metadata/w32file-unix.c

 {
-	gchar *located_from, *located_to;
-	int ret;
+	if (clonefile_ptr != NULL) {


Since you check clonefile_ptr before calling _wapi_clonefile, you can just g_assert(clonefile_ptr) here.

Changed to assert as you suggested.

jaykrell · 2018-08-10T22:49:41Z

mono/metadata/w32file-unix.c

-	MONO_EXIT_GC_SAFE;
+		MONO_ENTER_GC_SAFE;
+		ret = clonefile_ptr (
+			located_from == NULL ? from : located_from,


To avoid worrying about precedence I like to reverse this:
foo ? foo : other

But I realize this is how it was.

I prefer to keep it unchanged to keep the diff smaller. I'll remember your advice when writing new code.

filipnavara · 2018-08-13T13:53:29Z

@monojenkins build failed

luhenry · 2018-08-14T18:14:17Z

@kumpera could you please take another look at it?

kumpera · 2018-08-14T19:32:44Z

mono/metadata/w32file-unix.c

+#if HOST_DARWIN
+	libc_handle = dlopen ("/usr/lib/libc.dylib", 0);
+	g_assert (libc_handle);
+	clonefile_ptr = (clonefile_fn)dlsym (RTLD_DEFAULT, "clonefile");


We're not using libc_handle I don't get why are we dlopen'ing it.

Uh, my fault, should be used instead of RTLD_DEFAULT

kumpera

LGTM

luhenry · 2018-08-15T03:53:09Z

@monojenkins backport to 2018-08

luhenry · 2018-08-15T03:53:17Z

@monojenkins backport to 2018-06

monojenkins · 2018-08-15T03:54:15Z

@luhenry backporting to 2018-06 failed, the patch results in conflicts:

Applying: Use clonefile instead of copyfile on macOS.
Using index info to reconstruct a base tree...
M	mono/metadata/w32file-unix.c
Falling back to patching base and 3-way merge...
Auto-merging mono/metadata/w32file-unix.c
CONFLICT (content): Merge conflict in mono/metadata/w32file-unix.c
error: Failed to merge in the changes.
Patch failed at 0001 Use clonefile instead of copyfile on macOS.

Please backport manually!

* Use clonefile instead of copyfile on macOS. * Address PR review comments. * Address PR review comments (if -> assert). * RTLD_DEFAULT - > libc_handle

@filipnavara

[2018-06] Use clonefile instead of copyfile on macOS * Revert of #10010 * Backport of #10020 Fixes #10447 /cc @filipnavara

* Use clonefile instead of copyfile on macOS. * Address PR review comments. * Address PR review comments (if -> assert). * RTLD_DEFAULT - > libc_handle Commit migrated from mono/mono@583bae5

Use clonefile instead of copyfile on macOS.

385e7d5

filipnavara requested a review from luhenry as a code owner August 10, 2018 10:35

filipnavara changed the title ~~WIP: Use clonefile instead of copyfile on macOS.~~ Use clonefile instead of copyfile on macOS. Aug 10, 2018

luhenry mentioned this pull request Aug 10, 2018

[w32file] Fix usage of copyfile #10008

Closed

kumpera suggested changes Aug 10, 2018

View reviewed changes

Address PR review comments.

d95be82

luhenry suggested changes Aug 10, 2018

View reviewed changes

Address PR review comments (if -> assert).

2a4a4ff

jaykrell reviewed Aug 10, 2018

View reviewed changes

luhenry approved these changes Aug 14, 2018

View reviewed changes

kumpera suggested changes Aug 14, 2018

View reviewed changes

RTLD_DEFAULT - > libc_handle

16153e7

kumpera approved these changes Aug 14, 2018

View reviewed changes

luhenry merged commit 583bae5 into mono:master Aug 15, 2018

monojenkins mentioned this pull request Aug 15, 2018

[2018-08] Use clonefile instead of copyfile on macOS. #10119

Merged

lewurm mentioned this pull request Sep 3, 2018

[iOS] FileTest.LastWriteTimeSubMsCopy fails #10447

Closed

lewurm mentioned this pull request Sep 3, 2018

[2018-06] Use clonefile instead of copyfile on macOS #10455

Merged

monojenkins added a commit that referenced this pull request Sep 5, 2018

Merge pull request #10455 from lewurm/2018-06-osx-clonefile

9439ee6

[2018-06] Use clonefile instead of copyfile on macOS * Revert of #10010 * Backport of #10020 Fixes #10447 /cc @filipnavara

rolfbjarne mentioned this pull request Oct 29, 2018

Clone files if possible whenever it makes sense. dotnet/macios#5049

Merged

akoeplinger mentioned this pull request Apr 15, 2019

Bring back optimized file copy on macOS #14059

Open

adamsitnik mentioned this pull request Nov 3, 2022

File.Copy: Clone file when possible on MacOS dotnet/runtime#77835

Closed

Use clonefile instead of copyfile on macOS. #10020

Use clonefile instead of copyfile on macOS. #10020

Uh oh!

Conversation

filipnavara commented Aug 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

filipnavara commented Aug 10, 2018

Uh oh!

kumpera left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

filipnavara Aug 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

filipnavara commented Aug 13, 2018

Uh oh!

luhenry commented Aug 14, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kumpera left a comment

Choose a reason for hiding this comment

Uh oh!

luhenry commented Aug 15, 2018

Uh oh!

luhenry commented Aug 15, 2018

Uh oh!

monojenkins commented Aug 15, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

filipnavara commented Aug 10, 2018 •

edited

Loading

filipnavara Aug 10, 2018 •

edited

Loading