Just wanted to come back and add some more data. I did some perf testing on an 8 core machine, same checkouts. 60-70% perf improvement there. :D

So we were doing some testing here and super duper finally discovered the slowdown on Windows machines. Windows Defender is the culprit. When we disable Windows Defender, performance on Windows machines runs near the same speed as a Linux or OS/X box. Now that said, Defender or an equivalent is often enabled in corporate environments, so this optimization is still a good thing. We still see crazy perf gains for those with active virus protection...

That said, since we've found the real culprit and it's not actually NTFS... We're not just optimizing the Windows pathway anymore, we're really just optimizing checkouts in general. So I'm removing the flag that restricts this behavior for Windows only, because we could still see perf increases in other environments.

I'm curious to know what kind of behavior changes you see in non-Windows environments?

I just found I needed to share the index mutex over the git_filter_list__load_ext and ditch the filter_mutex. It seems somewhere deep down in git_filter_list__load_ext we end up touching the index, and I managed to get a crash there.

For Linux we see about a 10-20% increase in speed from previous versions of libgit2 using the same test branches.

Libgit2 is still slower than the CLI on Linux. Even after threading, CLI checkout performance is outstanding on Linux. After threading we are 100-200ms slower than the CLI, and about 30-60% slower than the CLI, but everything post threading in our tests with a large checkout operation is under 1 second. So there is more room for us to optimize the checkout code, but it would probably be chasing some minor perf gains in the grand scheme of things. Most of those performance tweaks would be more impactful on Windows machines than Linux.

I wonder if instead of locking the index on the whole filter list if we could utilize the git_attr_session to pass the index mutex down to where the index is actually being used. @ethomson thoughts?

This seems to be failing because of the bitbucket issue in PR #4584

We've been running this in GK through nodegit for almost 9 months now without seeing any craziness.

Force pushed to fix some conflicts from the git_buf_free -> git_buf_dispose change. Also, we'll need to continue cherrypicking this in NodeGit for the time being.

Due to how long this PR has been open, the history in the branch has been merged into a single commit, as it is much easier to rebase this branch as libgit2 changes.

@ethomson @tiennou @pks-t @carlosmn I know I've been bugging you a lot lately this week, but might I bug you one more and ask if anyone has time for this change 😄

(accesses to git_pool seems to also be a concern, as I don't think those are threadsafe)

Thank you for the review @tiennou! I apologize for the lack of commits as a story telling mechanism. Now that this is receiving attention, again, when I am reconciling your review, I will try to rebuild the patches to help make this review more digestable.

Thank you for the review @tiennou! I apologize for the lack of commits a story telling mechanism. Now that this is receiving attention, again. When I am reconciling your review, I will try to rebuild the patches to help make this review more digestable.

Another option is to take bits of this PR and merge them opportunistically, as it makes sense. I could imagine bringing in some refactoring, etc. This is sort of what we did for the original merge PR... I think that would be pretty straightforward if that was a direction that you wanted to go instead.

Just throwing that out there; I'm happy either way.

So while these numbers are not checkout in isolation, these are some numbers that the GitKraken team took after multithreading checkout in GitKraken.

In these documents, for reference, 2.4 GK refers to checkout without threading and 2.5 GK RC3 refers to checkout operations with threading.

Performance improvements seen on Windows when multithreading checkout

Performance improvements seen on Linux when multithreading checkout

Unfortunately, the repository we used is an old, large, private repo for the Axosoft product line, so I cannot share the repo that helped to create these numbers. You'll have to trust that this particular checkout operation is very large 😄.

I do not have Mac OS numbers at this moment, so it probably would be good to redo these numbers anyway in a more isolated form later.

I've put out my patchset here : master...tiennou:feature/threaded-checkout-clean.

It's not quite pretty enough, but I think it makes what's going on more isolated (at least to me 😉).

@tiennou I think we need to be careful in wd_item_is_removable to dispose the fullPath regardless of the success or failure of build_target_fullpath in your patch set. Perhaps we should be failing that method of git_buf_set fails as well?

@ethomson @tiennou Ok so I ran into some annoying conflicts early last week with this branch. I decided I would take a stab at cleaning up some of the items that needed work for this PR.

The major change is that we're based on #5602, because it allows us to easily address some of @ethomson's concerns he had expressed to me over performance regressions in the single threaded case due to over allocations of the tmp / target_path buffers.

The other changes are:

• perfdata now just uses a simple lock before writes
• completed_steps needed a lock to make it thread safe
• I've added a separate mutex for the git_filter_list__load_ext, as I am reasonably certain that it is the attribute cache on the repository that makes that method not threadsafe.

I've also tried hard to keep each commit easily digestible. Swapping the temporary buffers to use TLS has immeasurably reduced the footprint of what needs to be reviewed for this PR.

Now on to what I have not implemented yet.. I have not gone through and added this as configurable in an option (whether on checkout_options or in libgit2_opts), my hold up here is on testing. Checkout is such an integral part of many other things you can do in libgit2 that I want anything that could potentially run checkout to run in threaded mode during testing of the library... I was also thinking that whether you thread checkout or not seems like a library option rather than an individual checkout option, because from the standpoint of a user I want to roll out the performance benefits to every checkout that the library does without having to touch every possible endpoint of checkout.

So what I would like to do is just duplicate the offline checkout tests and at least one of the online tests that involves cloning and turn on checkout threading on those runs.. I also think, because of the nature of threading, we should require the offline checkout tests to run 4 times per test cycle to hopefully catch timing errors that could crop up someday.

I would really appreciate it if you guys could get back to me soon on this one, I would like to add the option and get this merged in asap - I would prefer not to deal with another merge conflict from this area.

I don't mean to harp, but we've been running this code for years in GitKraken and for users of NodeGit and haven't seen anything concerning from a stability or correct-output standpoint. I firmly believe that had there been any of those issues we would have seen a report of it by this point with how large of a rollout we've done on this and for how long it's been running in production. To set aside my hubris there, I suppose it's possible that the platforms we target have more favorable conditions for any threading issues though.

there are many places where we might be stepping all over our own threading expectations (ie. you should never share git_repository instances between threads, so I shudder each time I see that checkout_data parameter), and auditing for that is… complicated

(accesses to git_pool seems to also be a concern, as I don't think those are threadsafe)

@tiennou I think in general, the reason the pool is working correctly is that we are locking around its use in mkpath2file -> checkout_mkdir. I could alleviate your fears by moving the pool into TLS.

The only other place that I see that is potentially dangerous in an audit (while I'm fresh on this again) is access to the ODB in checkout_write_content where we are performing a lookup of a blob there, as well as freeing it. Should we be locking on blob lookup and free?

I know I've still got ergonomics to do on this, but seeing the fuzzer agree with this branch makes me feel better about it.