Submodule optimization #4016

novalis · 2016-11-24T06:56:00Z

The bulk of this work was done by @bpeabody. I just made some minor adjustments. This fixes #3756.

novalis · 2016-11-28T19:47:44Z

So, I know it's kind of audacious for me to request a merge with both CI systems failing, but honestly I think neither of these failures are my fault.

The same AppVeyor test failures appear on a few other recent pull requests:
https://ci.appveyor.com/project/libgit2/libgit2/build/2906/job/6da4tyuhgrdkgsb2
https://ci.appveyor.com/project/libgit2/libgit2/build/2903/job/1hfndyy74s5y2hdn

I also think the segfault on Travis is not my fault. The tests all pass for me locally, and give no warnings under valgrind's memcheck. There are, however, helgrind errors.

Here are a few other relatively recent cases of segfaults (on AppVeyor rather than Travis, but if the cause is what I think it is, it could appear on any platform).
https://ci.appveyor.com/project/libgit2/libgit2/build/2630/job/l4r12jcb4e42ur89
https://ci.appveyor.com/project/libgit2/libgit2/build/2543/job/vdmkmg8pg02xaei8

I can repro the segfault under a version of libgit2 immediately prior to my changes -- simply running the test suite a few dozen times is sufficient. The segfault is in attr_cache.free from git_repository.cleanup, which is one of the areas that helgrind flags as hinky.

Sadly, the bug never seems to reproduce itself under valgrind (or even gdb), but since helgrind is complaining, it seems probably threading-related.

Helgrind does not, afaict, complain about our new code.

novalis · 2016-11-28T22:16:01Z

I just did a no-op amend and now Travis passes. AppVeyor is still bad, but as I noted, it frequently fails this way.

novalis · 2016-11-29T16:13:37Z

Actually, after talking to some colleagues, I decided to rename the functions. No substantive changes.

novalis · 2016-12-02T17:07:27Z

BTW this fixes #3756

ethomson · 2016-12-04T16:06:30Z

I mentioned in slack that this is probably not in scope for this release, which we're just finishing up.

There's not a lot of documentation around how one might want to use this - when would I, as a consumer, need to clear the cache? Would I then benefit from repopulating it?

I think that @carlosmn asked in #3756 - why can this not be a thing that the library handles for you? Why would I, as a consumer, have better knowledge of when caching should happen than the library itself?

bpeabody · 2016-12-04T16:39:42Z

@ethomson I can think of a couple of ways the library could handle it for you.

The library could do caching automatically, invalidating it as necessary, etc.
Library operations (such as the diff that happens in the beginning of a rebase) that will access all submodules could be changed to load them in a batch operation so that caching isn't necessary.

bpeabody · 2016-12-04T16:46:40Z

Additionally, we could probably separate into a different pull request the performance fixes that don't involve a cache (e.g., s.t. all_submodules isn't O(N^2)).

novalis · 2016-12-05T19:59:29Z

Looking at the automatic caching idea:

The idea of manual caching is that the user sometimes knows better than libgit2 when it is safe to use the cache.

In the absence of the cache, each git_submodule_lookup checks the index, HEAD, and the working directory. The index check does not usually involve any i/o (because it is cached). The HEAD and wd checks do. This is somewhat inconsistent: why do we assume that HEAD could have changed out from under us, but not make that assumption about the index?

User code (e.g our git-meta, via nodegit) might know that it's going to open a repository once, do a thing, and then close it. So it's OK for that code to turn on caching as soon as it opens the module.

But the Atom editor (for instance) might not be wiling to make that assumption; maybe at some point you go in and checkout a new branch from the command-line. (I don't actually know if Atom uses this code -- it's just an example). And at that point, the cache needs invalidation.

More generally, git_submodule_lookup is a user-visible function. Imagine that it did do automatic caching, with 100% correct invalidation (modulo the index issue mentioned above). This would mean most every call to `git_submodule_lookup would have to do six iops (open HEAD, read HEAD, open refs/heads/master, read refs/heads/master, stat .gitmodules (to check whether it's clean), stat the submodule's wd path). The only thing the cache would be saving would be parsing .gitmodules, which admittedly is somewhat expensive. Which is more expensive depends on the size of the gitmodules file and the ratio of disk speed to cpu/memory speed.
We could also have git_submodule_lookup not expose a user-visible way of manipulating the cache -- instead, ordinary user calls to git_submodule_lookup would be uncached. But internal calls (e.g. from git_diff or git_checkout_index could first initialize the cache, then do their multiple calls, then tear down the cache. But this would not help users who do multiple submodule lookups on their own.

novalis · 2016-12-15T20:51:00Z

My conclusion from the above, btw, is that automatic caching doesn't make sense and that we ought to just merge this as-is.

ethomson · 2016-12-16T15:10:06Z

Thanks for the writeup @novalis . I think that makes sense. I have some review comments to make but I think the overall strategy makes sense.

ethomson · 2016-12-16T15:13:43Z

src/submodule.c

@@ -1566,6 +1617,33 @@ int git_submodule_location(unsigned int *location, git_submodule *sm)
 		location, NULL, NULL, NULL, sm, GIT_SUBMODULE_IGNORE_ALL);
 }

+int git_repository_submodule_cache_all(git_repository *repo)


Please don't put git_repository functions in a file that is not repository.c. Either: 1) these should either go in repository.c, or 2) they should be git_submodule_cache_all and git_submodule_cache_clear. The latter is obviously more brief, but I think (1) makes more sense.

ethomson · 2016-12-16T15:23:57Z

include/git2/repository.h

+ *
+ * @param repo the repository whose submodules will be cached.
+ */
+GIT_EXTERN(int) git_repository_submodule_cache_all(


There's no indication of when somebody should call cache_clear. You need to describe what operations would cause a cache coherency problem; it's not at all obvious to the reader and I suspect it's wildly dangerous to call cache_all and then (say) check out a new branch that would update a submodule? I'm just guessing, I don't know, and I'd like this to expand upon that.

I also think that you should move this into include/git2/sys/repository.h. This seems to me to be a cache that you have to manage rather carefully, and the sys/ directory is for the things that will hurt you badly if you get wrong.

ethomson · 2016-12-16T15:25:47Z

src/submodule.c

+static void free_submodule_names(git_strmap *names)
+{
+	git_buf *name = 0;
+	if (0 == names)


Please don't reverse these tests. I know that many people have gotten into a habit of this and find it a good practice, but please follow our existing code style.

Please don't use 0 for pointers.

This should be if (names == NULL).

ethomson · 2016-12-16T15:27:27Z

src/submodule.c

@@ -196,6 +209,17 @@ int git_submodule_lookup(

 	assert(repo && name);

+	if (0 != repo->submodule_cache) {


if (repo->submodule_cache != NULL), please.

ethomson · 2016-12-16T15:30:38Z

src/submodule.c

+{
+	git_submodule *sm;
+	assert(repo);
+	if (0 == repo->submodule_cache) {


if (repo->submodule_cache == NULL)...

ethomson · 2016-12-16T15:36:21Z

tests/submodule/lookup.c

+	git_repository_submodule_cache_all(g_repo);
+	cl_assert(0 == git_submodule_lookup(&sm, g_repo, "sm_unchanged"));
+	cl_assert(0 == git_submodule_lookup(&sm2, g_repo, "sm_unchanged"));
+	cl_assert(sm == sm2);


Using cl_assert is not ideal here, since it just will say "an assertion failed". We have a variety of helper functions that provide more explicit messages, which are helpful in debugging and understanding test failures:

For git functions, please use cl_git_pass, which will show the error messages properly when a function fails and returns nonzero. (eg, cl_git_pass(git_submodule_lookup...))

To compare two pointers, please use cl_assert_equal_p which will show more information in a nice format about the actual and expected values.

ethomson · 2016-12-16T15:36:37Z

tests/submodule/lookup.c

+
+	/* and that we get new objects again after clearing the cache. */
+	git_repository_submodule_cache_clear(g_repo);
+	cl_assert(0 == git_submodule_lookup(&sm2, g_repo, "sm_unchanged"));


Here, too, this should be cl_git_pass(...).

ethomson · 2016-12-16T15:38:52Z

I have some comments. Note that I haven't looked at the submodule logic to know if that was just changed whitespace or if there are substantive changes, too. I'll have to do that before we merge, unless you revert it. (Mixing whitespace changes with actual code changes is very hard to follow in many diff tools.)

novalis · 2016-12-16T17:56:16Z

I have a separate commit for the whitespace changes. Are you seeing other whitespace changes outside that commit?

ethomson · 2016-12-30T19:16:41Z

I have a separate commit for the whitespace changes.

Yep, thanks, that's fine then; I was looking at this in the GitHub web view and didn't realize that.

ethomson

Sorry, after giving this a final look, I noticed two more very minor issues. I hope you don't mind fixing these real quick so that I can merge it; apologies that I didn't catch these the first time around. Thanks!

ethomson · 2017-01-14T17:33:00Z

src/submodule.c

-	/* refresh the HEAD OID */
-	if (submodule_update_head(sm) < 0)
-		return -1;
+	if (0 == sm->repo->submodule_cache) {


Can you also reverse this test?

ethomson · 2017-01-14T17:33:00Z

src/submodule.c

@@ -416,7 +458,7 @@ typedef struct {
 	git_repository *repo;
 } lfc_data;

-static int all_submodules(git_repository *repo, git_strmap *map)
+int all_submodules(git_repository *repo, git_strmap *map)


Now that this is public (within the library anyway), this should have a name that indicates it's internal to the library. Something like git_submodule__all or git_submodule__map, whichever makes more sense to you.

Signed-off-by: David Turner <[email protected]>

Added `git_repository_submodule_cache_all` to initialze a cache of submodules on the repository so that operations looking up N submodules are O(N) and not O(N^2). Added a `git_repository_submodule_cache_clear` function to remove the cache. Also optimized the function that loads all submodules as it was itself O(N^2) w.r.t the number of submodules, having to loop through the `.gitmodules` file once per submodule. I changed it to process the `.gitmodules` file once, into a map. Signed-off-by: David Turner <[email protected]>

`git_submodule_status` is very slow, bottlenecked on `git_repository_head_tree`, which it uses through `submodule_update_head`. If the user has requested submodule caching, assume that they want this status cached too and skip it. Signed-off-by: David Turner <[email protected]>

novalis · 2017-01-21T15:56:31Z

OK, I think I've addressed all comments; there are no merge conflicts; the tests are green.

ethomson · 2017-01-21T18:57:23Z

Thanks for doing this @bpeabody and @novalis - and thank you especially for putting up with my laziness and bikeshedding. 😅

bpeabody · 2017-01-22T14:52:46Z

@ethomson: it was no trouble at all; thanks for all your help!

novalis force-pushed the submodule-optimization branch from 5d3de99 to beeaf92 Compare November 28, 2016 20:05

novalis force-pushed the submodule-optimization branch from beeaf92 to 89d0deb Compare November 29, 2016 16:13

novalis force-pushed the submodule-optimization branch from 89d0deb to deac747 Compare December 1, 2016 16:27

ethomson requested changes Dec 16, 2016

View reviewed changes

novalis force-pushed the submodule-optimization branch from deac747 to 1f5c2d0 Compare December 16, 2016 17:56

This was referenced Jan 4, 2017

git_repository_index succeeds on bare repo #4067

Closed

git_submodule_foreach segfaults in a bare repo #4068

Closed

ethomson requested changes Jan 14, 2017

View reviewed changes

novalis force-pushed the submodule-optimization branch from 1f5c2d0 to 4e9aca8 Compare January 18, 2017 20:21

novalis and others added 3 commits January 20, 2017 17:33

Fix formatting

ca05857

Signed-off-by: David Turner <[email protected]>

novalis force-pushed the submodule-optimization branch from 4e9aca8 to 673dff8 Compare January 20, 2017 22:34

ethomson merged commit 98f5387 into libgit2:master Jan 21, 2017

johnhaley81 mentioned this pull request Feb 6, 2017

Bump libgit to 0bf0526 nodegit/nodegit#1187

Merged

ksuther mentioned this pull request Feb 14, 2018

Develop gitx/gitx#117

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Submodule optimization #4016

Submodule optimization #4016

novalis commented Nov 24, 2016

novalis commented Nov 28, 2016

novalis commented Nov 28, 2016

novalis commented Nov 29, 2016

novalis commented Dec 2, 2016

ethomson commented Dec 4, 2016

bpeabody commented Dec 4, 2016

bpeabody commented Dec 4, 2016

novalis commented Dec 5, 2016 •

edited

Loading

novalis commented Dec 15, 2016

ethomson commented Dec 16, 2016

ethomson Dec 16, 2016

ethomson Dec 16, 2016

ethomson Dec 16, 2016

ethomson Dec 16, 2016

ethomson Dec 16, 2016

ethomson Dec 16, 2016

ethomson Dec 16, 2016

ethomson commented Dec 16, 2016

novalis commented Dec 16, 2016

ethomson commented Dec 30, 2016

ethomson left a comment

ethomson Jan 14, 2017

ethomson Jan 14, 2017

novalis commented Jan 21, 2017

ethomson commented Jan 21, 2017

bpeabody commented Jan 22, 2017

		@@ -196,6 +209,17 @@ int git_submodule_lookup(

		assert(repo && name);

		if (0 != repo->submodule_cache) {

Submodule optimization #4016

Submodule optimization #4016

Conversation

novalis commented Nov 24, 2016

novalis commented Nov 28, 2016

novalis commented Nov 28, 2016

novalis commented Nov 29, 2016

novalis commented Dec 2, 2016

ethomson commented Dec 4, 2016

bpeabody commented Dec 4, 2016

bpeabody commented Dec 4, 2016

novalis commented Dec 5, 2016 • edited Loading

novalis commented Dec 15, 2016

ethomson commented Dec 16, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ethomson commented Dec 16, 2016

novalis commented Dec 16, 2016

ethomson commented Dec 30, 2016

ethomson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

novalis commented Jan 21, 2017

ethomson commented Jan 21, 2017

bpeabody commented Jan 22, 2017

novalis commented Dec 5, 2016 •

edited

Loading