Thanks to visit codestin.com
Credit goes to github.com

Skip to content

support git 2.21's multi-pack-index #5399

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
4 tasks done
lhchavez opened this issue Feb 13, 2020 · 5 comments · Fixed by #5405
Closed
4 tasks done

support git 2.21's multi-pack-index #5399

lhchavez opened this issue Feb 13, 2020 · 5 comments · Fixed by #5405

Comments

@lhchavez
Copy link
Contributor

lhchavez commented Feb 13, 2020 •
edited by ethomson
Loading

Starting from version 2.21, git supports multi-pack-index, which allows for O(log n) scan of n objects among m packfiles (instead of O(m log n/m)) without needing to do a garbage collection to compact all the objects into a single .pack file. This is desirable in cases where there are lots of Delta islands or it is otherwise undesirable to merge large numbers of packfiles.

This should be relatively straightforward to implement. odb_pack.c would need to be modified so that the multi-pack-index's index is consulted first. Only if an OID is not found there, it should fall back to the current behavior of iterating over all the .pack files, although packfile_load__cb() should be modified to ignore any files that are indexed in multi-pack-index (to avoid degenerating back into O(m log n/m) complexity).

This can be split in four chunks for easier reviewing:

Relevant documentation:
@lhchavez
Copy link
Contributor Author

Part 1 should be out for review within ~a week.

lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 15, 2020
@lhchavez
multipack: Introduce a parser for multi-pack-index files
819cffd 
This change is the first in a series to add support for git's
multi-pack-index. This should speed up large repositories significantly.

Part of: libgit2#5399
@lhchavez lhchavez mentioned this issue Feb 15, 2020
Merged
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 15, 2020
@lhchavez
multipack: Introduce a parser for multi-pack-index files
12a1370 
This change is the first in a series to add support for git's
multi-pack-index. This should speed up large repositories significantly.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 15, 2020
@lhchavez
multipack: Introduce a parser for multi-pack-index files
f07d519 
This change is the first in a series to add support for git's
multi-pack-index. This should speed up large repositories significantly.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 15, 2020
@lhchavez
multipack: Introduce a parser for multi-pack-index files
a2f027a 
This change is the first in a series to add support for git's
multi-pack-index. This should speed up large repositories significantly.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 15, 2020
@lhchavez
multipack: Introduce a parser for multi-pack-index files
eeb0794 
This change is the first in a series to add support for git's
multi-pack-index. This should speed up large repositories significantly.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 15, 2020
@lhchavez
multipack: Introduce a parser for multi-pack-index files
5547823 
This change is the first in a series to add support for git's
multi-pack-index. This should speed up large repositories significantly.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 15, 2020
@lhchavez
multipack: Introduce a parser for multi-pack-index files
aa2b0bd 
This change is the first in a series to add support for git's
multi-pack-index. This should speed up large repositories significantly.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 16, 2020
@lhchavez
multipack: Introduce a parser for multi-pack-index files
b9d4288 
This change is the first in a series to add support for git's
multi-pack-index. This should speed up large repositories significantly.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 16, 2020
@lhchavez
multipack: Introduce a parser for multi-pack-index files
9b99abe 
This change is the first in a series to add support for git's
multi-pack-index. This should speed up large repositories significantly.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 17, 2020
@lhchavez
multipack: Introduce a parser for multi-pack-index files
22fedd3 
This change is the first in a series to add support for git's
multi-pack-index. This should speed up large repositories significantly.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 17, 2020
@lhchavez
multipack: Support multi-pack-index files in odb_pack.c
37218e5 
This change adds support for reading multi-pack-index files from the
packfile odb backend. This also makes git_pack_file objects open their
backing failes lazily in more scenarios, since the multi-pack-index can
avoid having to open them in some cases (yay!).

This change also refreshes the documentation found in src/odb_pack.c to
match the updated code.

Part of: libgit2#5399
@lhchavez lhchavez mentioned this issue Feb 17, 2020
Closed
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 17, 2020
@lhchavez
multipack: Introduce a parser for multi-pack-index files
f47ec5e 
This change is the first in a series to add support for git's
multi-pack-index. This should speed up large repositories significantly.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 18, 2020
@lhchavez
multipack: Introduce a parser for multi-pack-index files
9999b82 
This change is the first in a series to add support for git's
multi-pack-index. This should speed up large repositories significantly.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 18, 2020
@lhchavez
multipack: Add a way to write multi-pack-index files
8887926 
This change adds the git_multipack_index_writer_* functions to allow to
write and create `multi-pack-index` files from `.idx`/`.pack` files.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 18, 2020
@lhchavez
multipack: Introduce git_odb_write_multi_pack_index()
43f43f6 
This change introduces git_odb_write_multi_pack_index(), which creates a
`multi-pack-index` file from all the `.pack` files that have been loaded
in the ODB.

Fixes: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 18, 2020
@lhchavez
multipack: Introduce a parser for multi-pack-index files
a261322 
This change is the first in a series to add support for git's
multi-pack-index. This should speed up large repositories significantly.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 18, 2020
@lhchavez
multipack: Add a way to write multi-pack-index files
ba5d63d 
This change adds the git_multipack_index_writer_* functions to allow to
write and create `multi-pack-index` files from `.idx`/`.pack` files.

Part of: libgit2#5399
@lhchavez lhchavez mentioned this issue Feb 18, 2020
Merged
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 18, 2020
@lhchavez
multipack: Introduce git_odb_write_multi_pack_index()
90fdd56 
This change introduces git_odb_write_multi_pack_index(), which creates a
`multi-pack-index` file from all the `.pack` files that have been loaded
in the ODB.

Fixes: libgit2#5399
@lhchavez lhchavez mentioned this issue Feb 18, 2020
Merged
@pks-t
Copy link
Member

pks-t commented Feb 22, 2020

Quick note so you don't feel ignored: I'll start reviewing your code as soon as we've released v1.0, which should be soonish. Looking forward to having multipack index support!

lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 23, 2020
@lhchavez
multipack: Introduce a parser for multi-pack-index files
35e7640 
This change is the first in a series to add support for git's
multi-pack-index. This should speed up large repositories significantly.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 23, 2020
@lhchavez
multipack: Support multi-pack-index files in odb_pack.c
5e62c29 
This change adds support for reading multi-pack-index files from the
packfile odb backend. This also makes git_pack_file objects open their
backing failes lazily in more scenarios, since the multi-pack-index can
avoid having to open them in some cases (yay!).

This change also refreshes the documentation found in src/odb_pack.c to
match the updated code.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 23, 2020
@lhchavez
multipack: Add a way to write multi-pack-index files
c508636 
This change adds the git_multipack_index_writer_* functions to allow to
write and create `multi-pack-index` files from `.idx`/`.pack` files.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 23, 2020
@lhchavez
multipack: Introduce git_odb_write_multi_pack_index()
4d75a5b 
This change introduces git_odb_write_multi_pack_index(), which creates a
`multi-pack-index` file from all the `.pack` files that have been loaded
in the ODB.

Fixes: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 24, 2020
@lhchavez
multipack: Add a way to write multi-pack-index files
cacaac9 
This change adds the git_multipack_index_writer_* functions to allow to
write and create `multi-pack-index` files from `.idx`/`.pack` files.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Feb 24, 2020
@lhchavez
multipack: Introduce git_odb_write_multi_pack_index()
ca324c1 
This change introduces git_odb_write_multi_pack_index(), which creates a
`multi-pack-index` file from all the `.pack` files that have been loaded
in the ODB.

Fixes: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Oct 5, 2020
@lhchavez
multipack: Introduce a parser for multi-pack-index files
005e771 
This change is the first in a series to add support for git's
multi-pack-index. This should speed up large repositories significantly.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Oct 5, 2020
@lhchavez
midx: Support multi-pack-index files in odb_pack.c
6f6146f 
This change adds support for reading multi-pack-index files from the
packfile odb backend. This also makes git_pack_file objects open their
backing failes lazily in more scenarios, since the multi-pack-index can
avoid having to open them in some cases (yay!).

This change also refreshes the documentation found in src/odb_pack.c to
match the updated code.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Oct 5, 2020
@lhchavez
midx: Add a way to write multi-pack-index files
4cec6b3 
This change adds the git_midx_writer_* functions to allow to
write and create `multi-pack-index` files from `.idx`/`.pack` files.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Oct 5, 2020
@lhchavez
midx: Introduce git_odb_write_multi_pack_index()
cd7762d 
This change introduces git_odb_write_multi_pack_index(), which creates a
`multi-pack-index` file from all the `.pack` files that have been loaded
in the ODB.

Fixes: libgit2#5399
@sluongng
Copy link

what is the status for this issue?

I see active commits but I'm not quite sure regarding the progress.

@lhchavez do you need any help? should we split the issue/PRs into even smaller parts?

@lhchavez
Copy link
Contributor Author

lhchavez commented Oct 18, 2020 •
edited
Loading

what is the status for this issue?

i've been using this feature in production (from my branch) for months now.

@lhchavez do you need any help? should we split the issue/PRs into even smaller parts?

i think it's already as split into smaller parts as feasible / convenient (both for me and the reviewers). one of those parts has already landed!

ethomson pushed a commit that referenced this issue Nov 27, 2020
@lhchavez @ethomson
midx: Support multi-pack-index files in odb_pack.c
f847fa7 
This change adds support for reading multi-pack-index files from the
packfile odb backend. This also makes git_pack_file objects open their
backing failes lazily in more scenarios, since the multi-pack-index can
avoid having to open them in some cases (yay!).

This change also refreshes the documentation found in src/odb_pack.c to
match the updated code.

Part of: #5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Nov 27, 2020
@lhchavez
midx: Add a way to write multi-pack-index files
263a714 
This change adds the git_midx_writer_* functions to allow to
write and create `multi-pack-index` files from `.idx`/`.pack` files.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Nov 27, 2020
@lhchavez
midx: Introduce git_odb_write_multi_pack_index()
9c257a6 
This change introduces git_odb_write_multi_pack_index(), which creates a
`multi-pack-index` file from all the `.pack` files that have been loaded
in the ODB.

Fixes: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Nov 27, 2020
@lhchavez
odb: Add git_odb_options
908bc97 
This change introduces the git_odb_options struct and a bunch of
`*_ext()` functions that consumes it. This does nothing for now, but
paves the way to allow callers to choose whether or not to enable the
multi-pack-index feature, similar to what vanilla git does.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Nov 29, 2020
@lhchavez
midx: Add a way to write multi-pack-index files
aa8a417 
This change adds the git_midx_writer_* functions to allow to
write and create `multi-pack-index` files from `.idx`/`.pack` files.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Nov 29, 2020
@lhchavez
midx: Introduce git_odb_write_multi_pack_index()
79c6794 
This change introduces git_odb_write_multi_pack_index(), which creates a
`multi-pack-index` file from all the `.pack` files that have been loaded
in the ODB.

Fixes: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Nov 29, 2020
@lhchavez
midx: Add a way to write multi-pack-index files
4862427 
This change adds the git_midx_writer_* functions to allow to
write and create `multi-pack-index` files from `.idx`/`.pack` files.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Nov 29, 2020
@lhchavez
midx: Introduce git_odb_write_multi_pack_index()
3f2deed 
This change introduces git_odb_write_multi_pack_index(), which creates a
`multi-pack-index` file from all the `.pack` files that have been loaded
in the ODB.

Fixes: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Dec 13, 2020
@lhchavez
midx: Add a way to write multi-pack-index files
2ea0ae4 
This change adds the git_midx_writer_* functions to allow to
write and create `multi-pack-index` files from `.idx`/`.pack` files.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Dec 13, 2020
@lhchavez
midx: Introduce git_odb_write_multi_pack_index()
07fb62b 
This change introduces git_odb_write_multi_pack_index(), which creates a
`multi-pack-index` file from all the `.pack` files that have been loaded
in the ODB.

Fixes: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Dec 13, 2020
@lhchavez
midx: Add a way to write multi-pack-index files
e7ba51a 
This change adds the git_midx_writer_* functions to allow to
write and create `multi-pack-index` files from `.idx`/`.pack` files.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Dec 13, 2020
@lhchavez
midx: Introduce git_odb_write_multi_pack_index()
daf437c 
This change introduces git_odb_write_multi_pack_index(), which creates a
`multi-pack-index` file from all the `.pack` files that have been loaded
in the ODB.

Fixes: libgit2#5399
@lhchavez lhchavez mentioned this issue Jan 4, 2021
Open
8 tasks
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Jan 7, 2021
@lhchavez
midx: Add a way to write multi-pack-index files
dfbdc0c 
This change adds the git_midx_writer_* functions to allow to
write and create `multi-pack-index` files from `.idx`/`.pack` files.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Jan 7, 2021
@lhchavez
midx: Introduce git_odb_write_multi_pack_index()
6877d3c 
This change introduces git_odb_write_multi_pack_index(), which creates a
`multi-pack-index` file from all the `.pack` files that have been loaded
in the ODB.

Fixes: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Jan 8, 2021
@lhchavez
midx: Add a way to write multi-pack-index files
5a51a0b 
This change adds the git_midx_writer_* functions to allow to
write and create `multi-pack-index` files from `.idx`/`.pack` files.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Jan 8, 2021
@lhchavez
midx: Introduce git_odb_write_multi_pack_index()
37d87b7 
This change introduces git_odb_write_multi_pack_index(), which creates a
`multi-pack-index` file from all the `.pack` files that have been loaded
in the ODB.

Fixes: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Jul 27, 2021
@lhchavez
midx: Add a way to write multi-pack-index files
fff209c 
This change adds the git_midx_writer_* functions to allow to
write and create `multi-pack-index` files from `.idx`/`.pack` files.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Jul 27, 2021
@lhchavez
midx: Introduce git_odb_write_multi_pack_index()
55bed14 
This change introduces git_odb_write_multi_pack_index(), which creates a
`multi-pack-index` file from all the `.pack` files that have been loaded
in the ODB.

Fixes: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Aug 26, 2021
@lhchavez
odb: Add git_odb_options
1371ab8 
This change introduces the git_odb_options struct and a bunch of
`*_ext()` functions that consumes it. This does nothing for now, but
paves the way to allow callers to choose whether or not to enable the
multi-pack-index feature, similar to what vanilla git does.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Aug 27, 2021
@lhchavez
midx: Add a way to write multi-pack-index files
9d117e3 
This change adds the git_midx_writer_* functions to allow to
write and create `multi-pack-index` files from `.idx`/`.pack` files.

Part of: libgit2#5399
lhchavez added a commit to lhchavez/libgit2 that referenced this issue Aug 27, 2021
@lhchavez
midx: Introduce git_odb_write_multi_pack_index()
ea28590 
This change introduces git_odb_write_multi_pack_index(), which creates a
`multi-pack-index` file from all the `.pack` files that have been loaded
in the ODB.

Fixes: libgit2#5399
@ethomson
Copy link
Member

💥 Thanks for this! So excited to see this shipping.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

midx: Introduce git_odb_write_multi_pack_index() lhchavez/libgit2
4 participants
@lhchavez @ethomson @pks-t @sluongng