Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Support for shallow-since clones/fetches #7093

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Support for shallow-since clones/fetches #7093

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ee70038
Support for the "shallow-since" option in fetch requests.
Jun 19, 2025
2058a36
Assign the correct "shallow_since" default when no
Jun 19, 2025
385d88f
Fix comments in new unit tests.
Jun 19, 2025
9c8a6fa
Rename detault value of shallow_since - to "FULL", with value zero, f…
Jun 20, 2025
80fd296
Combine some duplicated error handling.
Jun 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Support for the "shallow-since" option in fetch requests.
  • Loading branch information
Malcolm Wood committed Jun 19, 2025
commit ee700384376e25c6da1dedd95562a5352cfa2020
23 changes: 20 additions & 3 deletions include/git2/remote.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -762,7 +762,10 @@ typedef enum {
GIT_FETCH_DEPTH_FULL = 0,

/** The fetch should "unshallow" and fetch missing data. */
GIT_FETCH_DEPTH_UNSHALLOW = 2147483647
GIT_FETCH_DEPTH_UNSHALLOW = 2147483647,

/** No "shallow-since" date specified. */
GIT_FETCH_SINCE_UNSPECIFIED = -1
} git_fetch_depth_t;

/**
Expand Down Expand Up @@ -808,12 +811,22 @@ typedef struct {
/**
* Depth of the fetch to perform, or `GIT_FETCH_DEPTH_FULL`
* (or `0`) for full history, or `GIT_FETCH_DEPTH_UNSHALLOW`
* to "unshallow" a shallow repository.
* to "unshallow" a shallow repository. Cannot be used in
* conjunction with a specific shallow_since date.
*
* The default is full (`GIT_FETCH_DEPTH_FULL` or `0`).
*/
int depth;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we have shallow_since, depth is no longer the only variable that determines whether a fetch is shallow or not. Consider clone.c: if (!options.fetch_opts.depth)
options.fetch_opts.download_tags = GIT_REMOTE_DOWNLOAD_TAGS_ALL;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we could have a new function that takes git_fetch_options and returns true/false if the fetch would be shallow.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried this, but it's difficult to get a combination of name and behaviour. You only know if the fetch "would be shallow" if you know how many commits there are, which we don't. An "unshallow" operation is largely indistinguishable from a shallow fetch with a large depth.


/**
* Date from which to fetch history, or 'GIT_FETCH_SINCE_UNSPECIFIED'
* to fetch all history.
* Cannot be used in conjuction with a specific depth.
*
* The default is to fetch all history (`GIT_FETCH_SINCE_UNSPECIFIED` or `-1`).
*/
git_time_t shallow_since;

/**
* Whether to allow off-site redirects. If this is not
* specified, the `http.followRedirects` configuration setting
Expand All @@ -837,7 +850,11 @@ typedef struct {
GIT_FETCH_PRUNE_UNSPECIFIED, \
GIT_REMOTE_UPDATE_FETCHHEAD, \
GIT_REMOTE_DOWNLOAD_TAGS_UNSPECIFIED, \
GIT_PROXY_OPTIONS_INIT }
GIT_PROXY_OPTIONS_INIT, \
GIT_FETCH_DEPTH_FULL, \
GIT_FETCH_SINCE_UNSPECIFIED, \
GIT_REMOTE_REDIRECT_NONE \
}

/**
* Initialize git_fetch_options structure
Expand Down
1 change: 1 addition & 0 deletions include/git2/sys/transport.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ typedef struct {
git_oid *shallow_roots;
size_t shallow_roots_len;
int depth;
git_time_t shallow_since;
} git_fetch_negotiation;

struct git_transport {
Expand Down
1 change: 1 addition & 0 deletions src/libgit2/fetch.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,7 @@ int git_fetch_negotiate(git_remote *remote, const git_fetch_options *opts)
if (opts) {
GIT_ASSERT_ARG(opts->depth >= 0);
remote->nego.depth = opts->depth;
remote->nego.shallow_since = opts->shallow_since;
}

if (filter_wants(remote, opts) < 0)
Expand Down
6 changes: 6 additions & 0 deletions src/libgit2/transports/local.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,12 @@ static int local_negotiate_fetch(
git_error_set(GIT_ERROR_NET, "shallow fetch is not supported by the local transport");
return GIT_ENOTSUPPORTED;
}
if (wants->shallow_since != GIT_FETCH_SINCE_UNSPECIFIED) {
git_error_set(
GIT_ERROR_NET,
"shallow-since fetch is not supported by the local transport");
return GIT_ENOTSUPPORTED;
}

/* Fill in the loids */
git_vector_foreach(&t->refs, i, rhead) {
Expand Down
27 changes: 27 additions & 0 deletions src/libgit2/transports/smart_pkt.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -828,13 +828,40 @@ int git_pkt_buffer_wants(
}

if (wants->depth > 0) {

if (wants->shallow_since != GIT_FETCH_SINCE_UNSPECIFIED) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we do this argument validation up in git_fetch_negotiate?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't see a better location that would definitely be hit by every codepath.

git_error_set(
GIT_ERROR_NET,
"depth and shallow-since must not be used together");
return GIT_ENOTSUPPORTED;
}

git_str deepen_buf = GIT_STR_INIT;

git_str_printf(&deepen_buf, "deepen %d\n", wants->depth);
git_str_printf(buf,"%04x%s", (unsigned int)git_str_len(&deepen_buf) + 4, git_str_cstr(&deepen_buf));

git_str_dispose(&deepen_buf);

if (git_str_oom(buf))
return -1;

} else if (wants->shallow_since != GIT_FETCH_SINCE_UNSPECIFIED) {

git_str shallow_since_buf = GIT_STR_INIT;

/* The git command-line option is "shallow-since", but the protocol uses "deepen-since" */
git_str_printf(
&shallow_since_buf, "deepen-since %d\n",
wants->shallow_since);

git_str_printf(
buf, "%04x%s",
(unsigned int)git_str_len(&shallow_since_buf) + 4,
git_str_cstr(&shallow_since_buf));

git_str_dispose(&shallow_since_buf);

if (git_str_oom(buf))
return -1;
}
Expand Down
2 changes: 1 addition & 1 deletion src/libgit2/transports/smart_protocol.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -435,7 +435,7 @@ int git_smart__negotiate_fetch(
if ((error = git_revwalk__push_glob(walk, "refs/*", &opts)) < 0)
goto on_error;

if (wants->depth > 0) {
if (wants->depth > 0 || wants->shallow_since != GIT_FETCH_SINCE_UNSPECIFIED) {
git_pkt_shallow *pkt;

if ((error = git_smart__negotiation_step(&t->parent, data.ptr, data.size)) < 0)
Expand Down
130 changes: 130 additions & 0 deletions tests/libgit2/online/shallow.c
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#include "clar_libgit2.h"
#include "futils.h"
#include "repository.h"
#include "date.h"

static int remote_single_branch(git_remote **out, git_repository *repo, const char *name, const char *url, void *payload)
{
Expand Down Expand Up @@ -124,6 +125,135 @@ void test_online_shallow__clone_depth_five(void)
git_repository_free(repo);
}

void test_online_shallow__clone_since_oldest(void)
{
git_str path = GIT_STR_INIT;
git_repository *repo;
git_revwalk *walk;
git_clone_options clone_opts = GIT_CLONE_OPTIONS_INIT;
git_oid oid;
git_oid *roots;
size_t roots_len;
size_t num_commits = 0;
int error = 0;

/* Specify a date before the old commit. */
git_time_t since_date;
cl_git_pass(git_date_parse(&since_date,"2005-04-05"));
clone_opts.fetch_opts.shallow_since = since_date;
clone_opts.remote_cb = remote_single_branch;

git_str_joinpath(&path, clar_sandbox_path(), "shallowclone_old");

cl_git_pass(git_clone(
&repo, "https://github.com/libgit2/TestGitRepository",
git_str_cstr(&path), &clone_opts));

cl_assert_equal_b(false, git_repository_is_shallow(repo));

cl_git_pass(git_repository__shallow_roots(&roots, &roots_len, repo));
cl_assert_equal_i(0, roots_len);

git_revwalk_new(&walk, repo);

git_revwalk_push_head(walk);

while ((error = git_revwalk_next(&oid, walk)) == GIT_OK) {
num_commits++;
}

cl_assert_equal_i(num_commits, 21);
cl_assert_equal_i(error, GIT_ITEROVER);

git__free(roots);
git_str_dispose(&path);
git_revwalk_free(walk);
git_repository_free(repo);
}

void test_online_shallow__clone_since_midpoint(void)
{
git_str path = GIT_STR_INIT;
git_repository *repo;
git_revwalk *walk;
git_clone_options clone_opts = GIT_CLONE_OPTIONS_INIT;
git_oid oid;
git_oid *roots;
size_t roots_len;
size_t num_commits = 0;
int error = 0;

/* Specify a date between the oldest and most recent commit. */
git_time_t since_date;
cl_git_pass(git_date_parse(&since_date, "2005-04-07 22:30:00 UTC"));
clone_opts.fetch_opts.shallow_since = since_date;
clone_opts.remote_cb = remote_single_branch;

git_str_joinpath(&path, clar_sandbox_path(), "shallowclone_midpoint");

cl_git_pass(git_clone(
&repo, "https://github.com/libgit2/TestGitRepository",
git_str_cstr(&path), &clone_opts));

cl_assert_equal_b(true, git_repository_is_shallow(repo));

cl_git_pass(git_repository__shallow_roots(&roots, &roots_len, repo));
cl_assert_equal_i(3, roots_len);
cl_assert_equal_s(
"d0114ab8ac326bab30e3a657a0397578c5a1af88",
git_oid_tostr_s(&roots[0]));
cl_assert_equal_s(
"49322bb17d3acc9146f98c97d078513228bbf3c0",
git_oid_tostr_s(&roots[1]));
cl_assert_equal_s(
"f73b95671f326616d66b2afb3bdfcdbbce110b44",
git_oid_tostr_s(&roots[2]));

git_revwalk_new(&walk, repo);

git_revwalk_push_head(walk);

while ((error = git_revwalk_next(&oid, walk)) == GIT_OK) {
num_commits++;
}

cl_assert_equal_i(num_commits, 1);
cl_assert_equal_i(error, GIT_ITEROVER);

git__free(roots);
git_str_dispose(&path);
git_revwalk_free(walk);
git_repository_free(repo);
}

void test_online_shallow__clone_since_recent(void)
{
git_str path = GIT_STR_INIT;
git_repository *repo;
git_clone_options clone_opts = GIT_CLONE_OPTIONS_INIT;
size_t num_commits = 0;
int error = 0;

/* Specify a date between the oldest and most recent commit. */
git_time_t since_date;
cl_git_pass(git_date_parse(&since_date, "2025-04-07 22:30:00 UTC"));
clone_opts.fetch_opts.shallow_since = since_date;
clone_opts.remote_cb = remote_single_branch;

git_str_joinpath(&path, clar_sandbox_path(), "shallowclone_recent");

error = git_clone(
&repo, "https://github.com/libgit2/TestGitRepository",
git_str_cstr(&path), &clone_opts);

/* Command-line git gives a different (but no more useful) error in this
* case - "error processing shallow info: 4". */
cl_assert_equal_i(error, GIT_EEOF);

git_str_dispose(&path);
git_repository_free(repo);
}

void test_online_shallow__unshallow(void)
{
git_str path = GIT_STR_INIT;
Expand Down