tq: standardized progress meter formatting#2811
Merged
Merged
Conversation
technoweenie
approved these changes
Jan 17, 2018
technoweenie
left a comment
Contributor
There was a problem hiding this comment.
Looks great. 1 super minor nit.
| // TransferBytes increments the number of bytes transferred | ||
| func (m *Meter) TransferBytes(direction, name string, read, total int64, current int) { | ||
| now := time.Now() | ||
|
|
Contributor
There was a problem hiding this comment.
I'd put this under the if m == nil check, since nothing happens with now before that return.
Contributor
Author
There was a problem hiding this comment.
Good call. I was wavering on this when writing the patch, and left it at the top because I had convinced myself that waiting until after the if would introduce imprecision in the averaging. While probably true to a degree, I'm sure that the effect is negligible, and the code would be more readable if the "first thing" that is supposed to happen came after the nullity check. Changed in cf8d827.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Nov 17, 2020
Since commit df2007d in PR git-lfs#646 it appears that the push_all_setup() helper function has accidentally short-circuited all but the first test calling it by exiting early with an exit code of 0 when the test repo exists. The calling tests thus appear to pass but are instead skipping all of their actual test logic. The original intent appears to have been to reuse a single test repo for the set of related tests of the --all push option. However, these tests do not leave the repo state in the same condition created by push_all_setup(), e.g., they replace the "origin" remote, and most importantly they each remove an LFS object file. Thus simply replacing "exit 0" with "return 0" in push_all_setup() is insufficient. To simplify matters, we no longer try to reuse the test repos between tests. Rather, we let push_all_setup() create a unique test repo for each caller by appending the distinct suffix argument provided by the caller to the name of the test repo. We then have to account for the fact that these tests have not fully run in some time, and therefore missed changes like commit db4e3b2 in PR git-lfs#2811 which updated the Git LFS progress meter output format. Several calls to "git push" also need adjustment to avoid errors such as "--all can't be combined with refspecs" and "tag shorthand without <tag>", and one short-circuited test had an incorrect count of "push ... => file1.dat" lines which we revise.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Sep 10, 2024
In commit db4e3b2 of PR git-lfs#2811 in 2018 we updated the progress meter text of Git LFS from "Git LFS" to the more explicit "Uploading LFS objects" or "Downloading LFS objects". A large number of were changed at this time to check for the revised text strings in the output logs the capture from Git and Git LFS operations. However, the "cloneSSL" test in our t/t-clone.sh test script was not updated because it does not actually run to completion, due to a long-standing bug in the conditional used to check the TRAVIS variable. As described in git-lfs#5658, the test instead always exits early and declares success. We expect to address this problem in a subsequent commit in this PR, so we first update the test to check for the same progress meter text as all our other tests, which will help ensure it succeeds once it is fully enabled. We also take the opportunity to add a similar check of the same progress meter message to the "clone ClientCert" test. This brings our tests into slightly closer alignment with each other, which should help ensure they stay consistent in the future. (Note that this test also does not run to completion at the moment, for the same reason as the "cloneSSL" test.)
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Sep 15, 2024
In commit db4e3b2 of PR git-lfs#2811 in 2018 we updated the progress meter text of Git LFS from "Git LFS" to the more explicit "Uploading LFS objects" or "Downloading LFS objects". A large number of were changed at this time to check for the revised text strings in the output logs the capture from Git and Git LFS operations. However, the "cloneSSL" test in our t/t-clone.sh test script was not updated because it does not actually run to completion, due to a long-standing bug in the conditional used to check the TRAVIS variable. As described in git-lfs#5658, the test instead always exits early and declares success. We expect to address this problem in a subsequent commit in this PR, so we first update the test to check for the same progress meter text as all our other tests, which will help ensure it succeeds once it is fully enabled. We also take the opportunity to add a similar check of the same progress meter message to the "clone ClientCert" test. This brings our tests into slightly closer alignment with each other, which should help ensure they stay consistent in the future. (Note that this test also does not run to completion at the moment, for the same reason as the "cloneSSL" test.)
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Oct 7, 2024
In commit db4e3b2 of PR git-lfs#2811 in 2018 we updated the progress meter text of Git LFS from "Git LFS" to the more explicit "Uploading LFS objects" or "Downloading LFS objects". A large number of were changed at this time to check for the revised text strings in the output logs the capture from Git and Git LFS operations. However, the "cloneSSL" test in our t/t-clone.sh test script was not updated because it does not actually run to completion, due to a long-standing bug in the conditional used to check the TRAVIS variable. As described in git-lfs#5658, the test instead always exits early and declares success. We expect to address this problem in a subsequent commit in this PR, so we first update the test to check for the same progress meter text as all our other tests, which will help ensure it succeeds once it is fully enabled. We also take the opportunity to add a similar check of the same progress meter message to the "clone ClientCert" test. This brings our tests into slightly closer alignment with each other, which should help ensure they stay consistent in the future. (Note that this test also does not run to completion at the moment, for the same reason as the "cloneSSL" test.)
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Oct 9, 2024
In commit db4e3b2 of PR git-lfs#2811 we updated the progress meter text of the Git LFS client from "Git LFS" to the more explicit "Uploading LFS objects" or "Downloading LFS objects". A large number of tests were changed in the same commit because they check the progress meter output in logs they capture from Git LFS operations. However, the "cloneSSL" test in our t/t-clone.sh test script was not updated because it has not actually run to completion for a long time, as described in git-lfs#5658, so it did not fail when the progress meter text changed. Due to a bug in the conditional used to check the TRAVIS variable, the test instead always exits early and declares success. We will address this problem in a subsequent commit in this PR, at which point the test will fail unless we adjust it to check for the same progress meter text as all our other tests, so we do that now. We also take the opportunity to add a similar check of the same progress meter message to the "clone ClientCert" test. This brings our tests in the t/t-clone.sh script into slightly closer alignment with each other, which should help ensure they stay consistent in the future. (Note that this test does not run to completion at the moment either, for the same reason as the "cloneSSL" test.)
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Oct 9, 2024
In commit db4e3b2 of PR git-lfs#2811 we updated the progress meter text of the Git LFS client from "Git LFS" to the more explicit "Uploading LFS objects" or "Downloading LFS objects". A large number of tests were changed in the same commit because they check the progress meter output in logs they capture from Git LFS operations. However, the "cloneSSL" test in our t/t-clone.sh test script was not updated because it has not actually run to completion for a long time, as described in git-lfs#5658, so it did not fail when the progress meter text changed. Due to a bug in the conditional used to check the TRAVIS variable, the test instead always exits early and declares success. We will address this problem in a subsequent commit in this PR, at which point the test will fail unless we adjust it to check for the same progress meter text as all our other tests, so we do that now. We also take the opportunity to add a similar check of the same progress meter message to the "clone ClientCert" test. This brings our tests in the t/t-clone.sh script into slightly closer alignment with each other, which should help ensure they stay consistent in the future. (Note that this test does not run to completion at the moment either, for the same reason as the "cloneSSL" test.)
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Oct 29, 2024
In commit db4e3b2 of PR git-lfs#2811 we updated the progress meter text of the Git LFS client from "Git LFS" to the more explicit "Uploading LFS objects" or "Downloading LFS objects". A large number of tests were changed in the same commit because they check the progress meter output in logs they capture from Git LFS operations. However, the "cloneSSL" test in our t/t-clone.sh test script was not updated because it has not actually run to completion for a long time, as described in git-lfs#5658, so it did not fail when the progress meter text changed. Due to a bug in the conditional used to check the TRAVIS variable, the test instead always exits early and declares success. We will address this problem in a subsequent commit in this PR, at which point the test will fail unless we adjust it to check for the same progress meter text as all our other tests, so we do that now. We also take the opportunity to add a similar check of the same progress meter message to the "clone ClientCert" test. This brings our tests in the t/t-clone.sh script into slightly closer alignment with each other, which should help ensure they stay consistent in the future. (Note that this test does not run to completion at the moment either, for the same reason as the "cloneSSL" test.)
chrisd8088
added a commit
to manturovDan/git-lfs
that referenced
this pull request
Mar 5, 2026
In subsequent commits in this PR we intend to add new tests to validate the changes we made to the Git LFS client in prior commits of this PR. We plan to add these tests to both the "t/t-askpass.sh" and the "t/t-credentials.sh" scripts in our shell test suite. Before we add our new tests, we first update the existing "credentials with useHttpPath, with wrong password" test in the "t/t-credentials.sh" script, and add a similar test to the "t/t-askpass.sh" script, which at present does not contain a test with an invalid set of credentials. We will then be able to use these tests as templates for the additional tests we expect to add in future commits. First, we revise the "credentials with useHttpPath, with wrong password" test to correct a check which at present will always trivially succeed. After the test runs the "git push" command, it attempts to confirm that the command failed to push a Git LFS object by checking that the command did not output a final progress message stating that the object was uploaded. To verify the lack of this message, the test uses the -c option of the grep(1) command and then checks that the returned value is zero. When the original version of this test was first introduced in commit 69aee53 of PR git-lfs#561, the "grep" command searched for the pattern "(1 of 1 files)" in the output of the "git push" command, which was a valid check given the format of the Git LFS client's progress messages at the time. We then revised the format of the client's progress messages in commit db4e3b2 of PR git-lfs#2811, and updated many of the "grep" commands in our test suite to reflect the new format at the same time. Most of these alterations were made correctly, but in the "credentials with useHttpPath, with wrong password" test the "grep" command's pattern was changed to "Uploading LFS objects: 100% (1/1), 0 B". Even if the "git push" command in the test were to unexpectedly succeed, the trailing "0 B" portion of this pattern would still not match any lines in the command's output, since the size of the test Git LFS object to be pushed is actually one byte. Thus this check will always pass, regardless of the success or failure of the "git push" command. To address this problem we simply remove the trailing portion of the pattern, as we only need to confirm that the "git push" command's output does not match the pattern "100% (1/1)". We then add another, final check to the test which uses a different technique to verify that the "git push" command did not upload a Git LFS object. Since we have already computed the OID of the object, we can call our refute_server_object() test assertion function to confirm that the object does not exist on the Git LFS remote. This is a more direct and less fragile method of checking the behaviour of the "git push" command than parsing its output messages, and should help avoid any future regressions of the type we have fixed in this commit. As well as making these changes to the "credentials with useHttpPath, with wrong password" test, we also add several new checks which use "grep" commands to ascertain that HTTP authorization errors occurred during the "git push" command. Specifically, we check that the client output two error messages, one after its request to the Git LFS lock verification API endpoint received a 403 status code response, and another after the client's request to the Git LFS Batch API endpoint received a 403 status code response. While adding these extra checks we also adjust the whitespace in the test, and we drop an "echo" statement whose value is minimal, even for debugging purposes. Finally, we replicate portions of the design of this test to create a new "askpass: push with core.askPass and wrong password" test in our "t/t-askpass.sh" script. This test establishes the same conditions as those from the existing "askpass: push with core.askPass" test, except that we set the LFS_ASKPASS_PASSWORD environment variable to contain an invalid password. As in our revised "credentials with useHttpPath, with wrong password" test, our new "askpass: push with core.askPass and wrong password" test then verifies the "git push" command made with the invalid credentials did not upload a Git LFS object, and that two 403 status code responses were received, one from the lock verification API endpoint and one from the Batch API endpoint.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request adopts the recommendation for a new style of progress meter proposed by @bitinn and @technoweenie in #2802, and #2802 (comment), respectively:
The style that is implemented in this pull request is exactly as @technoweenie's recommendation above:
Here's a large overview of the changes:
Remove the notion of 'skipped' files. This term has been used to indicate files that are unnecessary to re-transfer, for they are already present on either the client (in the case of a fetch) or the server (in the case of a push). This term has unfortunately also been the source of some confusion by users. To remove this source of confusion, let's instead treat "skipped" files as already having been transferred, and instead of incrementing a "skipped" count, simply increment the number of files and bytes as already having been transferred, without affecting the true transfer speed.
Introduce a transfer rate indicator. To match the style of progress meter used by Git to indicate push/pull operations, introduce to Git LFS's progress meter the average rate at which the transfer is occurring.1 To keep track of this, each time we receive a read update, we first check if it has been sufficiently long enough since the last time we calculated an average (to avoid a false sense of precision) and, if so, recalculate the average rate of transfer by using a Cumulative Moving Average (C.M.A) function, so as to weight more heavily recent transfer speeds, so that they dominate the average.
The average is calculated as:
Where:
avgis the average rate of transfer (given as bytes/second)nis the number of samples takensis the number of bytes transferred since the last sample was takendis the duration of time that has passed since the last sampleI am hoping that this new formatting will clarify what is going on when LFS uploads and downloads objects. The majority of this pull request is changing outdated assertions in the integration suite to match the new formatting.
Closes: #2802.
1: This is actually something that I've been wanting to add to LFS for some time now, but haven't found the right time to do it. I have published this Gist around a year ago at the time of writing this pull request. That code was originally intended for something exactly like this.
/cc @git-lfs/core #2802