tool_operate: keep failed partial download for retry auto-resume #15333
Conversation
|
Analysis of PR #15333 at 5d9ddcda: Test 498 failed, which has NOT been flaky recently, so there could be a real issue in this PR. Generated by Testclutch |
| outs->bytes = 0; | ||
| config->resume_from = outs->init; | ||
| curl_easy_setopt(curl, CURLOPT_RESUME_FROM_LARGE, | ||
| config->resume_from); |
There was a problem hiding this comment.
Isn't there a chance/risk that this option might get set in a retry scenario due to all the conditions being correct, but then the next attempt fails again and then in a second retry round the conditions are different and this line is not executed?
If so, the CURLOPT_RESUME_FROM_LARGE value set in the previous round will remain set, which then probably is wrong?
There was a problem hiding this comment.
If so, the
CURLOPT_RESUME_FROM_LARGEvalue set in the previous round will remain set, which then probably is wrong?
I updated outs->init and config->resume_from so let's say your scenario happens then it will truncate from the the updated resume from position which is still correct. The file is truncated to outs->init bytes. truncation can still happen if the conditions are not met.
There was a problem hiding this comment.
If so, the
CURLOPT_RESUME_FROM_LARGEvalue set in the previous round will remain set, which then probably is wrong?I updated outs->init and config->resume_from so let's say your scenario happens then it will truncate from the the updated resume from position which is still correct. The file is truncated to outs->init bytes. truncation can still happen if the conditions are not met.
@bagder I added a test and I made it cover what I think is your scenario. The test takes approximately 5 seconds to run because it has to make several retry requests to cover different scenarios and the minimum delay time for each retry is 1 second unless there is some way around that I'm not thinking of.
- Keep data from a failed download instead of discarding it on retry in some limited cases when we know it's ok (currently only HTTP 200/206). Prior to this change on failed transfer the tool truncated any outfile data written before retrying the transfer. This change adds an exception for HTTP downloads when the user requested auto-resume, because in that case we can keep the outfile data and resume from the new position. Closes #xxxx
5d9ddcd to
cf1f3e5
Compare
if the requested range is open ended then a server usually replies with the rest of the content. note however this response i changed is intentionally set to return less than content length to cause a retry
Prior to this change on failed transfer the tool truncated any outfile data written before retrying the transfer. This change adds an exception for HTTP downloads when the user requested auto-resume, because in that case we can keep the outfile data and resume from the new position.
Closes #xxxx
I tested this briefly last night and it seems to work as expected. (I've since added some more conditionals..)