Codestin Search App

adityapatwardhan · 2017-12-29T20:25:44Z

PR Summary

The change added two parameters, RetryCount and RetryIntervalSec to enable retry functionality.
When retrying a verbose message is sent out to inform the user.

PR Checklist

Note: Please mark anything not applicable to this PR NA.

PR has a meaningful title
- Use the present tense and imperative mood when describing your changes
Summarized changes
User facing Documentation needed
- Issue filed - Issue link: Add documentation for RetryCount and RetryIntervalSec MicrosoftDocs/PowerShell-Docs#1991
Change is not breaking
Make sure you've added a new test if existing tests do not effectively test the code changed
- Add [feature] if the change is significant or affectes feature tests
This PR is ready to merge and is not Work in Progress.
- If the PR is work in progress, please add the prefix WIP: to the beginning of the title and remove the prefix when the PR is ready.

markekraus · 2017-12-29T21:10:58Z

...t.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs

Please add a more meaningful summary. Capitalize the first word and end with a period.

markekraus · 2017-12-29T21:11:34Z

...t.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs

Please add a more meaningful summary. Capitalize the first word and end with a period.

markekraus · 2017-12-29T21:13:02Z

...t.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs

Can you investigate collapsing this to a single do/while instead of nested?

I think we should have the retry logic in GetResponse() rather than ProcessRecord().

Moved logic to GetResponse()

markekraus · 2017-12-29T21:23:26Z

...t.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs

We should probably dispose the current HttpResponseMessage before retrying.

markekraus · 2017-12-29T21:23:49Z

...t.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs

This should move to the end of the previous line.

markekraus · 2017-12-29T21:36:19Z

...t.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs

I believe it would be better to have RetryCount and RetryIntervalSec set on WebSession in PrepareSession() and then have this area of code retrieve the values from WebSession. It would mean extending WebRequestSession to include new Properties, but I believe that is the correct direction for these settings.

markekraus · 2017-12-29T21:51:23Z

...t.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs

This current logic leads to an interestingly weird combination. If you set -MaximumRedirection 0 -RetryCount 5 -RetryIntervalSec 1 and the response code is 301, this will still retry the same link 5 times because HttpResponseMessage.IsSuccessStatusCode is only true for response codes 200-299 inclusive... not sure how to rectify that.

also... since the logic for reducing the current redirection count is handled in GetResponse(), we may need to reset WebSession.MaximumRedirection to the MaximumRedirection property value before continuing.

Best that I can find, the HTTP spec doesn't say enough about rate limiting to make it something we can depend on reliably. However, I think we can say that retry only occurs on a 429 or 403 and document it as such unless there's data showing that web services are using other error codes.

@SteveL-MSFT I can see other usecases for this beyond just rate limiting. I could see using this to test 404 errors until a new page comes online after a publish/push and it $x attempts fail alert that the push/publish failed.

Perhaps it we limit it to 400-499 inclusive it would be best. In my experience with APIs I have seen more than 403 and 429 used for busy endpoints (including 404... Reddit... ugh) or rate limiting, but it's always a 400 level error

@markekraus that's a fair point. I think 4xx would be fine.

After further consideration, this should work for the following status codes: 304. 400-599 inclusive

304 could be used for monitoring the change of a resource and is not a redirect status even though it in the 300 range. Retries on 5XX errors makes perfect sense to overcome temporary service glitches and cloudflare shenanigans.

markekraus · 2017-12-29T22:03:35Z

...t.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs

See my comment about setting and retrieving RetryIntervalSec and RetryCount from WebSession.

SteveL-MSFT · 2018-01-02T19:39:59Z

...t.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs

Perhaps we should default to 1 min instead of 10 secs to be more polite.

SteveL-MSFT · 2018-01-02T19:55:08Z

...t.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs

I think we should have the retry logic in GetResponse() rather than ProcessRecord().

SteveL-MSFT · 2018-01-02T20:10:50Z

...t.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs

Best that I can find, the HTTP spec doesn't say enough about rate limiting to make it something we can depend on reliably. However, I think we can say that retry only occurs on a 429 or 403 and document it as such unless there's data showing that web services are using other error codes.

SteveL-MSFT · 2018-01-02T20:12:24Z

test/powershell/Modules/Microsoft.PowerShell.Utility/WebCmdlets.Tests.ps1

This won't work once we start localizing the error messages

I don't really like this test. I'd rather have a real failure emulated by WebListener. Add a new endpoint which take a session ID (new GUID), a status code, and a fail count. Until the initially provided fail count is reached when using that session ID it produces a response with the provided status code After fail count is reached it returns a normal 200 response. The response body should include the request count for the given session ID. This way we can test how it reacts with various responses and differing fail counts provided by actual endpoint errors instead of relying on fake logic to enter the code path and without relying on verbose messages.

iSazonov · 2018-02-21T16:13:04Z

@adityapatwardhan Could you please continue?

iSazonov · 2018-03-30T14:47:23Z

@adityapatwardhan Please resolve conflicts.

adityapatwardhan · 2018-03-30T15:56:08Z

I plan to resume work on this next week.

ffeldhaus · 2018-04-05T22:07:36Z

Did you consider implementing Exponential Backof or different retry strategies? AWS SDKs for instance use an Exponential Backof and it is good practice for many other web services.

iSazonov · 2018-04-06T04:59:58Z

@ffeldhaus Please open new Issue for your great suggestion - we shouldn't lost this.

ffeldhaus · 2018-04-06T18:53:08Z

I thought about this feature some more and I don't think it should be included in the current form without further discussion to prevent retries to cause issues or unwanted behavior and I don't think a Pull Request is the right place to discuss the design of this feature. As issue #5582 was already created for this feature, we should first discuss the design there and then continue with the Pull Request.

I will add my concerns and design ideas to issue #5582.

stale · 2018-05-06T19:12:56Z

This PR has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs within 10 days.
Thank you for your contributions.
Community members are welcome to grab these works.

iSazonov · 2018-05-10T14:22:32Z

We have #6447 - should we close the Issue?

/cc @SteveL-MSFT @markekraus

markekraus · 2018-05-10T15:45:35Z

@iSazonov Retry and Resume are separate features. Resume continues downloading a partially downloaded file. Retry tries the URI again if a non-susccess status code was received. This PR is for the Retry feature, #6447 is for the Resume feature.

stale · 2018-06-09T16:32:39Z

This PR has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs within 10 days.
Thank you for your contributions.
Community members are welcome to grab these works.

iSazonov · 2018-06-14T05:20:58Z

I wouldn't want to lose this.

adityapatwardhan · 2018-06-26T01:22:34Z

@markekraus Updated test and controller with echo for sessionId.

markekraus · 2018-06-26T08:43:51Z

@adityapatwardhan Two more things and I think this will be done. I just realized there are only tests for Invoke-Webrequest. Please add the same tests for Invoke-RestMethod. Also, instead of wrapping the tests in #region Retry tests, please use a Context block.

adityapatwardhan · 2018-06-26T16:17:57Z

@markekraus Since the code path is same, I believe adding all the tests will be duplication. I will add one test for Invoke-RestMethod to cover the scenario.

markekraus · 2018-06-26T17:00:57Z

@adityapatwardhan The convention for the web cmdlets is to test all features on both, even common features. There is always the possibility that the common features clash with the implementation features.

adityapatwardhan · 2018-06-26T17:43:40Z

@markekraus Please review again. I have added tests for Invoke-RestMethod. I moved them to a new describe block so they can be grouped together. The describes for other Invoke-RestMethod and Invoke-WebRequest tests were separate. Thanks for the feedback, very much appreciated.

markekraus · 2018-06-26T18:00:04Z

@adityapatwardhan there is a Describe "Invoke-WebRequest tests" -Tags "Feature" block and a Describe "Invoke-RestMethod tests" -Tags "Feature" and in those are various context blocks

for example,

PowerShell/test/powershell/Modules/Microsoft.PowerShell.Utility/WebCmdlets.Tests.ps1

Line 827 in e8d2a6b

Context "Invoke-WebRequest SkipHeaderVerification Tests" {

and

PowerShell/test/powershell/Modules/Microsoft.PowerShell.Utility/WebCmdlets.Tests.ps1

Line 2119 in e8d2a6b

Context "Invoke-RestMethod SkipHeaderVerification Tests" {

adityapatwardhan · 2018-06-27T17:05:39Z

@markekraus I decided to go with a separate Describe block so that we can use TestCase for It and re-use most of the test code. If we have tests for Invoke-RestMethod and Invoke-WebRequest in separate Describe blocks then there will be test code duplication. I prefer a separate Describe but since there is precedence I am not completely against adding them to existing Describe blocks.

Let me know your thoughts?

markekraus · 2018-06-27T17:10:17Z

@adityapatwardhan For all new features they have been separate context blocks in the top level describe blocks for each cmdlet. I'd prefer to stick to this pattern for now and look at changing things up in one of the open issues I have on refactoring the code and tests.

adityapatwardhan · 2018-06-27T17:29:57Z

@markekraus Ok, agreed. I will send an update in a few minutes.

adityapatwardhan · 2018-06-27T17:54:04Z

@markekraus Made the changes to the tests as discussed above.

markekraus · 2018-06-27T17:55:50Z

test/powershell/Modules/Microsoft.PowerShell.Utility/WebCmdlets.Tests.ps1

+
+    Context "Invoke-WebRequest retry tests" {
+
+        It "<Command> can retry - <Name>" -TestCases @(


<Command> -> Invoke-WebRequest since it's no longer in the testcases hash.

markekraus · 2018-06-27T17:56:53Z

test/powershell/Modules/Microsoft.PowerShell.Utility/WebCmdlets.Tests.ps1

+            $jsonError.error | Should -BeExactly 'Error: HTTP - 400 occurred.'
+        }
+
+        It "<Command> can retry with POST" {


<Command> -> Invoke-WebRequest since it's no longer in the testcases hash.

adityapatwardhan · 2018-06-27T18:00:34Z

@markekraus Good catch! Fixed.

markekraus

One final thing. Sorry :(

markekraus · 2018-06-27T18:07:35Z

test/tools/WebListener/README.md

+
+```json
+{
+  "failureResponsesSent":2


Can you update this to include the sessionId that is now also returned.

adityapatwardhan · 2018-06-27T19:10:33Z

@markekraus Fixed documentation.

markekraus

@adityapatwardhan Thanks for you patience and diligence. LGTM

iSazonov · 2018-06-28T05:07:15Z

Reopen to restart CIs.

adityapatwardhan · 2018-06-28T17:21:14Z

@iSazonov Thanks for restarting CIs. All tests passed, ready to merge?

iSazonov · 2018-06-29T04:07:22Z

@adityapatwardhan Thanks for great contribution!

adityapatwardhan added the Documentation Needed in this repo Documentation is needed in this repo label Dec 29, 2017

adityapatwardhan added this to the 6.1.0-Consider milestone Dec 29, 2017

adityapatwardhan requested review from SteveL-MSFT and markekraus December 29, 2017 20:25

adityapatwardhan requested review from BrucePay, JamesWTruher, PaulHigin, dantraMSFT, daxian-dbw and lzybkr as code owners December 29, 2017 20:25

adityapatwardhan mentioned this pull request Dec 29, 2017

Add documentation for RetryCount and RetryIntervalSec MicrosoftDocs/PowerShell-Docs#1991

Closed

7 tasks

markekraus suggested changes Dec 29, 2017

View reviewed changes

SteveL-MSFT requested changes Jan 2, 2018

View reviewed changes

daxian-dbw assigned iSazonov Jan 8, 2018

lzybkr removed their request for review March 14, 2018 16:35

ffeldhaus mentioned this pull request Apr 6, 2018

WebCmdlets should support retry #5582

Closed

stale bot added the Stale label May 6, 2018

stale bot removed the Stale label May 10, 2018

stale bot added the Stale label Jun 9, 2018

[Feature] Added tests for Invoke-RestMethod

23a8b3f

PaulHigin removed their request for review June 26, 2018 17:44

[Feature] Moved tests in existing Describe blocks

cecc58a

markekraus suggested changes Jun 27, 2018

View reviewed changes

[Feature] Fixed test title

661fcaa

markekraus suggested changes Jun 27, 2018

View reviewed changes

Fix documentation

e38edaa

markekraus approved these changes Jun 27, 2018

View reviewed changes

iSazonov closed this Jun 28, 2018

iSazonov reopened this Jun 28, 2018

iSazonov merged commit 15f6abe into PowerShell:master Jun 29, 2018

joeyaiello mentioned this pull request Oct 15, 2018

Docs needed for 'Added functionality to retry in Invoke-RestMethod and Invoke-WebRequest.' MicrosoftDocs/PowerShell-Docs#3057

Closed

joeyaiello removed Documentation Needed in this repo Documentation is needed in this repo labels Oct 15, 2018

adityapatwardhan deleted the WebCmdletRetry branch November 1, 2018 17:38


		Context "Invoke-WebRequest retry tests" {

		It "<Command> can retry - <Name>" -TestCases @(

Conversation

adityapatwardhan commented Dec 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

PR Checklist

Uh oh!

markekraus Dec 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markekraus Dec 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iSazonov commented Feb 21, 2018

Uh oh!

iSazonov commented Mar 30, 2018

Uh oh!

adityapatwardhan commented Mar 30, 2018

Uh oh!

ffeldhaus commented Apr 5, 2018

Uh oh!

iSazonov commented Apr 6, 2018

Uh oh!

ffeldhaus commented Apr 6, 2018

Uh oh!

stale bot commented May 6, 2018

Uh oh!

adityapatwardhan commented Dec 29, 2017 •

edited

Loading

markekraus Dec 29, 2017 •

edited

Loading

markekraus Dec 29, 2017 •

edited

Loading