[HttpListener] Hang of System.Net.HttpListener.Tests in CI #21940

danmoseley · 2017-05-24T15:37:55Z

https://ci.dot.net/job/dotnet_corefx/job/master/job/ubuntu14.04_debug_prtest/8257/consoleFull#130661499679494335-f7bd-47d0-8771-8661e00c2db2

22:05:43   Finished running tests. End time=05:05:43. Return value was 0
23:44:38 
Build timed out (after 120 minutes). Marking the build as aborted.
23:44:38 
Build was aborted
23:44:38 [xUnit] [INFO] - Starting to record.
23:44:38 [xUnit] [INFO] - Processing xUnit.Net-v2 (default)
23:44:38 
MSBUILD : error MSB4166: Child node "2" exited prematurely. Shutting down. Diagnostic information may be found in files in the temporary files directory named MSBuild_*.failure.txt.
23:44:38 [xUnit] [INFO] - [xUnit.Net-v2 (default)] - 155 test report file(s) were found with the pattern 'bin/**/testResults.xml' relative to '/mnt/j/workspace/dotnet_corefx/master/ubuntu14.04_debug_prtest' for the testing framework 'xUnit.Net-v2 (default)'.
23:44:38 /mnt/j/workspace/dotnet_corefx/master/ubuntu14.04_debug_prtest/Tools/tests.targets(345,5): warning MSB5021: "sh" and its child processes are being terminated in order to cancel the build. [/mnt/j/workspace/dotnet_corefx/master/ubuntu14.04_debug_prtest/src/System.Net.HttpListener/tests/System.Net.HttpListener.Tests.csproj]
23:44:39 Command execution failed with exit code 143.

Not actionable based on this information, but starting an issue.

The text was updated successfully, but these errors were encountered:

danmoseley · 2017-05-24T20:10:28Z

fyi @hughbe @stephentoub note these are in master.

danmoseley · 2017-05-24T20:10:38Z

Debian & Ubuntu:
https://mc.dot.net/#/user/mmitche/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fportable~2Fcli~2F/6c6eb50345329c470db608fef70710e52270c507/workItem/System.Net.HttpListener.Tests

karelz · 2017-05-24T22:08:43Z

Questions:

Does it happen only on Linux/Mac (product issue), or also on Windows (test issue)

mellinoe · 2017-05-25T00:01:04Z

If we see this actively hung in any CI jobs, please let me know and I will try to debug it. Bonus points to anyone who logs into a machine and attaches a debugger before the process terminates 😄

danmoseley · 2017-05-25T05:12:54Z

All the time now.
https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fportable~2Fcli~2F/build/20170525.01/workItem/System.Net.HttpListener.Tests/wilogs

stephentoub · 2017-05-25T05:28:30Z

Managed to catch one in CI. At least in the one I saw, it appears to be hanging here:
https://github.com/dotnet/corefx/blob/bbb63de0aac837895542e1b126ef71b3ba1a51aa/src/System.Net.HttpListener/tests/HttpRequestStreamTests.cs#L68

@geoffkizer, looks like this might be the same hang you're investigating locally...

geoffkizer · 2017-05-25T05:30:27Z

Yep, that's exactly what I see locally.

danmoseley · 2017-05-25T15:34:30Z

https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fportable~2Fcli~2F/build/20170525.02/workItem/System.Net.HttpListener.Tests/wilogs

danmoseley · 2017-05-25T16:52:46Z

@stephentoub @geoffkizer does this mean if we disable Read_FullLengthAsynchronous_Success it will avoid this hang? If so could one of you please disable it?
There is no need to go through CI for this, as it's so impactful I suggest hitting merge immediately.

stephentoub · 2017-05-25T17:08:45Z

does this mean if we disable Read_FullLengthAsynchronous_Success it will avoid this hang?

I don't think so; I think there are several (not sure how many) tests that hit the same underlying issue or issues. I don't believe it's a problem with a specific test, but rather with something in the implementation (or something systemic to many tests via code that's shared between them). It's possible we could find and disable enough tests to avoid the hangs.

@geoffkizer has been investigating...

stephentoub · 2017-05-25T17:22:33Z

But I'll see if I can temporarily disable enough to minimize the impact...

stephentoub · 2017-05-25T20:15:26Z

A bunch of tests have been temporarily disabled. Just search for 20246 to find them. Let's try to get to the bottom of this asap. Hopefully I disabled enough to prevent the hangs in CI.

danmoseley · 2017-05-26T03:43:00Z

Helix run failed again but I don't know whether it contained that change.
https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170526.01/workItem/System.Net.HttpListener.Tests/wilogs

KristinXie1 · 2017-05-31T06:57:13Z

Failed again here: https://mc.dot.net/#/product/netcore/200/source/official~2Fcorefx~2Frelease~2F2.0.0~2F/type/test~2Ffunctional~2Fportable~2Fcli~2F/build/20170531.01/workItem/System.Net.HttpListener.Tests/wilogs

karelz · 2017-05-31T07:10:42Z

@mellinoe was there any progress on tracking down the root cause of the hangs?

mellinoe · 2017-05-31T19:29:23Z

@karelz I think @stephentoub had caught this (or a related test) under a debugger at one point. I was not able to successfully get a local repro despite running the tests in a loop for a long time.

stephentoub · 2017-05-31T19:30:22Z

@geoffkizer and I fixed the issues we knew about. It's possible there's another lurking.

KristinXie1 · 2017-06-01T07:37:06Z

Failed again: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170601.05/workItem/System.Net.HttpListener.Tests/wilogs

danmoseley · 2017-06-01T22:23:29Z

Hung again. OSX. Last 3 were all OSX.

https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170601.06/workItem/System.Net.HttpListener.Tests/wilogs

stephentoub · 2017-06-01T22:25:30Z

Might be worth running the tests in a loop on macOS for a while and seeing if we can get something to hang.

karelz · 2017-06-01T22:36:05Z

@mellinoe can you please help? (if you don't have Mac, we could ask @wfurt to run it on his Mac ...)

mellinoe · 2017-06-01T22:45:28Z

I borrowed a MacBook and will try running the tests in a loop here, once I've got my environment sync'd up.

danmoseley · 2017-06-02T01:58:47Z

This is now the most impactful of all corefx test issues.

mellinoe · 2017-06-02T23:05:12Z

@stephentoub and I spent several hours debugging this earlier today, and we weren't able to identify the exact problem yet. We saw some strange things that made us doubt the validity of the info coming from lldb/SOS, and ended up stumped for now.

At the very least, the MacBook I'm using is able to reproduce the issue almost every single time I run the tests. So further investigations should be easy to do.

KristinXie1 · 2017-06-06T02:24:03Z

Failed again: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170606.01/workItem/System.Net.HttpListener.Tests/wilogs

KristinXie1 · 2017-06-06T07:22:27Z

This issue is repro on Core Tests (2.0.0), detail: https://mc.dot.net/#/product/netcore/200/source/official~2Fcorefx~2Frelease~2F2.0.0~2F/type/test~2Ffunctional~2Fcli~2F/build/20170605.01/workItem/System.Net.HttpListener.Tests/wilogs

KristinXie1 · 2017-06-07T06:21:38Z

Failed again on Core Tests (Master) in build 20170607.01, detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170607.01/workItem/System.Net.HttpListener.Tests/wilogs

KristinXie1 · 2017-06-12T06:49:08Z

Failed again, detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170612.01/workItem/System.Net.HttpListener.Tests

Priya91 · 2017-06-12T23:09:12Z

@mellinoe Can you share the repro that you used for this.. Did you run the httplistener tests in loop or full test run? Were you able to identify the test that's hanging?

mellinoe · 2017-06-13T02:47:58Z

@Priya91 There are several tests that hang. It repros on your laptop every time, just doing a regular run of the test project by itself.

KristinXie1 · 2017-06-14T06:01:36Z

Failed again: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170614.01/workItem/System.Net.HttpListener.Tests/wilogs

KristinXie1 · 2017-06-15T02:01:05Z

This issue is repro on Core Tests (2.0.0), detail: https://mc.dot.net/#/product/netcore/200/source/official~2Fcorefx~2Frelease~2F2.0.0~2F/type/test~2Ffunctional~2Fcli~2F/build/20170615.01/workItem/System.Net.HttpListener.Tests/wilogs

KristinXie1 · 2017-06-30T06:29:46Z

This issue is repro on OSX.1013, detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170630.01/workItem/System.Net.HttpListener.Tests/wilogs

danmoseley · 2017-06-30T16:38:29Z

Need an update from @mellinoe who has been investigating this.

mellinoe · 2017-06-30T18:03:13Z

I've debugged through a few repros at this point, but I haven't identified a root cause. One repro was hung in this method, https://github.com/dotnet/corefx/blob/master/src/System.Net.HttpListener/src/System/Net/Managed/HttpListenerRequest.Managed.cs#L302, in what could potentially be an infinite loop. That's the closest we've gotten to the bottom of this.

danmoseley · 2017-07-03T18:25:43Z

Per @karelz moving to 2.0
• OSX only
• Not clear it's a product bug.
• HttpListener is compat API only (we do not plan to evolve it)
• ASP.NET does not depend on it
• If it is test bug, we can fix it in servicing branch.
• If it is product bug we can service.

wfurt · 2018-06-08T04:17:13Z

I think the last association is incorrect: [UPDATE] The referenced comment was hidden as off-topic by @karelz

2017-07-16 17:48:30,947: INFO: proc(54): run_and_log_output: Output: System.Net.Sockets.SocketException : Too many open files in system

This seems to be clearly environmental issues.
As far as I can tell, we have no seen this one in long time.
I suggest to close it @karelz and we can open it back (or create new one) if it ever happens again.

karelz · 2018-06-08T17:40:13Z

Closing per @wfurt's analysis above.

stephentoub · 2020-01-23T03:05:07Z

There are two tests disabled against this issue:

runtime/src/libraries/System.Net.HttpListener/tests/HttpResponseStreamTests.cs

Lines 380 to 385 in 04f2226

    
           [ActiveIssue("https://github.com/dotnet/corefx/issues/20246")] // CI hanging frequently 
        
           [ActiveIssue("https://github.com/dotnet/corefx/issues/19534", TestPlatforms.OSX)] 
        
           [Theory] 
        
           [InlineData(true)] 
        
           [InlineData(false)] 
        
           public async Task Write_HeadersToClosedConnectionAsynchronously_ThrowsHttpListenerException(bool ignoreWriteExceptions)

runtime/src/libraries/System.Net.HttpListener/tests/HttpResponseStreamTests.cs

Lines 419 to 424 in 04f2226

    
           [ActiveIssue("https://github.com/dotnet/corefx/issues/20246")] // CI hanging frequently 
        
           [ActiveIssue("https://github.com/dotnet/corefx/issues/19534", TestPlatforms.OSX)] 
        
           [Theory] 
        
           [InlineData(true)] 
        
           [InlineData(false)] 
        
           public async Task Write_HeadersToClosedConnectionSynchronously_ThrowsHttpListenerException(bool ignoreWriteExceptions)

karelz assigned mellinoe May 24, 2017

karelz changed the title ~~Hang of System.Net.HttpListener.Tests in CI~~ [HttpListener] Hang of System.Net.HttpListener.Tests in CI May 24, 2017

stephentoub assigned geoffkizer May 25, 2017

stephentoub self-assigned this May 26, 2017

karelz unassigned stephentoub and geoffkizer Jun 1, 2017

This comment has been minimized.

Sign in to view

karelz unassigned mellinoe Dec 1, 2017

karelz closed this as completed Jun 8, 2018

stephentoub reopened this Jan 23, 2020

msftgits transferred this issue from dotnet/corefx Jan 31, 2020

msftgits added this to the Future milestone Jan 31, 2020

karelz modified the milestones: Future, 5.0 Feb 23, 2020

karelz modified the milestones: 5.0, Future May 7, 2020

wfurt self-assigned this Oct 23, 2020

wfurt mentioned this issue Oct 24, 2020

Write_HeadersToClosedConnectionSynchronously_ThrowsHttpListenerException test fails on macOS #21590

Open

wfurt removed their assignment Jun 15, 2021

[HttpListener] Hang of System.Net.HttpListener.Tests in CI #21940

[HttpListener] Hang of System.Net.HttpListener.Tests in CI #21940

Comments

danmoseley commented May 24, 2017

danmoseley commented May 24, 2017

danmoseley commented May 24, 2017 • edited by karelz Loading

karelz commented May 24, 2017

mellinoe commented May 25, 2017

danmoseley commented May 25, 2017

stephentoub commented May 25, 2017 • edited Loading

geoffkizer commented May 25, 2017

danmoseley commented May 25, 2017

danmoseley commented May 25, 2017

stephentoub commented May 25, 2017 • edited Loading

stephentoub commented May 25, 2017

stephentoub commented May 25, 2017 • edited Loading

danmoseley commented May 26, 2017

KristinXie1 commented May 31, 2017

karelz commented May 31, 2017

mellinoe commented May 31, 2017

stephentoub commented May 31, 2017

KristinXie1 commented Jun 1, 2017

danmoseley commented Jun 1, 2017

stephentoub commented Jun 1, 2017

karelz commented Jun 1, 2017

mellinoe commented Jun 1, 2017

danmoseley commented Jun 2, 2017

mellinoe commented Jun 2, 2017

KristinXie1 commented Jun 6, 2017

KristinXie1 commented Jun 6, 2017

KristinXie1 commented Jun 7, 2017

KristinXie1 commented Jun 12, 2017

Priya91 commented Jun 12, 2017 • edited Loading

mellinoe commented Jun 13, 2017

KristinXie1 commented Jun 14, 2017

KristinXie1 commented Jun 15, 2017

KristinXie1 commented Jun 30, 2017

danmoseley commented Jun 30, 2017

mellinoe commented Jun 30, 2017

danmoseley commented Jul 3, 2017

This comment has been minimized.

wfurt commented Jun 8, 2018 • edited by karelz Loading

karelz commented Jun 8, 2018

stephentoub commented Jan 23, 2020

danmoseley commented May 24, 2017 •

edited by karelz

Loading

stephentoub commented May 25, 2017 •

edited

Loading

stephentoub commented May 25, 2017 •

edited

Loading

stephentoub commented May 25, 2017 •

edited

Loading

Priya91 commented Jun 12, 2017 •

edited

Loading

wfurt commented Jun 8, 2018 •

edited by karelz

Loading