Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Occasional crashes in System.Net.Sockets.Tests on Alpine 3.15 #85666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
antonfirsov opened this issue May 2, 2023 · 15 comments · Fixed by #114964
Closed

Occasional crashes in System.Net.Sockets.Tests on Alpine 3.15 #85666

antonfirsov opened this issue May 2, 2023 · 15 comments · Fixed by #114964
Assignees
@rzikm
Labels
area-System.Net.Sockets in-pr There is an active PR which will close this issue when it is merged test-run-core Test failures in .NET Core test runs
Milestone
10.0.0

Comments

@antonfirsov
Copy link
Member

We see rare crashes of System.Net.Sockets.Tests on Alpine 3.15 because of a SocketException (22): Invalid argument being thrown from a continuation callback invoked from SocketAsyncEngine / SocketAsyncContext:

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-heads-main-272f116de4bf4fea9f/System.Net.Sockets.Tests/1/console.48b03307.log?helixlogtype=result

Starting:    System.Net.Sockets.Tests (parallel test collections = on, max threads = 2)
Unhandled exception. System.Net.Sockets.SocketException (22): Invalid argument
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1297
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.AcceptAsync(Socket socket, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 978
   at System.Net.Sockets.Socket.AcceptAsync(Socket acceptSocket, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 70
   at System.Net.Sockets.Socket.AcceptAsync() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 32
   at System.Net.Sockets.Socket.BeginAccept(AsyncCallback callback, Object state) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 2505
   at System.Net.Sockets.Tests.SocketTestServerAPM.OnAccept(IAsyncResult result) in /_/src/libraries/Common/tests/System/Net/Sockets/SocketTestServerAPM.cs:line 82
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 185
   at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 702
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3452
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:line 398
   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ValueTask.cs:line 665
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 959
   at System.Net.Sockets.SocketAsyncEventArgs.OnCompletedInternal() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs:line 210
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32 bytesTransferred, SocketFlags flags) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs:line 983
   at System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32 bytesTransferred, SocketFlags flags, SocketError socketError) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs:line 376
   at System.Net.Sockets.SocketAsyncEventArgs.AcceptCompletionCallback(IntPtr acceptedFileDescriptor, Byte[] socketAddress, Int32 socketAddressSize, SocketError socketError) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs:line 30
   at System.Net.Sockets.SocketAsyncContext.AcceptOperation.InvokeCallback(Boolean allowPooling) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 616
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.ProcessAsyncOperation(TOperation op) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 973
   at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation op) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 1382
   at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 341
   at System.Net.Sockets.SocketAsyncContext.AsyncOperation.Process() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 292
   at System.Net.Sockets.SocketAsyncContext.HandleEvents(SocketEvents events) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 2187
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs:line 242
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 919
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs:line 58
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskToAsyncResult.End[TResult](IAsyncResult asyncResult) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskToAsyncResult.cs:line 61
   at System.Net.Sockets.Socket.EndAccept(IAsyncResult asyncResult) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 2507
   at System.Net.Sockets.Tests.SocketTestServerAPM.OnAccept(IAsyncResult result) in /_/src/libraries/Common/tests/System/Net/Sockets/SocketTestServerAPM.cs:line 52
   at System.Threading.Tasks.TaskToAsyncResult.TaskAsyncResult..ctor(Task task, Object state, AsyncCallback callback) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskToAsyncResult.cs:line 144
   at System.Threading.Tasks.TaskToAsyncResult.Begin(Task task, AsyncCallback callback, Object state) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskToAsyncResult.cs:line 43
   at System.Net.Sockets.Socket.BeginAccept(AsyncCallback callback, Object state) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 2505
   at System.Net.Sockets.Tests.SocketTestServerAPM.OnAccept(IAsyncResult result) in /_/src/libraries/Common/tests/System/Net/Sockets/SocketTestServerAPM.cs:line 82
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 185
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 209
   at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 702
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 1929
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 919
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs:line 58
@antonfirsov antonfirsov added this to the 8.0.0 milestone May 2, 2023
@ghost
Copy link

ghost commented May 2, 2023

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

We see rare crashes of System.Net.Sockets.Tests on Alpine 3.15 because of a SocketException (22): Invalid argument being thrown from a continuation callback invoked from SocketAsyncEngine / SocketAsyncContext:

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-heads-main-272f116de4bf4fea9f/System.Net.Sockets.Tests/1/console.48b03307.log?helixlogtype=result

Starting:    System.Net.Sockets.Tests (parallel test collections = on, max threads = 2)
Unhandled exception. System.Net.Sockets.SocketException (22): Invalid argument
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1297
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.AcceptAsync(Socket socket, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 978
   at System.Net.Sockets.Socket.AcceptAsync(Socket acceptSocket, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 70
   at System.Net.Sockets.Socket.AcceptAsync() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 32
   at System.Net.Sockets.Socket.BeginAccept(AsyncCallback callback, Object state) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 2505
   at System.Net.Sockets.Tests.SocketTestServerAPM.OnAccept(IAsyncResult result) in /_/src/libraries/Common/tests/System/Net/Sockets/SocketTestServerAPM.cs:line 82
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 185
   at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 702
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3452
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:line 398
   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ValueTask.cs:line 665
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 959
   at System.Net.Sockets.SocketAsyncEventArgs.OnCompletedInternal() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs:line 210
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32 bytesTransferred, SocketFlags flags) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs:line 983
   at System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32 bytesTransferred, SocketFlags flags, SocketError socketError) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs:line 376
   at System.Net.Sockets.SocketAsyncEventArgs.AcceptCompletionCallback(IntPtr acceptedFileDescriptor, Byte[] socketAddress, Int32 socketAddressSize, SocketError socketError) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs:line 30
   at System.Net.Sockets.SocketAsyncContext.AcceptOperation.InvokeCallback(Boolean allowPooling) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 616
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.ProcessAsyncOperation(TOperation op) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 973
   at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation op) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 1382
   at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 341
   at System.Net.Sockets.SocketAsyncContext.AsyncOperation.Process() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 292
   at System.Net.Sockets.SocketAsyncContext.HandleEvents(SocketEvents events) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 2187
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs:line 242
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 919
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs:line 58
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskToAsyncResult.End[TResult](IAsyncResult asyncResult) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskToAsyncResult.cs:line 61
   at System.Net.Sockets.Socket.EndAccept(IAsyncResult asyncResult) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 2507
   at System.Net.Sockets.Tests.SocketTestServerAPM.OnAccept(IAsyncResult result) in /_/src/libraries/Common/tests/System/Net/Sockets/SocketTestServerAPM.cs:line 52
   at System.Threading.Tasks.TaskToAsyncResult.TaskAsyncResult..ctor(Task task, Object state, AsyncCallback callback) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskToAsyncResult.cs:line 144
   at System.Threading.Tasks.TaskToAsyncResult.Begin(Task task, AsyncCallback callback, Object state) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskToAsyncResult.cs:line 43
   at System.Net.Sockets.Socket.BeginAccept(AsyncCallback callback, Object state) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 2505
   at System.Net.Sockets.Tests.SocketTestServerAPM.OnAccept(IAsyncResult result) in /_/src/libraries/Common/tests/System/Net/Sockets/SocketTestServerAPM.cs:line 82
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 185
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 209
   at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 702
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 1929
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 919
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs:line 58
Author: antonfirsov
Assignees: -
Labels:

area-System.Net.Sockets
Milestone: 8.0.0

@rzikm
Copy link
Member

rzikm commented May 3, 2023

I had a quick look in the crash dump and I could not find useful info (such as which test caused it). Some parts of the heap are inaccessible (dotnet-dump kept crashing) and there is no reference to a test state machine in the thread callstack :/

@antonfirsov
Copy link
Member Author

This comes from SocketTestServerAPM which is only used by some test cases in DnsEndPointTest. What SocketTestServerAPM does is to enforce APM API-s for the listener/server socket, which makes no sense to me from DNS testing POV -- the server is not the SUT there.

IMO it's very unlikely that we have a production bug here. If we don't want to bother to understand why is SocketTestServerAPM hitting an SocketException (22): Invalid argument, we could just simplify DnsEndPointTest and get rid of SocketTestServerAPM entirely.

@wfurt
Copy link
Member

wfurt commented May 3, 2023

man page documents this for accept:

       EINVAL Socket is not listening for connections, or addrlen is invalid (e.g., is negative).
       EINVAL (accept4()) invalid value in flags.

it may be interesting to dump the related Socket state.

@wfurt wfurt modified the milestones: 8.0.0, Future Jun 7, 2023
@liveans
Copy link
Member

liveans commented Aug 15, 2023 •
edited
Loading

Similar crash on Debian.11.Arm32.Open too.
Helix Log

Unhandled exception. System.Net.Sockets.SocketException (22): Invalid argument
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1387
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.AcceptAsync(Socket socket, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1050
   at System.Net.Sockets.Socket.AcceptAsync(Socket acceptSocket, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 70
   at System.Net.Sockets.Socket.AcceptAsync() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 32
   at System.Net.Sockets.Socket.BeginAccept(AsyncCallback callback, Object state) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 2591
   at System.Net.Sockets.Tests.SocketTestServerAPM.OnAccept(IAsyncResult result) in /_/src/libraries/Common/tests/System/Net/Sockets/SocketTestServerAPM.cs:line 82
   at System.Threading.Tasks.TaskToAsyncResult.TaskAsyncResult.<.ctor>b__2_0() in /_/src/libraries/Common/src/System/Threading/Tasks/TaskToAsyncResult.cs:line 154
   at System.Threading.Tasks.AwaitTaskContinuation.<>c.<.cctor>b__17_0(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 670
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 179
   at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 702
   at System.Threading.Tasks.AwaitTaskContinuation.Run(Task task, Boolean canInlineContinuationTask) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 572
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3468
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:line 398
   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ValueTask.cs:line 665
   at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.SignalCompletion() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Sources/ManualResetValueTaskSourceCore.cs:line 235
   at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.SetResult(TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Sources/ManualResetValueTaskSourceCore.cs:line 68
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1031
   at System.Net.Sockets.SocketAsyncEventArgs.OnCompletedInternal() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs:line 210
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32 bytesTransferred, SocketFlags flags) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs:line 981
   at System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32 bytesTransferred, SocketFlags flags, SocketError socketError) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs:line 382
   at System.Net.Sockets.SocketAsyncEventArgs.AcceptCompletionCallback(IntPtr acceptedFileDescriptor, Memory`1 socketAddress, SocketError socketError) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs:line 30
   at System.Net.Sockets.SocketAsyncContext.AcceptOperation.InvokeCallback(Boolean allowPooling) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 647
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.ProcessAsyncOperation(TOperation op) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 1004
   at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation op) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 1413
   at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 340
   at System.Net.Sockets.SocketAsyncContext.AsyncOperation.Process() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 291
   at System.Net.Sockets.SocketAsyncContext.HandleEvents(SocketEvents events) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 2197
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs:line 242
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 913
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs:line 58
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 105
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskToAsyncResult.End[TResult](IAsyncResult asyncResult) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskToAsyncResult.cs:line 61
   at System.Net.Sockets.Socket.EndAccept(IAsyncResult asyncResult) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 2593
   at System.Net.Sockets.Tests.SocketTestServerAPM.OnAccept(IAsyncResult result) in /_/src/libraries/Common/tests/System/Net/Sockets/SocketTestServerAPM.cs:line 52
   at System.Threading.Tasks.TaskToAsyncResult.TaskAsyncResult..ctor(Task task, Object state, AsyncCallback callback) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskToAsyncResult.cs:line 144
   at System.Threading.Tasks.TaskToAsyncResult.Begin(Task task, AsyncCallback callback, Object state) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskToAsyncResult.cs:line 43
   at System.Net.Sockets.Socket.BeginAccept(AsyncCallback callback, Object state) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 2591
   at System.Net.Sockets.Tests.SocketTestServerAPM.OnAccept(IAsyncResult result) in /_/src/libraries/Common/tests/System/Net/Sockets/SocketTestServerAPM.cs:line 82
   at System.Threading.Tasks.TaskToAsyncResult.TaskAsyncResult.<.ctor>b__2_0() in /_/src/libraries/Common/src/System/Threading/Tasks/TaskToAsyncResult.cs:line 154
   at System.Threading.Tasks.AwaitTaskContinuation.<>c.<.cctor>b__17_0(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 670
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 179
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 203
   at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 702
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 1929
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 1266
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 913
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs:line 58
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 105
Fatal error. Internal CLR error. (0x80131506)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.Task+<>c.<ThrowAsync>b__128_1(System.Object)
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()
./RunTests.sh: line 168:    21 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Sockets.Tests.runtimeconfig.json --depsfile

@danmoseley
Copy link
Member

Regardless, if dotnet dump crashes it would be good to have a big open on that.

@karelz karelz modified the milestones: Future, 9.0.0 Aug 15, 2023
@karelz karelz added the test-run-core Test failures in .NET Core test runs label Aug 15, 2023
@wfurt
Copy link
Member

wfurt commented Nov 6, 2023

do you remember @rzikm @antonfirsov if the original issue was x64 or arm?

@rzikm
Copy link
Member

rzikm commented Nov 6, 2023

I believe it was ARM

@liveans
Copy link
Member

liveans commented Jul 3, 2024

Do we still have Alpine 3.15 in our pipeline?

/cc @wfurt @rzikm

@rzikm
Copy link
Member

rzikm commented Jul 3, 2024 •
edited
Loading

Do we still have Alpine 3.15 in our pipeline?

Not according to the yml files.

❯ rg Alpine C:\source\dotnet\runtime\eng\pipelines\
C:\source\dotnet\runtime\eng\pipelines\global-build.yml                                                                                                                                                                                               
104:      # Build native assets on Alpine. This exercises more modern musl libc changes that have a tendendy to break source-build.

C:\source\dotnet\runtime\eng\pipelines\libraries\helix-queues-setup.yml
44:        - (Alpine.316.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.16-helix-amd64
46:        - (Alpine.318.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.18-helix-amd64
50:      - (Alpine.318.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.18-helix-arm64v8
51:      - (Alpine.316.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.16-helix-arm64v8

C:\source\dotnet\runtime\eng\pipelines\coreclr\templates\helix-queues-setup.yml
80:        - (Alpine.317.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.17-helix-amd64
82:        - (Alpine.317.Amd64)[email protected]/dotnet-buildtools/prereqs:alpine-3.17-helix-amd64
87:        - (Alpine.316.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7
89:        - (Alpine.316.Arm32)[email protected]/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7
94:        - (Alpine.317.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.17-helix-arm64v8
96:        - (Alpine.317.Arm64)[email protected]/dotnet-buildtools/prereqs:alpine-3.17-helix-arm64v8

@liveans
Copy link
Member

liveans commented Jul 3, 2024

Do we still have Alpine 3.15 in our pipeline?

Not according to the yml files.

❯ rg Alpine C:\source\dotnet\runtime\eng\pipelines\
C:\source\dotnet\runtime\eng\pipelines\global-build.yml                                                                                                                                                                                               
104:      # Build native assets on Alpine. This exercises more modern musl libc changes that have a tendendy to break source-build.

C:\source\dotnet\runtime\eng\pipelines\libraries\helix-queues-setup.yml
44:        - (Alpine.316.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.16-helix-amd64
46:        - (Alpine.318.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.18-helix-amd64
50:      - (Alpine.318.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.18-helix-arm64v8
51:      - (Alpine.316.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.16-helix-arm64v8

C:\source\dotnet\runtime\eng\pipelines\coreclr\templates\helix-queues-setup.yml
80:        - (Alpine.317.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.17-helix-amd64
82:        - (Alpine.317.Amd64)[email protected]/dotnet-buildtools/prereqs:alpine-3.17-helix-amd64
87:        - (Alpine.316.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7
89:        - (Alpine.316.Arm32)[email protected]/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7
94:        - (Alpine.317.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.17-helix-arm64v8
96:        - (Alpine.317.Arm64)[email protected]/dotnet-buildtools/prereqs:alpine-3.17-helix-arm64v8

Ok, I'll check if this is still happening in other platforms. If it's not, I'll close this issue.
Thanks @rzikm!

@wfurt
Copy link
Member

wfurt commented Jul 3, 2024

I would be surprised if this is specific to 3.15. More interesting question IMHO is if any of the ARM(32) images would be able to reproduce it. This looks like rare race condition and product bug

@liveans
Copy link
Member

liveans commented Jul 11, 2024 •
edited
Loading

I can't repro this locally either with x64/ARM32/ARM64
I saw only 1 crash in main within last 60 days and it's not Alpine/Linux, it was Windows.11.amd64 (I can't see logs from that, but not sure if that is the same issue)
Perhaps, we should close this, and reopen again if we see same pattern/issue again?

@liveans
Copy link
Member

liveans commented Jul 11, 2024

Triage: 1 crash on different platform (Windows.Amd64) on 6/3. No crashes for this platform and arch in last 60 days, and can't repro this locally (on x64, ARM32/64 with both Windows/Linux). Closing this for now, and we can re-open again if we see any crash.

Feel free to re-open if you don't agree with the decision.

@liveans liveans closed this as completed Jul 11, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Aug 11, 2024
@antonfirsov antonfirsov reopened this Apr 23, 2025
@antonfirsov antonfirsov modified the milestones: 9.0.0, 10.0.0 Apr 23, 2025
@rzikm
Copy link
Member

rzikm commented Apr 23, 2025

We have started seeing this again occasionally

----- start Tue Apr 22 12:33:32 UTC 2025 =============== To repro directly: =====================================================
pushd .
/root/helix/work/correlation/dotnet exec --runtimeconfig System.Net.Sockets.Tests.runtimeconfig.json --depsfile System.Net.Sockets.Tests.deps.json xunit.console.dll System.Net.Sockets.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing 
popd
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Net.Sockets.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Sockets.Tests (found 217 of 1875 test cases)
  Starting:    System.Net.Sockets.Tests (parallel test collections = on [2 threads], stop on fail = off)
Unhandled exception. System.Net.Sockets.SocketException (22): Invalid argument
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1395
   at System.Net.Sockets.Socket.AcceptAsync(Socket acceptSocket, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 70
   at System.Net.Sockets.Socket.AcceptAsync() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 32
   at System.Net.Sockets.Socket.BeginAccept(AsyncCallback callback, Object state) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 2743
   at System.Net.Sockets.Tests.SocketTestServerAPM.OnAccept(IAsyncResult result) in /_/src/libraries/Common/tests/System/Net/Sockets/SocketTestServerAPM.cs:line 82
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 179
   at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 702
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3485
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:line 387
   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ValueTask.cs:line 643
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs:line 280
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 1154
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 127
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 104
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskToAsyncResult.End[TResult](IAsyncResult asyncResult) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskToAsyncResult.cs:line 61
   at System.Net.Sockets.Socket.EndAccept(IAsyncResult asyncResult) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 2745
   at System.Net.Sockets.Tests.SocketTestServerAPM.OnAccept(IAsyncResult result) in /_/src/libraries/Common/tests/System/Net/Sockets/SocketTestServerAPM.cs:line 52
   at System.Threading.Tasks.TaskToAsyncResult.TaskAsyncResult..ctor(Task task, Object state, AsyncCallback callback) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskToAsyncResult.cs:line 131
   at System.Threading.Tasks.TaskToAsyncResult.Begin(Task task, AsyncCallback callback, Object state) in /_/src/libraries/Common/src/System/Threading/Tasks/TaskToAsyncResult.cs:line 42
   at System.Net.Sockets.Socket.BeginAccept(AsyncCallback callback, Object state) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 2743
   at System.Net.Sockets.Tests.SocketTestServerAPM.OnAccept(IAsyncResult result) in /_/src/libraries/Common/tests/System/Net/Sockets/SocketTestServerAPM.cs:line 82
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 179
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 203
   at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 702
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 1927
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 1154
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 127
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 104
[createdump] Gathering state for process 34 dotnet
[createdump] Crashing thread 0173 signal 6 (0006)
[createdump] Writing crash report to file /home/helixbot/dotnetbuild/dumps/coredump.34.dmp.crashreport.json
[createdump] Crash report successfully written
[createdump] Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.34.dmp
[createdump] Written 485093376 bytes (118431 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 1038ms
./RunTests.sh: line 178:    34 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Sockets.Tests.runtimeconfig.json --depsfile System.Net.Sockets.Tests.deps.json xunit.console.dll System.Net.Sockets.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Tue Apr 22 12:34:00 UTC 2025 ----- exit code 134 ----------------------------------------------------------

This was on mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-helix-amd64

it seems, to me that there is a race between SocketTestServerAPM.Dispose and (recursive) SocketTestServerAPM.OnAccept that runs on the background. On some platform, it leads to SocketError.InvalidArgument which is rethrown from the OnAccept method.

the _disposed guard does not work because there is race between setting it and rethrowing

runtime/src/libraries/Common/tests/System/Net/Sockets/SocketTestServerAPM.cs

Lines 32 to 39 in 8e5da14

protected override void Dispose(bool disposing)
{
if (disposing)
{
_socket.Dispose();
_disposed = true;
}
}

runtime/src/libraries/Common/tests/System/Net/Sockets/SocketTestServerAPM.cs

Lines 84 to 94 in 8e5da14

catch (SocketException e)
{
if (_disposed ||
e.SocketErrorCode == SocketError.OperationAborted ||
e.SocketErrorCode == SocketError.Interrupted)
{
return;
}
throw;
}

@rzikm rzikm self-assigned this Apr 23, 2025
@dotnet-policy-service dotnet-policy-service bot added the in-pr There is an active PR which will close this issue when it is merged label Apr 23, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@rzikm rzikm

Labels
area-System.Net.Sockets in-pr There is an active PR which will close this issue when it is merged test-run-core Test failures in .NET Core test runs
Projects
None yet
Milestone
10.0.0
Development

Successfully merging a pull request may close this issue.

Do not rethrow from SocketsTestServerAPM.OnAccept rzikm/dotnet-runtime
6 participants
@karelz @danmoseley @antonfirsov @wfurt @rzikm @liveans