Possible GC hang while building msbuild with mono/2018-08 and mono/master #10863
Description
Mono version: Mono JIT compiler version 5.18.0.103 (2018-08/56bed71f118 Tue Sep 18 07:06:42 EDT 2018)
Steps:
- This seemed to help with reproducing the issue:
nuget locals all -Clear, but not always. - Build msbuild from
xplat-masterbranch (HEAD: 17218b611a97315eaef9fa4a8add5ef049a2ff8e) with
$ make - This gets stuck at the bootstrapped build while restoring packages. This is not reproducible every time though.
I attached to the process with lldb and ran monobt all and got https://gist.github.com/radical/b8130fcb125714737d5515f53b4c5031 .
IIUC, some of the threads are waiting *Threading.Wait*, and some seem to be waiting for the gc to resume them. Thread #19 seems to be
the one which causes the gc to kick in:
thread #19
frame #0: 0x00007fff7e735246 libsystem_kernel.dylib`semaphore_wait_trap + 10
frame #1: 0x0000000108f1c678 mono`mono_os_sem_timedwait + 344
frame #2: 0x0000000108f1bec5 mono`mono_threads_wait_pending_operations + 101
frame #3: 0x0000000108ebff93 mono`sgen_client_stop_world + 563
frame #4: 0x0000000108ed32d6 mono`sgen_stop_world + 70
frame #5: 0x0000000108ed005d mono`sgen_perform_collection + 157
frame #6: 0x0000000108ec5a40 mono`sgen_alloc_obj_nolock + 928
frame #7: 0x0000000108ec2697 mono`mono_gc_alloc_vector + 119
frame #8: 0x109176fd9 (wrapper managed-to-native) object:__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr) [{0x7fcbfd002958} + 0x89] (0x109176f50 0x109177055) [0x7fcbfbd0dd60 - MSBuild.dll]
frame #9: 0x000000010b14b6ad mscorlib.dll.dylib`System_Collections_Generic_Dictionary_2_TKey_REF_TValue_REF_Initialize_int + 157
frame #10: 0x000000010b1496f9 mscorlib.dll.dylib`System_Collections_Generic_Dictionary_2_TKey_REF_TValue_REF__ctor_int_System_Collections_Generic_IEqualityComparer_1_TKey_REF + 73
frame #11: 0x10e7e7bc3 Microsoft.Build.Internal.CommunicationsUtilities:GetEnvironmentVariables () [{0x7fcbfc8f59a0} + 0xb3] (0x10e7e7b10 0x10e7e8192) [0x7fcbfbd0dd60 - MSBuild.dll]
frame #12: 0x10ed05473 Microsoft.Build.BackEnd.RequestBuilder:SetEnvironmentVariableBlock (System.Collections.Generic.IDictionary`2<string, string>) [{0x7fcbfd162870} + 0x43] (0x10ed05430 0x10ed054b9) [0x7fcbfbd0dd60 - MSBuild.dll]
frame #13: 0x000000010affc651 mscorlib.dll.dylib`System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_InvokeMoveNext_object + 177
frame #14: 0x000000010aeff4eb mscorlib.dll.dylib`System_Threading_ExecutionContext_RunInternal_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 427
frame #15: 0x000000010aeff333 mscorlib.dll.dylib`System_Threading_ExecutionContext_Run_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 67
frame #16: 0x000000010affc4fa mscorlib.dll.dylib`System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_Run + 186
frame #17: 0x000000010af3d1b9 mscorlib.dll.dylib`System_Threading_Tasks_TaskSchedulerAwaitTaskContinuation__c__Runb__2_0_object + 89
frame #18: 0x000000010af3346c mscorlib.dll.dylib`System_Threading_Tasks_Task_InnerInvoke + 156
frame #19: 0x000000010af33188 mscorlib.dll.dylib`System_Threading_Tasks_Task_Execute + 56
frame #20: 0x000000010af333bc mscorlib.dll.dylib`System_Threading_Tasks_Task_ExecutionContextCallback_object + 92
frame #21: 0x000000010aeff4eb mscorlib.dll.dylib`System_Threading_ExecutionContext_RunInternal_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 427
frame #22: 0x000000010aeff333 mscorlib.dll.dylib`System_Threading_ExecutionContext_Run_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 67
frame #23: 0x000000010af39a37 mscorlib.dll.dylib`System_Threading_Tasks_Task_ExecuteWithThreadLocal_System_Threading_Tasks_Task_ + 247
frame #24: 0x000000010af332f9 mscorlib.dll.dylib`System_Threading_Tasks_Task_ExecuteEntry_bool + 233
frame #25: 0x000000010af42784 mscorlib.dll.dylib`System_Threading_Tasks_TaskScheduler_TryExecuteTask_System_Threading_Tasks_Task + 68
frame #26: 0x10ed16d43 Microsoft.Build.BackEnd.RequestBuilder/DedicatedThreadsTaskScheduler:<InjectThread>b__6_0 () [{0x7fcbfd162d98} + 0xc3] (0x10ed16c80 0x10ed16e23) [0x7fcbfbd0dd60 - MSBuild.dll]
frame #27: 0x000000010af01f2b mscorlib.dll.dylib`System_Threading_ThreadHelper_ThreadStart_Context_object + 171
frame #28: 0x000000010aeff4eb mscorlib.dll.dylib`System_Threading_ExecutionContext_RunInternal_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 427
frame #29: 0x000000010aeff333 mscorlib.dll.dylib`System_Threading_ExecutionContext_Run_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 67
frame #30: 0x000000010aeff2a8 mscorlib.dll.dylib`System_Threading_ExecutionContext_Run_System_Threading_ExecutionContext_System_Threading_ContextCallback_object + 104
frame #31: 0x000000010af020b3 mscorlib.dll.dylib`System_Threading_ThreadHelper_ThreadStart + 67
frame #32: 0x1093701a9 (wrapper runtime-invoke) object:runtime_invoke_void__this__ (object,intptr,intptr,intptr) [{0x7fcbfd049590} + 0xa9] (0x109370100 0x109370278) [0x7fcbfbd0dd60 - MSBuild.dll]
frame #33: 0x0000000108c3c203 mono`mono_jit_runtime_invoke + 1619
frame #34: 0x0000000108e39254 mono`do_runtime_invoke + 84
frame #35: 0x0000000108e3b01b mono`mono_runtime_delegate_try_invoke + 203
frame #36: 0x0000000108e5ee6e mono`start_wrapper + 750
frame #37: 0x00007fff7e906661 libsystem_pthread.dylib`_pthread_body + 340
frame #38: 0x00007fff7e90650d libsystem_pthread.dylib`_pthread_start + 377
frame #39: 0x00007fff7e905bf9 libsystem_pthread.dylib`thread_start + 13
The PR #10339 for building msbuild xplat-master with mono/master is failing on the CI. See the logs for OS X Extended build (https://jenkins.mono-project.com/job/test-mono-pull-request-mac-sdk/105/).