Thanks to visit codestin.com
Credit goes to github.com

Skip to content

corlib LLVM tests hang/crash a lot #9198

New issue
New issue
Open
Open
corlib LLVM tests hang/crash a lot#9198
Labels
area-Runtime: JIT
@akoeplinger

Description

@akoeplinger
akoeplinger
opened on Jun 18, 2018

Steps to Reproduce

  1. Compile Mono with LLVM enabled
  2. Run corlib tests with LLVM
  3. It hangs/crashes a lot

Current Behavior

Either it hangs (https://jenkins.mono-project.com/view/Mono/job/test-mono-mainline-mac-sdk/label=osx-amd64/1661/):

CSC     [net_4_x-macos] net_4_x_corlib_test.dll
Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.Monitor.Monitor_wait (object,int) [0x00000] in <2cf22219e1ee42768c7b5b87d63b9615#1C769E2D-FFEF-86EB-7769-B27320531547>:0
  at System.Threading.Monitor.ObjWait (bool,int,object) <0x00055>
  at System.Threading.Monitor.Wait (object,int,bool) <0x0001d>
  at System.Threading.Monitor.Wait (object,int) <0x0000e>
  at System.Threading.ManualResetEventSlim.Wait (int,System.Threading.CancellationToken) [0x00141] in <2cf22219e1ee42768c7b5b87d63b9615#1C769E2D-FFEF-86EB-7769-B27320531547>:0
  at System.Threading.Tasks.Task.SpinThenBlockingWait (int,System.Threading.CancellationToken) <0x00120>
  at System.Threading.Tasks.Task.InternalWait (int,System.Threading.CancellationToken) <0x001b5>
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task) <0x0002b>
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task) <0x00024>
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () <0x0000f>
  at Microsoft.CodeAnalysis.CSharp.MethodCompiler.WaitForWorkers () <0x000a1>
  at Microsoft.CodeAnalysis.CSharp.MethodCompiler.CompileMethodBodies (Microsoft.CodeAnalysis.CSharp.CSharpCompilation,Microsoft.CodeAnalysis.CSharp.Emit.PEModuleBuilder,bool,bool,bool,Microsoft.CodeAnalysis.DiagnosticBag,System.Predicate`1<Microsoft.CodeAnalysis.CSharp.Symbol>,System.Threading.CancellationToken) <0x0015f>
  at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.CompileMethods (Microsoft.CodeAnalysis.Emit.CommonPEModuleBuilder,bool,bool,bool,Microsoft.CodeAnalysis.DiagnosticBag,System.Predicate`1<Microsoft.CodeAnalysis.ISymbol>,System.Threading.CancellationToken) <0x00254>
  at Microsoft.CodeAnalysis.CommonCompiler.CompileAndEmit (Microsoft.CodeAnalysis.TouchedFileLogger,Microsoft.CodeAnalysis.Compilation&,System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.Diagnostics.DiagnosticAnalyzer>,System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.AdditionalTextFile>,System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.EmbeddedText>,Microsoft.CodeAnalysis.DiagnosticBag,System.Threading.CancellationToken,System.Threading.CancellationTokenSource&,bool&,Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver&) [0x00216] in <b28b918e025b48209f28967b441780d7#1C769E2D-FFEF-86EB-7769-B27320531547>:0
  at Microsoft.CodeAnalysis.CommonCompiler.RunCore (System.IO.TextWriter,Microsoft.CodeAnalysis.ErrorLogger,System.Threading.CancellationToken) <0x0031d>
  at Microsoft.CodeAnalysis.CommonCompiler.Run (System.IO.TextWriter,System.Threading.CancellationToken) [0x00035] in <b28b918e025b48209f28967b441780d7#1C769E2D-FFEF-86EB-7769-B27320531547>:0
  at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc/<>c__DisplayClass1_0.<Run>b__0 (System.IO.TextWriter) <0x0000c>
  at Microsoft.CodeAnalysis.CommandLine.ConsoleUtil.RunWithUtf8Output<int> (bool,System.IO.TextWriter,System.Func`2<System.IO.TextWriter, int>) <0x0005a>
  at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc.Run (string[],Microsoft.CodeAnalysis.BuildPaths,System.IO.TextWriter,Microsoft.CodeAnalysis.IAnalyzerAssemblyLoader) <0x001f1>
  at (wrapper delegate-invoke) <Module>.invoke_int_string[]_BuildPaths_TextWriter_IAnalyzerAssemblyLoader (string[],Microsoft.CodeAnalysis.BuildPaths,System.IO.TextWriter,Microsoft.CodeAnalysis.IAnalyzerAssemblyLoader) <0x00111>
  at Microsoft.CodeAnalysis.CommandLine.DesktopBuildClient.RunLocalCompilation (string[],Microsoft.CodeAnalysis.BuildPaths,System.IO.TextWriter) <0x00028>
  at Microsoft.CodeAnalysis.CommandLine.BuildClient.RunCompilation (System.Collections.Generic.IEnumerable`1<string>,Microsoft.CodeAnalysis.BuildPaths,System.IO.TextWriter) <0x0024a>
  at Microsoft.CodeAnalysis.CommandLine.DesktopBuildClient.Run (System.Collections.Generic.IEnumerable`1<string>,Microsoft.CodeAnalysis.CommandLine.RequestLanguage,Microsoft.CodeAnalysis.CommandLine.CompileFunc,Microsoft.CodeAnalysis.IAnalyzerAssemblyLoader) <0x001de>
  at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.MainCore (string[]) <0x00099>
  at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.Main (string[]) <0x0000c>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_int_object (object,intptr,intptr,intptr) <0x000bf>
Memory around native instruction pointer (0x7fffa8fd4bf2):
0x7fffa8fd4be2  ff ff c3 90 90 90 b8 31 01 00 02 49 89 ca 0f 05  .......1...I....
0x7fffa8fd4bf2  73 08 48 89 c7 e9 b3 90 ff ff c3 90 90 90 b8 2e  s.H.............
0x7fffa8fd4c02  01 00 02 49 89 ca 0f 05 73 08 48 89 c7 e9 9b 90  ...I....s.H.....
0x7fffa8fd4c12  ff ff c3 90 90 90 b8 2d 01 00 02 49 89 ca 0f 05  .......-...I....

Native stacktrace:

	0   mono                                0x000000010ab62145 mono_handle_native_crash + 885
	1   libsystem_platform.dylib            0x00007fffa90b5b3a _sigtramp + 26
	2   ???                                 0x00007ff500000007 0x0 + 140690243715079
	3   mono                                0x000000010ad880c0 mono_os_cond_timedwait + 112
	4   mono                                0x000000010acf4609 mono_w32handle_timedwait_signal_handle + 201
	5   mono                                0x000000010acf44d1 mono_w32handle_wait_one + 769
	6   mono                                0x000000010ad2a64e ves_icall_System_Threading_Monitor_Monitor_wait + 430
	7   ???                                 0x0000000118de61ce 0x0 + 4712194510
* Assertion at mini-exceptions.c:1343, condition `ud->num_frames + 1 < ud->max_frames' not met

/Users/builder/jenkins/workspace/test-mono-mainline-mac-sdk/label/osx-amd64/scripts/ci/babysitter: Command `make` timed out
/Users/builder/jenkins/workspace/test-mono-mainline-mac-sdk/label/osx-amd64/scripts/ci/babysitter: Saw timeout in test case None (never allowed). Will halt testing.
*** end(3601): corlib-tests-llvm: \e[43mUnstable\e[0m

Or it crashes (https://jenkins.mono-project.com/view/Mono/job/test-mono-mainline-mac-sdk/label=osx-amd64/1659):

CSC     [net_4_x-macos] net_4_x_corlib_test.resources.dll
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
* Assertion at tramp-amd64.c:146, condition `(guint64)(orig_code - 11) % 8 == 0' not met

Stacktrace:

  at <unknown> <0xffffffff>
  at Microsoft.Cci.MetadataWriter.GetMethodSignatureHandle (Microsoft.Cci.IMethodReference) <0x00018>
  at Microsoft.Cci.MemberRefComparer.GetHashCode (Microsoft.Cci.ITypeMemberReference) <0x0016c>
  at System.Collections.Generic.Dictionary`2<Microsoft.Cci.ITypeMemberReference, int>.TryInsert (Microsoft.Cci.ITypeMemberReference,int,System.Collections.Generic.InsertionBehavior) <0x0008c>
  at System.Collections.Generic.Dictionary`2<Microsoft.Cci.ITypeMemberReference, int>.Add (Microsoft.Cci.ITypeMemberReference,int) <0x00011>
  at Microsoft.Cci.MetadataWriter/InstanceAndStructuralReferenceIndex`1<T_REF>.AddItem (T_REF,int) <0x00057>
  at Microsoft.Cci.MetadataWriter/HeapOrReferenceIndexBase`1<T_REF>.Add (T_REF) <0x00066>
  at Microsoft.Cci.MetadataWriter/HeapOrReferenceIndexBase`1<T_REF>.GetOrAdd (T_REF) <0x00047>
  at Microsoft.Cci.FullMetadataWriter.GetOrAddMemberReferenceHandle (Microsoft.Cci.ITypeMemberReference) <0x00016>
  at Microsoft.Cci.MetadataWriter.GetMemberReferenceHandle (Microsoft.Cci.ITypeMemberReference) <0x00009>
  at Microsoft.Cci.ReferenceIndexer.RecordTypeMemberReference (Microsoft.Cci.ITypeMemberReference) <0x00013>
  at Microsoft.Cci.ReferenceIndexerBase.Visit (Microsoft.Cci.ITypeMemberReference) <0x00015>
  at Microsoft.Cci.ReferenceIndexerBase.Visit (Microsoft.Cci.IMethodReference) <0x00124>
  at Microsoft.Cci.MetadataVisitor.Visit (Microsoft.Cci.ICustomAttribute) <0x000ee>
  at Microsoft.Cci.MetadataVisitor.Visit (System.Collections.Generic.IEnumerable`1<Microsoft.Cci.ICustomAttribute>) <0x00081>
  at Microsoft.Cci.ReferenceIndexer.Visit (Microsoft.CodeAnalysis.Emit.CommonPEModuleBuilder) <0x000a1>
  at Microsoft.Cci.MetadataWriter.CreateIndices () <0x000a4>
  at Microsoft.Cci.MetadataWriter.BuildMetadataAndIL (Microsoft.Cci.PdbWriter,System.Reflection.Metadata.BlobBuilder,System.Reflection.Metadata.BlobBuilder,System.Reflection.Metadata.BlobBuilder,System.Reflection.Metadata.Blob&,System.Reflection.Metadata.Blob&) <0x00036>
  at Microsoft.Cci.PeWriter.WritePeToStream (Microsoft.CodeAnalysis.Emit.EmitContext,Microsoft.CodeAnalysis.CommonMessageProvider,System.Func`1<System.IO.Stream>,System.Func`1<System.IO.Stream>,Microsoft.Cci.PdbWriter,string,bool,bool,bool,System.Nullable`1<System.Security.Cryptography.RSAParameters>,System.Threading.CancellationToken) [0x00066] in <b28b918e025b48209f28967b441780d7#FFB5A974-1C13-DB4A-498C-37DEE53EBD90>:0
  at Microsoft.CodeAnalysis.Compilation.SerializePeToStream (Microsoft.CodeAnalysis.Emit.CommonPEModuleBuilder,Microsoft.CodeAnalysis.DiagnosticBag,Microsoft.CodeAnalysis.CommonMessageProvider,System.Func`1<System.IO.Stream>,System.Func`1<System.IO.Stream>,System.Func`1<System.IO.Stream>,Microsoft.Cci.PdbWriter,string,bool,bool,bool,bool,System.Nullable`1<System.Security.Cryptography.RSAParameters>,System.Threading.CancellationToken) <0x00195>
  at Microsoft.CodeAnalysis.Compilation.SerializeToPeStream (Microsoft.CodeAnalysis.Emit.CommonPEModuleBuilder,Microsoft.CodeAnalysis.Compilation/EmitStreamProvider,Microsoft.CodeAnalysis.Compilation/EmitStreamProvider,Microsoft.CodeAnalysis.Compilation/EmitStreamProvider,System.Func`2<Microsoft.DiaSymReader.ISymWriterMetadataProvider, Microsoft.DiaSymReader.SymUnmanagedWriter>,Microsoft.CodeAnalysis.DiagnosticBag,bool,bool,bool,string,System.Nullable`1<System.Security.Cryptography.RSAParameters>,System.Threading.CancellationToken) [0x00133] in <b28b918e025b48209f28967b441780d7#FFB5A974-1C13-DB4A-498C-37DEE53EBD90>:0
  at Microsoft.CodeAnalysis.CommonCompiler.CompileAndEmit (Microsoft.CodeAnalysis.TouchedFileLogger,Microsoft.CodeAnalysis.Compilation&,System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.Diagnostics.DiagnosticAnalyzer>,System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.AdditionalTextFile>,System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.EmbeddedText>,Microsoft.CodeAnalysis.DiagnosticBag,System.Threading.CancellationToken,System.Threading.CancellationTokenSource&,bool&,Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver&) [0x003fb] in <b28b918e025b48209f28967b441780d7#FFB5A974-1C13-DB4A-498C-37DEE53EBD90>:0
  at Microsoft.CodeAnalysis.CommonCompiler.RunCore (System.IO.TextWriter,Microsoft.CodeAnalysis.ErrorLogger,System.Threading.CancellationToken) <0x0031d>
  at Microsoft.CodeAnalysis.CommonCompiler.Run (System.IO.TextWriter,System.Threading.CancellationToken) [0x00035] in <b28b918e025b48209f28967b441780d7#FFB5A974-1C13-DB4A-498C-37DEE53EBD90>:0
  at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc/<>c__DisplayClass1_0.<Run>b__0 (System.IO.TextWriter) <0x0000c>
  at Microsoft.CodeAnalysis.CommandLine.ConsoleUtil.RunWithUtf8Output<int> (bool,System.IO.TextWriter,System.Func`2<System.IO.TextWriter, int>) <0x0005a>
  at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc.Run (string[],Microsoft.CodeAnalysis.BuildPaths,System.IO.TextWriter,Microsoft.CodeAnalysis.IAnalyzerAssemblyLoader) <0x001f1>
  at (wrapper delegate-invoke) <Module>.invoke_int_string[]_BuildPaths_TextWriter_IAnalyzerAssemblyLoader (string[],Microsoft.CodeAnalysis.BuildPaths,System.IO.TextWriter,Microsoft.CodeAnalysis.IAnalyzerAssemblyLoader) <0x00111>
  at Microsoft.CodeAnalysis.CommandLine.DesktopBuildClient.RunLocalCompilation (string[],Microsoft.CodeAnalysis.BuildPaths,System.IO.TextWriter) <0x00028>
  at Microsoft.CodeAnalysis.CommandLine.BuildClient.RunCompilation (System.Collections.Generic.IEnumerable`1<string>,Microsoft.CodeAnalysis.BuildPaths,System.IO.TextWriter) <0x0024a>
  at Microsoft.CodeAnalysis.CommandLine.DesktopBuildClient.Run (System.Collections.Generic.IEnumerable`1<string>,Microsoft.CodeAnalysis.CommandLine.RequestLanguage,Microsoft.CodeAnalysis.CommandLine.CompileFunc,Microsoft.CodeAnalysis.IAnalyzerAssemblyLoader) <0x001de>
  at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.MainCore (string[]) <0x00096>
  at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.Main (string[]) <0x0000c>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_int_object (object,intptr,intptr,intptr) <0x000bf>
Memory around native instruction pointer (0x7fffa8fd4d42):
0x7fffa8fd4d32  ff ff c3 90 90 90 b8 48 01 00 02 49 89 ca 0f 05  .......H...I....
0x7fffa8fd4d42  73 08 48 89 c7 e9 63 8f ff ff c3 90 90 90 b8 4c  s.H...c........L
0x7fffa8fd4d52  01 00 02 49 89 ca 0f 05 73 08 48 89 c7 e9 4b 8f  ...I....s.H...K.
0x7fffa8fd4d62  ff ff c3 90 90 90 b8 49 01 00 02 49 89 ca 0f 05  .......I...I....

Native stacktrace:

	0   mono                                0x000000010364f165 mono_handle_native_crash + 885
	1   libsystem_platform.dylib            0x00007fffa90b5b3a _sigtramp + 26
	2   ???                                 0x0000000000000003 0x0 + 3
	3   libsystem_c.dylib                   0x00007fffa8f3a420 abort + 129
	4   mono                                0x00000001038710bf mono_log_write_logfile + 351
	5   mono                                0x0000000103889c82 monoeg_assertion_message + 194
	6   mono                                0x00000001036a4ead mono_arch_patch_callsite + 717
	7   mono                                0x0000000103652230 common_call_trampoline + 2064
	8   mono                                0x00000001036519b9 mono_magic_trampoline + 105
	9   ???                                 0x0000000103b0a393 0x0 + 4356875155
	10  ???                                 0x000000010dbb4999 0x0 + 4525345177
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace
Waiting for signalled thread to collect stacktrace

Debug info from gdb:

(lldb) command source -s 0 '/tmp/mono-gdb-commands.lAdyYx'
Executing commands in '/tmp/mono-gdb-commands.lAdyYx'.
(lldb) process attach --pid 18955
Waiting for signalled thread to collect stacktrace
error: attach failed: unable to attach

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

/bin/sh: line 1: 18955 Abort trap: 6           MONO_PATH="./../../class/lib/build:$MONO_PATH" CSC_SDK_PATH_DISABLED= /Users/builder/jenkins/workspace/test-mono-mainline-mac-sdk/label/osx-amd64/external/bockbuild/builds/mono/runtime/mono-wrapper --aot-path=/Users/builder/jenkins/workspace/test-mono-mainline-mac-sdk/label/osx-amd64/external/bockbuild/builds/mono/mcs/class/lib/build --gc-params=nursery-size=64m /Users/builder/jenkins/workspace/test-mono-mainline-mac-sdk/label/osx-amd64/external/bockbuild/builds/mono/external/roslyn-binaries/Microsoft.Net.Compilers/Microsoft.Net.Compilers.2.8.2/tools/csc.exe /codepage:65001 /nologo /noconfig /deterministic -unsafe -nostdlib -nowarn:612,618,3001,3002,3003,1635 -d:INSIDE_CORLIB,MONO_CULTURE_DATA -d:LIBC -d:FEATURE_PAL,GENERICS_WORK,FEATURE_LIST_PREDICATES,FEATURE_SERIALIZATION,FEATURE_ENCODINGNLS,FEATURE_ASCII,FEATURE_LATIN1,FEATURE_UTF7,FEATURE_UTF32,MONO_HYBRID_ENCODING_SUPPORT,FEATURE_ASYNC_IO,NEW_EXPERIMENTAL_ASYNC_IO,FEATURE_UTF32,FEATURE_EXCEPTIONDISPATCHINFO,FEATURE_CORRUPTING_EXCEPTIONS,FEATURE_EXCEPTION_NOTIFICATIONS,FEATURE_STRONGNAME_MIGRATION,FEATURE_USE_LCID,FEATURE_FUSION,FEATURE_CRYPTO,FEATURE_X509_SECURESTRINGS,FEATURE_SYNCHRONIZATIONCONTEXT,FEATURE_SYNCHRONIZATIONCONTEXT_WAIT,HAS_CORLIB_CONTRACTS -d:FEATURE_MACL -d:FEATURE_REMOTING,MONO_COM,FEATURE_COMINTEROP,FEATURE_ROLE_BASED_SECURITY -d:MONO_FEATURE_THREAD_ABORT -d:MONO_FEATURE_THREAD_SUSPEND_RESUME -d:MONO_FEATURE_MULTIPLE_APPDOMAINS -d:FEATURE_PORTABLE_SPAN -d:NET_4_0 -d:NET_4_5 -d:NET_4_6 -d:MONO -d:WIN_PLATFORM -nowarn:1699 -nostdlib /debug:portable -optimize /features:peverify-compat /langversion:latest -target:library -r:./../../class/lib/net_4_x/mscorlib.dll Test/resources/culture-es-ES.cs -resource:Test/resources/Resources.es-ES.resources -out:es-ES/net_4_x_corlib_test.resources.dll
make[1]: *** [es-ES/net_4_x_corlib_test.resources.dll] Error 134
make: *** [do-run-test] Error 1
/Users/builder/jenkins/workspace/test-mono-mainline-mac-sdk/label/osx-amd64/scripts/ci/babysitter: Test suite terminated with code 2, and suite cannot report test case data. Halting.
*** end(362): corlib-tests-llvm: \e[43mUnstable\e[0m

Expected Behavior

no hangs/crashes.

On which platforms did you notice this

[ x ] macOS
[ ] Linux
[ ] Windows

Version Used: master

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-Runtime: JIT

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions