NativeAOT'd il2cpp.exe random crashing on Windows x64. #121632
Description
Description
While looking for a different instability, someone internally found that we are occasionally seeing the nativeaot'd il2cpp executable crash on Windows x64. The crash seems rare. We don't have reproduction steps yet. We will spend some time trying to reproduce. If we get more information I'll update the issue.
The info we do have is a crash dump. Here is some info from the dump.
il2cpp.exe!RhpInterfaceDispatch4() Line 68
at D:\a\_work\1\s\src\coreclr\nativeaot\Runtime\amd64\StubDispatch.asm(68)
il2cpp.exe!Unity_IL2CPP_DataModel_Unity_IL2CPP_DataModel_CustomAttributeProviderExtensions__HasAttribute() Line 7
at C:\build\output\unity\il2cpp\Unity.IL2CPP.DataModel\CustomAttributeProviderExtensions.cs(7)
il2cpp.exe!Unity_IL2CPP_Unity_IL2CPP_AssemblyConversion_PrimaryCollection_Steps_Global_AnalyticsCollection__ProcessItem() Line 37
at C:\build\output\unity\il2cpp\Unity.IL2CPP\AssemblyConversion\PrimaryCollection\Steps.Global\AnalyticsCollection.cs(37)
il2cpp.exe!Unity_IL2CPP_Unity_IL2CPP_AssemblyConversion_Steps_Base_ScheduledItemsStepFunc_5<System___Canon__System___Canon__System___Canon__System___Canon__System___Canon>__WorkerWrapper() Line 45
at C:\build\output\unity\il2cpp\Unity.IL2CPP\AssemblyConversion\Steps.Base\ScheduledItemsStepFunc.cs(45)
il2cpp.exe!Unity_IL2CPP_Unity_IL2CPP_Contexts_Scheduling_PhaseWorkScheduler_1_ContinueWithResultsWorkItem_4<System___Canon__System___Canon__System___Canon__System___Canon__System___Canon>__InvokeWorker() Line 620
at C:\build\output\unity\il2cpp\Unity.IL2CPP\Contexts.Scheduling\PhaseWorkScheduler.cs(620)
il2cpp.exe!Unity_IL2CPP_Unity_IL2CPP_Contexts_Scheduling_PhaseWorkScheduler_1_BaseContinueWorkItem_2<System___Canon__System___Canon__System___Canon>__Invoke() Line 534
at C:\build\output\unity\il2cpp\Unity.IL2CPP\Contexts.Scheduling\PhaseWorkScheduler.cs(534)
il2cpp.exe!Unity_IL2CPP_Unity_IL2CPP_Contexts_Scheduling_PhaseWorkScheduler_1<System___Canon>__WorkerLoop() Line 338
at C:\build\output\unity\il2cpp\Unity.IL2CPP\Contexts.Scheduling\PhaseWorkScheduler.cs(338)
il2cpp.exe!S_P_CoreLib_System_Threading_Thread__StartThread() Line 447
at /_/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Threading/Thread.NativeAot.cs(447)
il2cpp.exe!S_P_CoreLib_System_Threading_Thread__ThreadEntryPoint() Line 225
at /_/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Threading/Thread.NativeAot.Windows.cs(225)
kernel32.dll!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()
The code it's crashing during is pretty unremarkable
public static class CustomAttributeProviderExtensions
{
public static bool HasAttribute(this ICustomAttributeProvider customAttributeProvider, string @namespace, string name)
{
for (int i = 0; i < customAttributeProvider.CustomAttributes.Count; i++)
{
if (customAttributeProvider.CustomAttributes[i].AttributeType.Name == name &&
customAttributeProvider.CustomAttributes[i].AttributeType.Namespace == @namespace)
return true;
}
return false;
}
}
Looking at the assembly we see that
DEFINE_INTERFACE_DISPATCH_STUB 4
00007FF627326BE0 mov r11,qword ptr [r10+8]
00007FF627326BE4 mov rax,qword ptr [rcx]
00007FF627326BE7 cmp rax,qword ptr [r11+20h]
00007FF627326BEB jne Attempt1 (07FF627326BF1h)
00007FF627326BED jmp qword ptr [r11+28h]
00007FF627326BF1 cmp rax,qword ptr [r11+30h]
00007FF627326BF5 jne Attempt1+0Ah (07FF627326BFBh)
00007FF627326BF7 jmp qword ptr [r11+38h]
00007FF627326BFB cmp rax,qword ptr [r11+40h]
00007FF627326BFF jne Attempt1+14h (07FF627326C05h)
00007FF627326C01 jmp qword ptr [r11+48h]
00007FF627326C05 cmp rax,qword ptr [r11+50h] <---- crashes here
00007FF627326C09 jne Attempt1+1Eh (07FF627326C0Fh)
00007FF627326C0B jmp qword ptr [r11+58h]
00007FF627326C0F jmp RhpInterfaceDispatchSlow (07FF627327380h)
00007FF627326C14 nop word ptr [rax+rax]
Note the previous probe locations are accessible. This one seems to be on a new page which is not accessible.
- (void**)(r11+48h) 0x000002cd88540ff8 {0x0000000000000000} void * *
0x0000000000000000 void *
- (void**)(r11+50h) 0x000002cd88541000 {???} void * *
<Unable to read memory> void *
This code seems to be where the crash happens : https://github.com/Unity-Technologies/runtime/blob/unity-main/src/coreclr/nativeaot/Runtime/amd64/StubDispatch.asm#L68
As a side note, we've had reports of the nativeaot'd il2cpp executable crashing on macOS arm64. This led to us recently switching back to using the coreclr il2cpp executable. You can see reports of this issue by Unity users
https://discussions.unity.com/t/6000-2-cant-build-with-unsafe-il2cpp-via-cli/1686652/17
https://discussions.unity.com/t/ci-compilations-throw-exit-code-139-only-macosx-builds/1683807
We've had 1 report of it internally. We've haven't collected any information from the macOS arm64 crash. Whether or not the macOS arm64 crash and the windows x64 crash are related I don't know. I thought I'd mention the macOS arm64 crash just in case.
Reproduction Steps
We don't have repro steps.
Expected behavior
il2cpp.exe doesn't crash
Actual behavior
il2cpp.exe crashes
Regression?
We don't know. It is suspicious that we just spotted the issue. We've been using the nativeaot'd il2cpp.exe on windows x64 since .NET 7. We are now on .NET8, we've used a few different versions of .NET8. Then again, this crash seems to be fairly rare so maybe we just haven't noticed.
Known Workarounds
No response
Configuration
Windows x64. The crash happened on a Windows 10 machine on our CI. We haven't tried to reproduce on windows 11.
We used the 8.0.204 .NET SDK to build the nativeaot'd il2cpp.exe. il2cpp.exe is targeting net8.
Other information
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
Status