Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

is this fail only on x86?

is this fail only on x86?

looks like it. https://dev.azure.com/dnceng-public/public/_build/results?buildId=1027838&view=ms.vss-test-web.build-test-results-tab

ping @mangod9

@janvorli @cshung, were there any changes recently in x86 stack walking related to your interpreter changes? Dont believe so, but since you guys have been recently looking into this code?

There were some possibly related changes made by @filipnavara recently.

Do we have some commit range where this may have started happening? While I did touch some of the code I didn't make any conscious changes for the current default runtime configuration (ie. no FEATURE_EH_FUNCLETS defined).

I've taken a look at the dump. This is the code of the method that it is unwinding from (the actual location is marked):

0:000> !U /d 0bdf6778
Normal JIT generated code
MS_jumps1_il.VT.ToStringHelper()
ilAddr is 0BEC20B0 pImport is 07D30990
Begin 0BDF6778, size 91
0bdf6778 55              push    ebp
0bdf6779 8bec            mov     ebp,esp
0bdf677b 56              push    esi
0bdf677c 50              push    eax
0bdf677d 894df8          mov     dword ptr [ebp-8],ecx
0bdf6780 8b4df8          mov     ecx,dword ptr [ebp-8]
0bdf6783 8b09            mov     ecx,dword ptr [ecx]
0bdf6785 badcffab08      mov     edx,8ABFFDCh ("->ToStringHelper")
0bdf678a ff15b878e609    call    dword ptr ds:[9E678B8h] (System.String.Concat(System.String, System.String), mdToken: 060009C6)
0bdf6790 8b55f8          mov     edx,dword ptr [ebp-8]
0bdf6793 e8ccc21a66      call    coreclr!JIT_CheckedWriteBarrierEAX (71fa2a64)
0bdf6798 8b45f8          mov     eax,dword ptr [ebp-8]
0bdf679b 8b4004          mov     eax,dword ptr [eax+4]
0bdf679e 83f802          cmp     eax,2
0bdf67a1 774a            ja      0bdf67ed
0bdf67a3 8d0d1068df0b    lea     ecx,ds:[0BDF6810h]
0bdf67a9 8b0c81          mov     ecx,dword ptr [ecx+eax*4]
0bdf67ac 8d158067df0b    lea     edx,ds:[0BDF6780h]
0bdf67b2 03ca            add     ecx,edx
0bdf67b4 ffe1            jmp     ecx
0bdf67b6 8b45f8          mov     eax,dword ptr [ebp-8]
0bdf67b9 33c9            xor     ecx,ecx
0bdf67bb 894804          mov     dword ptr [eax+4],ecx
0bdf67be 8b4df8          mov     ecx,dword ptr [ebp-8]
0bdf67c1 8d65fc          lea     esp,[ebp-4]
0bdf67c4 5e              pop     esi (gcstress)
0bdf67c5 5d              pop     ebp (gcstress)
0bdf67c6 ff25a864b90b    jmp     dword ptr ds:[0BB964A8h] (gcstress)
0bdf67cc 8b45f8          mov     eax,dword ptr [ebp-8]
0bdf67cf 33c9            xor     ecx,ecx
0bdf67d1 894804          mov     dword ptr [eax+4],ecx
0bdf67d4 8b4df8          mov     ecx,dword ptr [ebp-8]
0bdf67d7 8d65fc          lea     esp,[ebp-4]
0bdf67da 5e              pop     esi (gcstress)
0bdf67db 5d              pop     ebp (gcstress)
0bdf67dc ff259064b90b    jmp     dword ptr ds:[0BB96490h] (gcstress)
0bdf67e2 b80c00ac08      mov     eax,8AC000Ch ("VT")
0bdf67e7 8d65fc          lea     esp,[ebp-4]
>>> 0bdf67ea 5e              pop     esi
0bdf67eb 5d              pop     ebp (gcstress)
0bdf67ec c3              ret (gcstress)
0bdf67ed b978baa308      mov     ecx,8A3BA78h (MT: System.Exception)
0bdf67f2 e81dc8bff7      call    039f3014 (JitHelp: CORINFO_HELP_NEWSFAST)
0bdf67f7 8bf0            mov     esi,eax
0bdf67f9 8bce            mov     ecx,esi
0bdf67fb ff15c87ae609    call    dword ptr ds:[9E67AC8h] (gcstress)
0bdf6801 8bce            mov     ecx,esi
0bdf6803 e8981efb65      call    coreclr!IL_Throw (71da86a0)
0bdf6808 cc              int     3

It ends up calling UnwindEbpDoubleAlignFrameEpilog with the epilogBase pointing to a copy of the code from 0bdf67e7 above, so it looks like this:

0be67a0f 8d65fc          lea     esp,[ebp-4]
0be67a12 5e              pop     esi
0be67a13 5d              pop     ebp
0be67a14 c3              ret

At that point, it tries to invoke SKIP_POP_REG, but as you can see, the instruction is lea esp,[ebp-4] and so the assert fails.
Looking at the code of the UnwindEbpDoubleAlignFrameEpilog and how it got to the problematic spot:

The relevant info members are set as follows:

localloc = 0
savedRegsCountExclFP = 1
rawStkSize = 4

So due to info->rawStkSize == sizeof(void*), we get to the branch where we expect a register pop, but there is the lea instead.

So it seems to me some change in x86 codegen is probably breaking the unwinding code assumptions. @dotnet/jit-contrib, were there any recent changes in this area?

@filipnavara - is it related to #115089?

Do we have some commit range where this may have started happening?

It first started failing on 4/26

is it related to #115089?

That PR affects only built-in COM interop and not JIT codegen. I don't see anything related in the failing tests.

It first started failing on 4/26

I briefly checked the commits on that day. There was nothing quite standing out, although there was the async2 VM merge in the timeframe.

Alright, found the commit range to be 77f71dc...4144fda

I assume you mean 4144fda...77f71dc (switched the start/end)

Thanks, that's helpful!

JFYI, my commits:

• a2779f6 was fixing some off-by-one errors in instruction pointers, but at that time the code path was only used for FEATURE_EH_FUNCLETS on x86 (not enabled). It's now used for all configurations in main, but that was done only few days ago, long after the first failure.
• 97b50b0 should affect only FEATURE_EH_FUNCLETS platforms, so once again should not be relevant to x86 (yet).

The commit range is still too large to figure out a potential culprit. @kunalspathak it would be great to see what the code generated for the method looked like before the range you've mentioned. Maybe seeing the diff would help us better understand which change could have done that.

It looks like the dump is from running JIT/Methodical/VT/callconv/jumps1.il (not the test listed at the top, JIT/Methodical/flowgraph/dev10_bug642944/GCMaskForGSCookie/GCMaskForGSCookie.dll)

I'm guessing the change that caused this is #114899.

@jakobbotsch What was the purpose of this change?

@janvorli Should the x86 unwinder be able to handle either the "pop ecx" or "lea" form, and adjust appropriately?

The change is for the potential case where there is a jmp IL instruction and pop ecx would override the argument value. We can switch the JIT to keep emitting pop ecx in the epilogs that aren't actually doing any jumping, but I am unsure if that will fix the problem (do we then just hit it in the epilogs that do jump instead?).

Note that we had similar case already that was handled in the same way:

Here the other case emits add esp, 4 instead of pop ecx though, which I suppose gets handled by the VM.

Should the x86 unwinder be able to handle either the "pop ecx" or "lea" form, and adjust appropriately?

Yes, I think the unwinder should be modified to accommodate to what JIT generates. It is good that we now understand where the change comes from and that it was intentional.