[Kernel] Fix NORMAL mutex self-relock inconsistency between fast and slow lock paths - #884

What this does

Fixes a narrow but real inconsistency in PthreadMutexLockCore (KernelPthreadCompatExports.cs). There are two places that handle the same case — a thread calling scePthreadMutexLock on a MutexTypeNormal mutex it already owns — and they disagreed.

Background

• #451 added an uncontended fast-path check (before acquiring state.SyncRoot) that treats self-relock on a NORMAL mutex as compatibility recursion: increment the recursion count, return OK.
• #748 later found that behavior was starving other threads on some title, and deliberately reverted the other copy of this same check — the one inside the SyncRoot-protected slow path — back to a real EDEADLK. Good, evidence-based fix, explained clearly in that PR's commit message.
• But #748's revert didn't touch the fast-path copy from #451, since it's a separate if block earlier in the same function. So since #748, the actual outcome of a NORMAL self-relock has depended on which of the two checks a given call happens to reach — OK via the fast path, EDEADLOCK via the slow path — rather than on anything about guest behavior.

This PR makes the fast path match the slow path's current (intentional, evidence-based) behavior: real EDEADLK, no silent recursion.

Testing

• dotnet test --no-restore — 874 passed, 0 failed (873 existing + 1 new).
• Added NormalMutex_SelfRelockIsADeadlockNotSilentRecursion to PthreadMutexSemanticsTests.cs, pinning the now-consistent behavior via the fast (uncontended, single-thread) path specifically — this test would have failed before this change.
• Real-game testing: I ran into this exact case while investigating black-screen/hang reports on Stray (PPSA02101, Stray [PPSA02100] #519) and Paw Patrol: Grand Prix (PPSA05521, Paw Patrol: Grand Prix (PPSA05521) — startup spins in a codepoint-enumeration loop whose bound is a live-read shared counter #875) — both show a scePthreadMutexLock retry loop returning ORBIS_GEN2_ERROR_DEADLOCK repeatedly on the same guest mutex. I want to be upfront that this fix does not change either title's observed behavior — both were already reaching the SyncRoot-protected slow path, which already returned EDEADLK before this change (confirmed by re-running both against this branch: identical result). So this is a real, narrower inconsistency than I initially suspected while investigating those two titles; it's worth fixing on its own merits (any title that happens to take the fast path deserves the same, intentional Fix Windows fixed-memory allocation and mutex handling issues #748 behavior as one that takes the slow path), but it is not a fix for Stray [PPSA02100] #519 or Paw Patrol: Grand Prix (PPSA05521) — startup spins in a codepoint-enumeration loop whose bound is a live-read shared counter #875 — those remain open with the actual cause of the self-relock still unclear.

AI assistance disclosure

I used Claude Code (Anthropic) to help investigate KernelPthreadCompatExports.cs, trace the git history of both self-relock checks (via git log -p -L) to understand why they diverged, write the fix and the new test, and draft this PR description. I reviewed the code change, the git history it's based on, and the test/real-game verification myself and can explain and defend it in review.

Checklist

• I have read and followed CONTRIBUTING.md.
• I tested my changes or marked the testing section as N/A.
• I wrote this pull request description myself and did not paste AI-generated explanations. (see AI assistance disclosure above — I did not write this text myself)
• I listed the game(s) I tested (or marked the testing section as N/A).

🤖 Generated with Claude Code