Goal

As a way to reduce the risk of bing-bang change, we are approaching the project of transitioning to cooperative suspend by a middle ground approach: hybrid suspend. It consist in using cooperative suspend for runtime and managed code, and preemptive suspend for any other kind of code (external libraries, OS, etc.). Thus, preemptive suspend is used only for code which is not controlled by Mono, while cooperative suspend is only used for code which is fully controlled by Mono. This forces us to make sure we transition to the right state, to ensure we use the proper mechanism of suspending.

A hopefully exhaustive list of where these transitions are needed is:

1. managed-to-native wrappers
2. native-to-managed wrappers
3. MONO_API functions
4. trampolines
5. external libraries calls (libc, btls, etc.)

Points 1. and 2. are quite easy to achieve, and are already working for supporting cooperative suspend on watchOS. Some of 3., 4. and 5. have already been done for watchOS and Mono's CI for cooperative suspend. Unfortunately, 3. and 4. are not complete enough to fully support VSfM, XI/XM and XA, or any other embedding project.

We want to switch to hybrid suspend by default by 2018-08, and to do so we have identified some necessary steps to ship, and some longer-term improvement steps.

Tasks

For 2018-08

• Implement hybrid suspend (native -> preemptive; runtime+managed -> cooperative) - [coop] Hybrid suspend #8068
• MONO_API functions and trampolines
  • List functions loaded from XI/XM and XA, and functions which are public API: https://gist.github.com/luhenry/f450d55128deddcedc2e27852ab6dc23 /cc @luhenry
  • Ensure transition in the above functions and trampolines
• Performance: measure impact of hybrid suspend + fix major regressions
  • Some possible measurements
    • time-to-suspend
    • wall clock time: benchmarks, VSfM (startup time, new projects, new file, etc.)
• Documentation: how to disable, what's needed to transition embedders, how to use diagnostics
• Timeout time-to-suspend on CI
• Diagnostics: mono_trace in mono-threads-debug.h
• Resolve failing tests
  • abort-cctor.exe [threads] Delay abort via handler block guard when in abort protected block. #7599
  • bug-58782-plain-throw.exe and bug-58782-capture-and-throw.exe [runtime] Rework how the native-to-managed eh callback is called. #5837
  • [coop] Fix sporadic test failure in System.Runtime.Remoting.RemotingServicesTest.Bug46473 #7981 - Sporadic MonoTests.System.Runtime.Remoting.RemotingServicesTest.Bug46473 BLOCKING to BLOCKING transition (new issue created from comments below)
  • Sporadic bug-18026.exe timeouts - Hybrid Suspend #6921 (comment) - fixed by [coop] mono_coop_cond_signal should switch to GC Safe thread state #9347
  • System.Threading.Tests.ThreadLocalTests.RunThreadLocalTest8_Values_NegativeCases failure
  • Fix Mono.Profiler.Log-xunit tests - Fix Mono.Profiler.Log-xunit tests under hybrid suspend #8355
  • marshal_mutex deadlock - Hybrid suspend deadlock taking the marshal mutex #9407 - fixed in [marshal] Use a MonoCoopMutex for marshal_mutex #9408
  • Make mono_threads_enter_gc_safe_region_unbalanced_with_info cheaper on normal platforms Make mono_threads_enter_gc_safe_region_unbalanced_with_info cheaper on non-bitcode platforms #9414
  • Turn it on by default on desktop [coop] Turn hybrid suspend on by default on desktop platforms #9811

For Future

• Static analysis: Centrinel checks for missing transitions in MONO_API functions;
  • Centrinel checks to insert MONO_REQ_GC_UNSAFE_MODE assertions in functions that touch managed memory.
• Diagnostics:
  • Fix BLOCKING -> RUNNING transitions to do no locking - Hybrid suspend: backtrace in GC Unsafe mode considered harmful (on Linux) #8356
  • Integrate with verbose crashes feature
• MONO_API functions:
  • All other MONO_API functions not loaded by XI/XM/XA
• Performance: fix tail of regressions
• Documentation: how does it work

Documentation

https://paper.dropbox.com/doc/What-I-know-about-suspend-GtXdT1CxuMsqwSBFVLqQf