Leaving this as draft for the time being as I want to add tests for the newly added queues and until we decide whether the approach for generating FiberIds is solid (more below)

/claim #8611
/fixes #8611
/split @ghostdogpr

@jdegoes assuming you're happy to reward the bounty for this PR, I'd like to share it with @ghostdogpr as we worked on this together. Is there something I need to do to indicate to the Algora bot to split the payment?

Now, let's dig into the changes in this PR

Avoiding global resource contention

Anyone that gave it a go at resolving this issue (myself included) probably kept running into loops where optimizing something didn't show any improvement in the benchmarks. The reason for this is that there are multiple points where threads are attempting to read/write from the same globally initialised resource, effectively limiting the overall throughput of the ZIO runtime to the throughput of that individual object. So unless all of the places of contention are resolved, the benchmark wouldn't show any improvement - even if the individual optimization was solid. The biggest culprits:

1. ZScheduler#globalQueue
2. WeakConcurrentBag#nursery
3. FiberRef.make
4. ZScheduler#submittedLocations

For (1) and (2), the solution I came up with was to use a "partitioned" queue consisting of multiple sub-queues, so when threads are offering / polling from them the chance of contention is reduced. Interestingly, when I first thought of this approach and shared an initial implementation with @ghostdogpr, he pointed out it was scarily similar to the cats-effect global queue. At this point we knew we were on the right track and did what any software engineer worth their buck would do; "ported" (i.e., stole) any improvement we could see in their implementation into ours.

Now (3), this is something I'm very conflicted about, and would appreciate feedback on. The current implementation uses an AtomicInteger and increments it whenever a new FiberId is created. The problem with this approach is that when multiple threads are creating fibers, there is contention on updating the AtomicInteger. This acts up as a big bottleneck that limits the overall scalability of the ZIO runtime. The solution in this PR is to allocate a random Int instead, but there are 2 issues with this approach:

1. it's extremely unlikely but technically possible that 2 fibers created with the same startTimeMilli and location will have the same id. One way to reduce the chances even more is to use a random Long as the id.
2. Previously, Fiber.Runtime objects were ordered based on (startTimeMilli, id) and with these changes we lose the ability to order based on id in case of a startTimeMilli collision. Having said that, the ordering is only used in zio.test macros, so this might not be a big issue?

Finally, (4) was a bit easier to solve. Instead of having a globally-defined map to store the submittedLocations, the way to avoid the single resource contention was by having each ZScheduler.Worker submit its own locations in its own Map. Then the ZScheduler.Supervisor (the only place that reads from submittedLocations), aggregates the submitted locations from all workers. Since the supervisor only needs to read these locations when it suspects a worker thread is blocking unnecessarily (which is never if users properly wrap blocking code in ZIO.blocking), it's much cheaper than having all workers write to the same globally shared Map.

Improved thread scheduling(?)

Now this is an interesting one. When a thread is spawning multiple child fibers (e.g., in the case of ZIO.foreachPar or when using .fork in a loop), we found that it's much more performant if the thread yields after every X number of forks. This brings a huge improvement because the main thread occasionally yields from enqueueing more runnables into the global queues, giving a chance for the newly spawned workers to complete tasks in their local queues work prior to yielding themselves (in the case of async jobs). We found that yielding every ~100 forks is a good enough tradeoff between the added overhead of yielding and improved thread scheduling.

CPU hot-path optimizations

Besides resolving global resource-contention and thread-scheduling this PR also optimizes the following which were found to improve this fairly substantially

1. The values added to WeakConcurrentBag are now wrapped in WeakReference. This means that fibers can be GC'd much earlier
2. Only a single thread is allowed to perform GC in WeakConcurrentBag under "auto gc" conditions
3. Made multiple optimizations to FiberRefs to avoid boxing of Ints as much as possible and avoid calling transform on the Map if we know it won't transform any of the entries

Benchmarking results

TLDR:

• ZScheduler-based runtime is ~x6.5 faster with FiberRoots enabled
• ZScheduler-based runtime is ~x15 faster without FiberRoots enabled
• Loom-based runtime ~x2 faster. The bottleneck at this point is the Loom executor scheduler and unfortunately we can't optimize it

cats-effect (baseline)

[info] Benchmark                        (n)   Mode  Cnt    Score   Error  Units
[info] ForkJoinBenchmark.catsForkJoin             10000  thrpt   10  2906.462 ±  14.976  ops/s

series/2.x (Note: this is the same with / without FiberRoots enabled)

JDK 17 - ZScheduler
[info] Benchmark                        (n)   Mode  Cnt    Score   Error  Units
[info] ForkJoinBenchmark.zioForkJoin  10000  thrpt   10  299.090 ± 5.982  ops/s

JDK21 - Loom
[info] Benchmark                        (n)   Mode  Cnt    Score    Error  Units
[info] ForkJoinBenchmark.zioForkJoin  10000  thrpt   10  448.071 ± 28.781  ops/s

PR

JDK 17 - ZScheduler
[info] Benchmark                                    (n)   Mode  Cnt     Score     Error  Units
[info] ForkJoinBenchmark.zioForkJoin              10000  thrpt   10  1931.372 ±  31.926  ops/s
[info] ForkJoinBenchmark.zioForkJoinNoFiberRoots  10000  thrpt   10  4812.394 ± 282.183  ops/s

JDK 21 - Loom
[info] Benchmark                                    (n)   Mode  Cnt    Score   Error  Units
[info] ForkJoinBenchmark.zioForkJoin              10000  thrpt   10  660.001 ± 5.386  ops/s
[info] ForkJoinBenchmark.zioForkJoinNoFiberRoots  10000  thrpt   10  758.261 ± 6.501  ops/s