-
Couldn't load subscription status.
- Fork 1.4k
Fix #785, Fix #1441, prevent race condition on interrupted variable by moving interrupted into atomic state
#1792
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix #785, Fix #1441, prevent race condition on interrupted variable by moving interrupted into atomic state
#1792
Conversation
|
A clean reproduction for testing: import java.util.concurrent.atomic.{AtomicLong, LongAdder}
import zio.{UIO, _}
import scala.language.reflectiveCalls
object ZioInterruptLeakOrDeadlockRepo extends zio.App {
val leakedCounter = new AtomicLong(0L)
val startedCounter = new LongAdder
val completedCounter = new LongAdder
val awakeCounter = new LongAdder
val pendingGauge = new LongAdder
def run(args: List[String]): ZIO[Environment, Nothing, Int] = {
val leakOrDeadlockTest = for {
_ <- UIO {
startedCounter.increment()
pendingGauge.increment()
}
sleepInterruptFiber <- IO.never.fork
// This blocks / leaks once every 20,000 rounds or so
_ <- sleepInterruptFiber.interrupt
.ensuring(UIO {
awakeCounter.increment()
pendingGauge.decrement()
})
_ <- UIO(completedCounter.increment())
} yield ()
val main = for {
_ <-
ZIO.runtime[Any].map(_.Platform.executor.metrics.get)
.flatMap {
metrics => UIO(new Thread(() => {
while (true) {
println(s"started=${startedCounter.longValue()} awake=${awakeCounter.longValue()} completed=${completedCounter.longValue()} pending=${pendingGauge.longValue()} queued=${metrics.size}")
Thread.sleep(1000L)
}
}).start())
}
_ <- leakOrDeadlockTest.forever
} yield 0
main
}
}
|
|
wow, you found it! Awesome detective work 🕵 |
|
Race scenario: @regiskuckaertz |
| } | ||
|
|
||
| case Executing(status, observers0) => | ||
| case Executing(status, observers0, _) => |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This extra flag is necessary in your fix. In my fix the model changed a bit, so pushing the "set" prior to the atomic ref update should in theory take care of it. But this looks good for now!
|
@neko-kai Can you add the test? |
b83f879 to
069d2f7
Compare
|
@jdegoes I've added a test that just runs IO.fork.interrupt 10K times, multiplied by test builds it should probably be able to catch a regression over many builds. I've also added a benchmark that does the same IO.fork.interrupt and should definitely start hanging on regression. I've also added a benchmark for |
…riable by moving `interrupted` into atomic state Also, prevent potential volatility problems with accessing interruptStack from multiple threads by moving `interruptible` state into Suspended
…oduces the deadlock scenario)
069d2f7 to
aa06c05
Compare
|
I believe another inconsistency with IO.effectAsyncMaybe[Nothing, Unit] {
k =>
Future {
Thread.sleep(200L)
k(IO.dieMessage("INCOHERENT CONTINUE"))
}
Some(IO.unit)
}
.flatMap(_ => ZIO.never)
.ensuring(IO.dieMessage("CAN'T HAPPEN!"))
.uninterruptiblethat "overtakes" the fiber from an async that was "cancelled", but only if the fiber enters suspension at an opportune time |
|
There's also a remaining race with object ZioIncoherentInterruptAsyncRepro extends zio.App {
def run(args: List[String]): ZIO[Environment, Nothing, Int] = {
val test =
ZIO.effectAsync[Environment, Nothing, Unit](k =>
Future {
Thread.sleep(200L)
k(console.putStrLn("INCOHERENT CONTINUE"))
})
.ensuring(console.putStrLn("Async finalizer!") *> ZIO.never)
.ensuring(console.putStrLn("CAN'T HAPPEN!"))
test.fork
.flatMap(_.interrupt.delay(50.millis))
.as(0)
}
}Basically, every single point where multi-threaded access can happen in non-atomic or non-guarded way is a problem (: |
|
Will address these two - #1792 (comment), #1792 (comment) - separately. |
|
@neko-kai Is the performance of race improved after these fixes? |
|
@iravid |
|
Ah too bad. Ok :-) |
…riable by moving `interrupted` into atomic state (zio#1792) * Fix zio#785, Fix zio#1441, prevent race condition on `interrupted` variable by moving `interrupted` into atomic state Also, prevent potential volatility problems with accessing interruptStack from multiple threads by moving `interruptible` state into Suspended * return enterAsync after-check, do not memoize `interruptible` in enterAsync loop * Add benchmarks for fork-interrupt and empty race (fork-interrupt reproduces the deadlock scenario) * Add regression test in RTSSpec * Return case class match * Refactor flatten-effectTotal to effectSuspendTotal * add missing private in FiberContext
Also, prevent potential volatility problems with accessing interruptStack from multiple threads by moving
interruptiblestate into Suspended