Add ZStream#flatMapParSwitch #1222
Conversation
|
In the current implementation, the Streams produced from each element of the outer stream can enter the "execution window of E.g. in As far as I know, flatMapPar is also non-deterministic in this way. |
|
Hi @vasilmkd, looks really good from a cursory look. I hope to have more time to dig in tomorrow. Thank you for contributing this so quickly! |
|
Regarding the ordering: I think it'd be a positive change to move the |
I made this change, however, I see no difference in behavior. |
| ) | ||
| // This finalizer makes sure that in all cases, the driver stops spawning new streams | ||
| // and the inner fibers are signalled to interrupt and actually exit. | ||
| .ensuring(interruptInners.succeed(()) *> permits.withPermits(n)(ZIO.unit)) |
There was a problem hiding this comment.
This needs to be an ensuringFirst, otherwise the finalizer ordering will be wrong; ensuring appends a finalizer so it runs at the end. This means that the finalizer for the driver stream runs first, and only then will the inner fibers be interrupted.
You can actually keep the equivalence to the previous formulation with ZManaged#onExitFirst.
There was a problem hiding this comment.
Oh but the tests are still passing 🤨 Well this is interesting. I'll dig in some more.
There was a problem hiding this comment.
Ah yes. This is probably just racey. There's nothing preventing the inners from completing before the outer in the finalizer ordering test.
There was a problem hiding this comment.
Wait, that's why we have the n permits semaphore (permits), the driver finalizer kicks off, tells the inner fibers to terminate and waits for their termination (until it succeeds in collecting n permits from the semaphore), at which point all of the inner fibers will have terminated and finalized.
Or am I missing something?
There was a problem hiding this comment.
No, that's exactly it. However, consider that the driver stream has its own set of finalizers (from self) that need to run; for example, the stream you're flatMapping might have acquired a resource that the inner fibers are using.
When you do ensuring, the finalizer passed to ensuring is appended to the list of finalizers, not prepended. So the resulting set of finalizers for the driver stream is:
selfFinalizer *> interruptInners.succeed(()) *> permits.withPermits(n)(ZIO.unit)
Which means that the resources acquired in self would be released before the inner fibers.
There was a problem hiding this comment.
So just change ensuring to ensuringFirst? Both here and in flatMapPar right?
There was a problem hiding this comment.
I'd go with .fork.onExitFirst(...), and interrupt the driver fiber before interruptInners.succeed(()) (if it is Exit.Success).
There was a problem hiding this comment.
Well, I may be wrong here, but the driver fiber is not available in its own definition block. Doesn't fork on a ZManaged, when exiting, interrupt the fiber itself, then run all of the finalizers?
There was a problem hiding this comment.
You're right, that won't be possible with ensuringFirst, but onExitFirst lets you access the result. But I think you're right with your note about the interruption of the fiber, so ensuringFirst is actually sufficient. Thanks for being extra attentive here :-)
|
@vasilmkd Let's deal with the execution order issue you've noticed separately. Can you open an issue that describes what you're seeing? |
|
I will, let me finish up the other PRs first. I think I have some sample code too. |
|
@iravid This is also ready for review. |
* Implement ZStream#flatMapParSwitch * Test ZStream#flatMapParSwitch * Change finalizer code ZStream#flatMapParSwitch * Change finalizer code ZStream#flatMapPar * Release the latch slightly later ZStream#flatMapPar * Fix finalizer execution order
Resolves #1203