Modify ZStream#mergeWith implementation to use a queue #3691
Conversation
af6d0a8 to
4979220
Compare
|
I will update at some point tomorrow to fix Dotty compilation issue. I think I can also get rid of the fiber ref Edit: Done, fixed dotty compilation issue and refactored the solution |
7f65a41 to
e68e435
Compare
There was a problem hiding this comment.
Hi @luis3m ! Thanks a lot picking this up 💪 I think there's a few edge cases to iron out and I've left a few comments below. Let me know if it makes sense.
|
Taking a look why |
| ) | ||
| } | ||
| for { | ||
| handoff <- ZStream.Handoff.make[Take[E1, O3]].toManaged_ |
There was a problem hiding this comment.
This is actually an interesting approach we could take for all concurrent combinators. It's easier to reason about because it doesn't introduce any implicit buffers. If the user wants a buffer, they can easily add one with .buffer.
There was a problem hiding this comment.
Yes! and Handoff is quite nice as it acts as a semaphore between the two fibers as well.
|
@iravid by changing I suppose I must add fibers interruption back for those cases where one fiber finished but the other is still trying to pull. Edit: tried a test case but it seems interrupting fibers isn't needed anyway 🤔 |
|
Great work! I think it's almost there, there's that end of stream invariant to sort out but you may have a better idea than I did. |
|
Solved conflicts |
|
@regiskuckaertz I'd been trying to figure out why the following code wasn't working. for {
done <- done.get
take <- if (done.contains(true)) handoff.poll.some else handoff.take
result <- take.done // previously IO.done(take)
} yield resultBy moving back to Edit: I guess only
|
|
| _ <- handler(chunksL.map(_.map(l)), List(L, E).contains(strategy)).fork.toManaged_ | ||
| _ <- handler(chunksR.map(_.map(r)), List(R, E).contains(strategy)).fork.toManaged_ |
There was a problem hiding this comment.
| _ <- handler(chunksL.map(_.map(l)), List(L, E).contains(strategy)).fork.toManaged_ | |
| _ <- handler(chunksR.map(_.map(r)), List(R, E).contains(strategy)).fork.toManaged_ | |
| _ <- handler(chunksL.map(_.map(l)), List(L, E).contains(strategy)).forkManaged | |
| _ <- handler(chunksR.map(_.map(r)), List(R, E).contains(strategy)).forkManaged |
Tests pass with this change, and we get the benefit of knowing that the running streams are definitely interrupted before their ZManaged finalizers run.
There was a problem hiding this comment.
Ah zipWithLatest is failing. I'll check why...
There was a problem hiding this comment.
@iravid yes, I had that one failing with forkManaged
There was a problem hiding this comment.
Pulling was hanging for some reason. Both fibers were stuck and the test was timing out
There was a problem hiding this comment.
Looks like a bug in Handoff, I think. Digging into it.
There was a problem hiding this comment.
Well it's definitely something with the TestClock. The test hangs in the schedule's sleep.
There was a problem hiding this comment.
Are you looking into it or is this something Adam should check?
There was a problem hiding this comment.
Yes, found the problem. We're just using it wrong. The adjust call happens concurrently with the fibers sleeping in the stream. Checking how best to fix it now. Basically the same thing as assertWithChunkCoordination fixes.
There was a problem hiding this comment.
Oh I must have not saved the file between running the tests. Changing forkDaemon -> fork in ZManaged#fork does fix the problem.
Now I'm pretty sure it's because the TestClock doesn't see the producer fibers suspending.
For now @luis3m let's change this from .fork.toManaged_ to .fork.interruptible.toManaged(_.interrupt). This works correctly.
Closes #3682