-
Couldn't load subscription status.
- Fork 1.4k
ZStream#buffer variations #1685
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
@iravid can you please assist me with the tests? Thank you. I'm curious how Editing that seems to hang on my machine. private def bufferFastProducerSlowConsumer =
unsafeRun(
for {
ref <- Ref.make(List[Int]())
latch <- Promise.make[Nothing, Unit]
s = Stream.range(1, 9).tap(i => ref.update(i :: _) *> latch.succeed(()).when(i == 8)).buffer(4)
l <- s.process.use { as =>
for {
_ <- as
_ <- latch.await
l <- ref.get
} yield l
}
} yield l.reverse must_=== (1 to 8).toList
)
Obviously in the above example 8 is not close enough to be evaluated, so it hangs. Anyways, I don't think the test is very good in the first place. Update: |
|
@vasilmkd Hey, nice work on this! The test works exactly as you described it. Happy to accept improvements if you have any suggestions :-) |
|
I'm curious why the test was changed. |
|
I think I changed it a while back because it wasn't very accurate before. |
|
To be honest I have no idea how to test the other buffer implementations. |
|
For the unbounded case, I think your tests are fine. For the other implementations, you need to work with a couple of latches to control the stream's progress, apply backpressure after the first element and release the backpressure once enough elements have passed. Then you consume the rest of the stream and verify that the intermediate elements have been dropped, in the sliding case, or that the last elements were dropped in the Dropping case. |
|
@iravid I require help for the failing |
|
Ok, looking now! |
|
Oh yes, I definitely forgot about the fact that we're signalling |
|
@iravid I'm happy to leave this for after the hackathon. |
|
I started a fresh |
|
As a bonus, I have reduced allocations in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One comment about the busy-wait in bufferSignal.
|
@iravid Thanks for your help. Ready for review. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think your previous approach was better. To avoid skipping elements, we need to switch to "draining" state when the signal has been resolved with end/failure.
|
@iravid This is ready for review once again. Hopefully the code comments make sense. I'd like to know whether we should keep |
| // completed their execution. | ||
| take.await | ||
| } else { | ||
| // Otherwise, the queue is empty, so we interrupt the fiber |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why does take.isDone == false mean that the queue is empty?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh I understand why you went down that route. That's a race condition. There's no guarantee that the take has completed when you check it. It's safer to check the queue's size.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does that mean I can get rid of the take Promise as well?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's see if we're finally there.
|
@vasilmkd Yes, this one looks good! There's still a slim chance for a skipped element there in Could you do a rebase? |
|
@iravid Rebased. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome work, thanks @vasilmkd :-)
|
My pleasure. Let's just wait for CI first. I don't trust my rebasing skills yet. |
Resolves #1659.