Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@vasilmkd
Copy link
Contributor

@vasilmkd vasilmkd commented Sep 15, 2019

Resolves #1659.

@vasilmkd
Copy link
Contributor Author

vasilmkd commented Sep 15, 2019

@iravid can you please assist me with the tests? Thank you.

I'm curious how bufferFastProducerSlowConsumer works exactly and how it doesn't lead to a race condition.

Editing that seems to hang on my machine.

private def bufferFastProducerSlowConsumer =
  unsafeRun(
    for {
      ref   <- Ref.make(List[Int]())
      latch <- Promise.make[Nothing, Unit]
      s     = Stream.range(1, 9).tap(i => ref.update(i :: _) *> latch.succeed(()).when(i == 8)).buffer(4)
      l <- s.process.use { as =>
            for {
              _ <- as
              _ <- latch.await
              l <- ref.get
            } yield l
          }
    } yield l.reverse must_=== (1 to 8).toList
  )

...when(i == 4)).buffer(2) must be hitting some special case, tap is evaluated before the value 4 is accepted into the buffer (because 1 is dequeued, 2 and 3 are in the queue and 4 is waiting).

Obviously in the above example 8 is not close enough to be evaluated, so it hangs.

Anyways, I don't think the test is very good in the first place.

Update:
I looked at the git history prior to the process change and it seems to me that this test has undergone a major change.

@iravid
Copy link
Member

iravid commented Sep 17, 2019

@vasilmkd Hey, nice work on this! The test works exactly as you described it. Happy to accept improvements if you have any suggestions :-)

@vasilmkd
Copy link
Contributor Author

I'm curious why the test was changed.

@iravid
Copy link
Member

iravid commented Sep 17, 2019

I think I changed it a while back because it wasn't very accurate before.

@vasilmkd
Copy link
Contributor Author

To be honest I have no idea how to test the other buffer implementations.

@iravid
Copy link
Member

iravid commented Sep 17, 2019

For the unbounded case, I think your tests are fine.

For the other implementations, you need to work with a couple of latches to control the stream's progress, apply backpressure after the first element and release the backpressure once enough elements have passed. Then you consume the rest of the stream and verify that the intermediate elements have been dropped, in the sliding case, or that the last elements were dropped in the Dropping case.

@vasilmkd
Copy link
Contributor Author

@iravid I require help for the failing bufferSliding test. The implementation of bufferSliding seems fairly innocent to me and should work.
I was much more worried about bufferDropping but it seems I have done a decent enough job of implementing. I don't believe this was anticipated when this issue was opened.

@iravid
Copy link
Member

iravid commented Sep 21, 2019

Ok, looking now!

@iravid
Copy link
Member

iravid commented Sep 21, 2019

Oh yes, I definitely forgot about the fact that we're signalling Take.End and Take.Fail too.

@vasilmkd
Copy link
Contributor Author

@iravid I'm happy to leave this for after the hackathon.

@vasilmkd
Copy link
Contributor Author

I started a fresh StreamSpec2 because any additions that I made to StreamSpec resulted in UTF8 string too long compiler error, very nondescript.

@vasilmkd
Copy link
Contributor Author

As a bonus, I have reduced allocations in unTake by not going through Option.

Copy link
Member

@iravid iravid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment about the busy-wait in bufferSignal.

@vasilmkd
Copy link
Contributor Author

@iravid Thanks for your help. Ready for review.

Copy link
Member

@iravid iravid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think your previous approach was better. To avoid skipping elements, we need to switch to "draining" state when the signal has been resolved with end/failure.

@vasilmkd
Copy link
Contributor Author

@iravid This is ready for review once again. Hopefully the code comments make sense.

I'd like to know whether we should keep toUnboundedQueue. Same in ZStreamChunk.

// completed their execution.
take.await
} else {
// Otherwise, the queue is empty, so we interrupt the fiber
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does take.isDone == false mean that the queue is empty?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I understand why you went down that route. That's a race condition. There's no guarantee that the take has completed when you check it. It's safer to check the queue's size.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does that mean I can get rid of the take Promise as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's see if we're finally there.

@iravid
Copy link
Member

iravid commented Oct 1, 2019

@vasilmkd Yes, this one looks good! There's still a slim chance for a skipped element there in signalFirst but I don't know how to solve that currently. For this type of combinator I think this is good enough though.

Could you do a rebase?

@vasilmkd
Copy link
Contributor Author

vasilmkd commented Oct 1, 2019

@iravid Rebased.

Copy link
Member

@iravid iravid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work, thanks @vasilmkd :-)

@vasilmkd
Copy link
Contributor Author

vasilmkd commented Oct 1, 2019

My pleasure. Let's just wait for CI first. I don't trust my rebasing skills yet.

@iravid iravid merged commit d6b6cd2 into zio:master Oct 3, 2019
@vasilmkd vasilmkd deleted the zstream-buffer branch October 3, 2019 20:14
Twizty pushed a commit to Twizty/zio that referenced this pull request Nov 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add ZStream#bufferUnbounded, ZStream#bufferDropping, ZStream#bufferSliding

2 participants