Add ZStream#interleave, ZStream#interleaveWith, ZStream#combine #1355

adamgfraser · 2019-08-05T21:30:38Z

Adds an interleave method on ZStream for deterministic merging of streams. Uses a stream of boolean values to determine whether to pull from the left or right stream each time so could be used for simply alternating between two streams, pulling from each stream with different frequency, or using effects to determine which stream to pull from next.

@iravid Would you take a look when you get the chance?

iravid

@adamgfraser Great work. This looks like a great combinator.

The only thought I have is that the implementations of zipWith, mergeWith and this combinator are all eerily similar. I wonder if we can abstract over them? @jdegoes could you have a look at this?

iravid · 2019-08-07T05:49:29Z

streams/shared/src/main/scala/zio/stream/ZStream.scala

+   * either `s1` or `s2` are exhausted further requests for values from them
+   * will be ignored.
+   */
+  final def interleave[R, E, A](


How about an infix version too?

Definitely.

iravid · 2019-08-07T05:54:32Z

streams/shared/src/main/scala/zio/stream/ZStream.scala

+            rightQueue <- s2.toQueue()
+          } yield (queue, leftQueue, rightQueue)
+
+          ZManaged.fromEffect {


Hmm I think the scoping is off here. ZManaged#use is usually not needed in stream combinators, unless you are sure you want the finalizers to fire. Something like this would defer them until the caller decides they should run:

for { queue <- b.toQueue leftQueue <- s1.toQueue rightQueue <- s2.toQueue result <- loop(false, false, s, queue, leftQueue, rightQueue).toManaged_ } yield result

Yes, that works. I see now that it doesn't make sense to use a resource and then just wrap it in another resource.

iravid · 2019-08-07T06:03:59Z

streams/shared/src/main/scala/zio/stream/ZStream.scala

+   * will be ignored.
+   */
+  final def interleave[R, E, A](
+    b: ZStream[R, E, Boolean],


Is this expressive enough to handle two streams with different lengths? if all the caller of this combinator can do is provide a stream of booleans, surely they can't know if one of the streams ended and therefore they will not be able to "switch over" to reading just one side.

Maybe we need this to be a function that uses the queues themselves?

I was thinking that if the stream of booleans eventually requested values from both side like Stream(true, false).forever then the caller would be able to get all the values, but it is a little indirect.

What if we added a parameter f: (ZStream[R, E, A], ZStream[R, E, A]) => ZStream[R, E, A) where the first stream is the result of interleaving and the second is the "leftover" stream? So if the caller wanted to pull all remaining elements from the leftover stream they could use _ ++ _, if they wants to ignore to ignore them they could use _._1, or they could apply more complex logic.

Sorry, I read more carefully and saw that you skip over a side that has ended. That makes sense! I think this is fine in this form then.

I suggest that we have a shorthand combinator for the common case of Stream(false, true).forever though :-) This combinator can be called interleaveWith and the shorter one interleave.

jdegoes · 2019-08-07T08:24:06Z

@adamgfraser Great work. This looks like a great combinator.

The only thought I have is that the implementations of zipWith, mergeWith and this combinator are all eerily similar. I wonder if we can abstract over them? @jdegoes could you have a look at this?

Perhaps a unification might look something like: combine((dest, left, right) => ...), i.e. you get a destination queue (or at least the offer part of it), and you can push things there from either left, or right, in whatever order you choose; all scoped by Managed to ensure things cleanup properly.

iravid · 2019-08-07T08:41:32Z

Right, I agree with that direction. What do you think @adamgfraser? Want to give this a shot?

adamgfraser · 2019-08-07T09:04:36Z

@iravid Yes, I'm on it!

adamgfraser · 2019-08-08T02:16:54Z

@iravid @jdegoes Take a look at this.

iravid · 2019-08-08T05:57:18Z

streams/shared/src/main/scala/zio/stream/ZStream.scala

+            left   <- s1.toQueue(lc)
+            right  <- s2.toQueue(rc)
+            dest   <- Queue.unbounded[Take[E, C]].toManaged(_.shutdown)
+            _      <- f0(left, right, dest).toManaged_


Don't we need to fork here?

Do we actually need an output queue? I haven't tested if all combinators work with this, but what if we do the following:

f0: (Queue[Take[E, A]], Queue[Take[E, B]]) => ZIO[R, E, Take[E, C]]

run f0 with ZStream.unfoldM and unTake

Does that make sense?

Let me give it a try.

Yes we can.

iravid

Looks excellent! See my comment on mergeWith. We can either enhance combine further with a state parameter or defer the enhancement and the mergeWith re-implementation - your call!

iravid · 2019-08-08T18:50:24Z

streams/shared/src/main/scala/zio/stream/ZStream.scala

+            result <- ZStream
+                       .unfoldM((left, right)) {
+                         case (left, right) =>
+                           f0(left, right).map(take => Some((take, (left, right))))


Ah we could actually drop the unTake and signal errors/completion here directly. Take.option can help here.

iravid · 2019-08-08T20:10:36Z

streams/shared/src/main/scala/zio/stream/ZStream.scala

-    }
+    for {
+      queue  <- ZStream.managed(queue)
+      result <- ZStream.combine(self, that)((_, _) => queue.take)


😂

I think this could be implemented if ZStream.combine allowed the function to keep a state between the invocations, like an unfold. Then you could use raceWith to race takes against the two queues. Whenever one side wins, you pass the pending side through the state to the next invocation. That would save the need for the output queue here.

That said, we can defer that to a follow-up ticket that enhances combine and implements mergeWith in terms of it if you'd like to get interleave in sooner.

Yes, I think that makes sense. I guess we could potentially use a Ref to pass the loser from one invocation to another like I did to manage state in the other combinators. But I was definitely feeling the lack of ability to maintain some state in writing them. It would probably be both easier to use and more performant to pass the state explicitly.

iravid · 2019-08-08T20:11:17Z

streams/shared/src/main/scala/zio/stream/ZStream.scala

-            right <- that.toQueue[E2, B](rc)
-            s     <- ZManaged.fromEffect(loop(false, false, left, right, s))
-          } yield s
+  ): ZStream[R1, E1, C] = {


Nice code removal here!

iravid · 2019-08-08T20:15:16Z

streams/shared/src/main/scala/zio/stream/ZStream.scala

+   * will be ignored.
+   */
+  final def interleave[R, E, A](
+    b: ZStream[R, E, Boolean],


Sorry, I read more carefully and saw that you skip over a side that has ended. That makes sense! I think this is fine in this form then.

I suggest that we have a shorthand combinator for the common case of Stream(false, true).forever though :-) This combinator can be called interleaveWith and the shorter one interleave.

iravid · 2019-08-08T20:16:36Z

streams/shared/src/main/scala/zio/stream/ZStream.scala

  final def bracket[R, E, A](acquire: ZIO[R, E, A])(release: A => ZIO[R, Nothing, _]): ZStream[R, E, A] =
    managed(ZManaged.make(acquire)(release))

+  final def combine[R, E, A, B, C](s1: ZStream[R, E, A], s2: ZStream[R, E, B], lc: Int = 2, rc: Int = 2)(


This one could also use an infix version.

It's also interesting to think of how would the generalized n-stream version would look like!

Is it better to just do everything infix? That seems more consistent with most of the other methods.

Yes, good point. Probably only variadic combinators should have a prefix form.

And yes about generalizing further to multiple streams! But that I think we can save for another PR! 😃

adamgfraser · 2019-08-09T11:15:44Z

This is ready for review.

iravid

Great work @adamgfraser!

iravid · 2019-08-09T11:45:58Z

streams/shared/src/main/scala/zio/stream/ZStream.scala


-            s <- ZManaged.fromEffect(loop(false, false, s, queue))
-          } yield s
+    self.combine(that)((false, false, Option.empty[Loser])) {


This is so nice!

adamgfraser · 2019-08-09T19:55:57Z

@iravid Thanks for all your help! Excited to get this merged!

iravid · 2019-08-09T19:59:41Z

@adamgfraser Happily and thank you for working on this - I was wondering for a long time how we'd abstract over all of these binary merging operators.

ghostdogpr · 2019-08-15T14:49:51Z

@adamgfraser seems the test interleaveWith is flaky, I’ve seen it fail on an unrelated PR.

https://circleci.com/gh/zio/zio/16550

adamgfraser · 2019-08-15T14:51:33Z

@ghostdogpr Let me take a look.

iravid · 2019-08-15T14:54:05Z

Might also be related to the process refactoring as I haven’t seen that fail before.

adamgfraser · 2019-08-15T15:11:41Z

@ghostdogpr Have you rebased against master? I'm not able to reproduce locally.

ghostdogpr · 2019-08-15T23:34:18Z

Yeah it happened on a PR that rebased against master. The failure only happened once and disappeared after re-running the build.

adamgfraser added 2 commits August 5, 2019 15:49

add ZStream#interleave

4d2a5e3

linting changes

57e9ec2

iravid reviewed Aug 7, 2019

View reviewed changes

adamgfraser added 2 commits August 7, 2019 21:57

unify stream combinators

3b379c9

add missing parameter type

487d123

iravid reviewed Aug 8, 2019

View reviewed changes

refactor

ab49280

iravid reviewed Aug 8, 2019

View reviewed changes

keep internal state

07f174d

iravid approved these changes Aug 9, 2019

View reviewed changes

iravid changed the title ~~Add ZStream#interleave~~ Add ZStream#interleave, ZStream#interleaveWith, ZStream#combine Aug 9, 2019

iravid merged commit 652c3d5 into zio:master Aug 9, 2019

adamgfraser deleted the interleave branch August 9, 2019 19:54

Uh oh!

Add ZStream#interleave, ZStream#interleaveWith, ZStream#combine #1355

Add ZStream#interleave, ZStream#interleaveWith, ZStream#combine #1355

Uh oh!

Conversation

adamgfraser commented Aug 5, 2019

Uh oh!

iravid left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdegoes commented Aug 7, 2019

Uh oh!

iravid commented Aug 7, 2019

Uh oh!

adamgfraser commented Aug 7, 2019

Uh oh!

adamgfraser commented Aug 8, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iravid left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamgfraser commented Aug 9, 2019

Uh oh!

iravid left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamgfraser commented Aug 9, 2019

Uh oh!

iravid commented Aug 9, 2019

Uh oh!

ghostdogpr commented Aug 15, 2019

Uh oh!

adamgfraser commented Aug 15, 2019

Uh oh!