Replace List with Chunk in ZTransducer.collectAllN #3886

svranesevic · 2020-06-26T20:46:31Z

Replace usage of List in favor of a more performant Chunk throughout ZTransducer.collectAllN

CLAassistant · 2020-06-26T20:46:36Z

All committers have signed the CLA.

simpadjo · 2020-06-26T22:39:41Z

Yes, it's very tempting to get rid of lists. Unfortunately, Chunk :+ is not as fast.
You can get around it by using ChunkBuilder or even ArrayBuilder as an accumulator.

svranesevic · 2020-06-27T08:04:10Z

Thought about going the ChunkBuilder route, nice addition there would be giving the builder a size hint in collectAllN.

iravid · 2020-06-27T08:56:10Z

Hey @svranesevic! Thank you for following up on this.

A ChunkBuilder would be best here indeed. Appending elements to a chunk is fast, but not as fast as building with a ChunkBuilder.

svranesevic · 2020-06-27T08:58:24Z

@iravid @simpadjo Thank you for the feedback, on it!

…ollectAllWhileM}

iravid · 2020-06-27T11:05:56Z

Thanks @svranesevic, this revision looks great. The only thing we need to do is make sure the creation of the ChunkBuilder is suspended because it is mutable. You can see how that is done in ZSink.collectAll for an example.

svranesevic · 2020-06-27T22:58:22Z

Feedback needed whether beca969 is good enough or I should rewrite ZTransducer.{collectAllWhile|collectAllWhileM} along the lines of 161616e

simpadjo · 2020-06-29T20:13:41Z

streams/shared/src/main/scala/zio/stream/ZTransducer.scala

+  def collectAllN[I](n: Long): ZTransducer[Any, Nothing, I, Chunk[I]] =
+    ZTransducer {
+
+      def go(in: Chunk[I], builder: ChunkBuilder[I], size: Long): (ChunkBuilder[Chunk[I]], ChunkBuilder[I], Long) =


You can append in to builder all at once instead of appending elements one by one. It is faster and would save some lines of code as well

Good idea, I was wrapping my mind around transducer internals while writing this 😅
With that approach we need to handle case when appending the in would produce Chunk that has more elements than N by splitting the in accordingly and carrying over leftover. I will check Chunk#splitAt but afaik it should be better than current impl.

simpadjo · 2020-06-29T20:22:21Z

streams/shared/src/main/scala/zio/stream/ZTransducer.scala

+              }
+
+          case Some(in) =>
+            stateRef.modify {


ChunkBuilder could be reused after emitting a chunk, if I'm not wrong

It could be reused and I almost went for it, but after giving it a thought I'd argue that it's reusability is implementation detail as ChunkBuilder is not ReusableBuilder as of right now.

ChunkBuilder has a public clear method, so yes

simpadjo · 2020-06-29T20:27:53Z

@svranesevic I have couple of ideas how to simplify implementation a bit. Let me explore it today or tomorrow.

svranesevic · 2020-06-29T21:28:05Z

@svranesevic I have couple of ideas how to simplify implementation a bit. Let me explore it today or tomorrow.

Sure thing, looking forward to it!

simpadjo · 2020-06-30T10:26:20Z

@svranesevic how about finalizing collectAllN and then rewriting other transducers in a separate PR?
They also need some polishing so let's keep the PR scope reasonable. Optimizing collectAllN and introducing sizeHint is already a good addition!

Regarding collectAllN, I suggest

not reallocating ChunkBuilder, use just one and clean it
update ChunkBuilder in a bulk way

svranesevic · 2020-06-30T13:24:28Z

@svranesevic how about finalizing collectAllN and then rewriting other transducers in a separate PR?
They also need some polishing so let's keep the PR scope reasonable. Optimizing collectAllN and introducing sizeHint is already a good addition!

Regarding collectAllN, I suggest

not reallocating ChunkBuilder, use just one and clean it

update ChunkBuilder in a bulk way

I agree with scoping down the PR to collectAllN 👍
Regarding the suggestions:

I hold to my disagreement on reusing the ChunkBuilder as it is Builder for which, by the abstraction, behavior is undefined after calling the Builder#result. Reason being that we should rely on abstractions not on implementation details, so here we either bump up ChunkBuilder to be ReusableBuilder or @iravid and/or more folks hop in and vote for the reusing the ChunkBuilder as it is.
"update ChunkBuilder in a bulk way" - 💯

simpadjo · 2020-06-30T13:49:43Z

I guess the main reason why ChunkBuilder is not a ReusableBuilder is that there are not many people who even heard about this trait :)

iravid · 2020-07-01T13:07:59Z

Admittedly reusing the ChunkBuilder is probably unsafe with streams of Any due to the use of classtags. But that's probably not a valid usecase.

In any case, let's scope this down to collectAllN like @simpadjo suggested, keep the implementation without reusing the builder and switch to bulk additions to the builder. Then we can merge 💪

svranesevic · 2020-07-01T20:01:59Z

streams/shared/src/main/scala/zio/stream/ZTransducer.scala

   */
-  def collectAllN[I](n: Long): ZTransducer[Any, Nothing, I, List[I]] =
-    foldUntil[I, List[I]](Nil, n)((list, element) => element :: list).map(_.reverse).filter(_.nonEmpty)
+  def collectAllN[I](n: Int): ZTransducer[Any, Nothing, I, Chunk[I]] =


Implementation detail that bubbled up due to the Chunk's size being Int, might be a deal breaker for this change?

Affects https://github.com/zio/zio/pull/3886/files#diff-e2c16046e05624c39e46b9e4c2870b9cR1596 and https://github.com/zio/zio/pull/3886/files#diff-e2c16046e05624c39e46b9e4c2870b9cR1603.

This is a fair compromise, n is fine as an Int here.

iravid · 2020-07-02T06:46:15Z

streams-tests/shared/src/test/scala/zio/stream/ZTransducerSpec.scala

      suite("collectAllN")(
        testM("happy path") {
-          assertM(run(ZTransducer.collectAllN[Int](3), List(Chunk(1, 2, 3, 4))))(equalTo(Chunk(List(1, 2, 3), List(4))))
+          assertM(run(ZTransducer.collectAllN[Int](3), List(Chunk(1, 2, 3, 4))))(


@svranesevic Last request and then this is good to merge: the implementation is a bit more involved now, so can we add a property check? Just equivalence with List#grouped should be sufficient.

iravid · 2020-07-15T17:37:55Z

Hey @svranesevic, need any help getting this over the finish line?

svranesevic · 2020-07-16T08:34:27Z

Hey @iravid, haven't had time lately to wrap this up, sorry about it, will do today!

svranesevic · 2020-07-17T12:32:39Z

Ping @iravid

iravid · 2020-07-17T12:52:20Z

Looks good @svranesevic! Thanks!

svranesevic requested a review from iravid as a code owner June 26, 2020 20:46

svranesevic added 2 commits June 27, 2020 11:07

Replace List with Chunk in ZTransducer.{collectAllN|collectAllWhile|c…

7ff0922

…ollectAllWhileM}

Use ChunkBuilder

bd83fe5

svranesevic force-pushed the ztransducer_collect_all_chunk branch from bda1b39 to bd83fe5 Compare June 27, 2020 09:32

svranesevic added 2 commits June 27, 2020 11:57

Add missing ChunkBuilder constructor with size hint for Scala 2.13+

37de047

Fix mdoc

2872a50

svranesevic added 2 commits June 27, 2020 16:26

Suspend ChunkBuilder creation

beca969

More involved ZTransducer.collectAllN

161616e

Overhaul collectAllWhile and collectAllWhileM

109b459

simpadjo reviewed Jun 29, 2020

View reviewed changes

svranesevic added 2 commits June 30, 2020 21:27

Scope PR to ZTransducer.collectAllN

af6d3cc

Simplify ZTransducer.collectAllN

b916aed

svranesevic changed the title ~~Replace List with Chunk in ZTransducer.{collectAllN|collectAllWhile|collectAllWhileM}~~ Replace List with Chunk in ZTransducer.collectAllN Jul 1, 2020

Avoid Chunk concatenation

cba36e6

svranesevic commented Jul 1, 2020

View reviewed changes

iravid reviewed Jul 2, 2020

View reviewed changes

svranesevic added 4 commits July 16, 2020 18:40

Merge branch 'master' into ztransducer_collect_all_chunk

f4fa7cf

Add property based check IterableLike#grouped equivalence

658bb77

Use generated grouping size

22a0f60

Remove usage of deprecated Iterator#toIterable

100061a

iravid merged commit 598aeb6 into zio:master Jul 17, 2020

svranesevic deleted the ztransducer_collect_all_chunk branch July 17, 2020 13:09

Uh oh!

Replace List with Chunk in ZTransducer.collectAllN #3886

Replace List with Chunk in ZTransducer.collectAllN #3886

Uh oh!

Conversation

svranesevic commented Jun 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Jun 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simpadjo commented Jun 26, 2020

Uh oh!

svranesevic commented Jun 27, 2020

Uh oh!

iravid commented Jun 27, 2020

Uh oh!

svranesevic commented Jun 27, 2020

Uh oh!

iravid commented Jun 27, 2020

Uh oh!

svranesevic commented Jun 27, 2020

Uh oh!

simpadjo Jun 29, 2020

Choose a reason for hiding this comment

Uh oh!

svranesevic Jun 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

simpadjo Jun 29, 2020

Choose a reason for hiding this comment

Uh oh!

svranesevic Jun 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

simpadjo Jun 30, 2020

Choose a reason for hiding this comment

Uh oh!

simpadjo commented Jun 29, 2020

Uh oh!

svranesevic commented Jun 29, 2020

Uh oh!

simpadjo commented Jun 30, 2020

Uh oh!

svranesevic commented Jun 30, 2020

Uh oh!

simpadjo commented Jun 30, 2020

Uh oh!

iravid commented Jul 1, 2020

Uh oh!

svranesevic Jul 1, 2020

Choose a reason for hiding this comment

Uh oh!

iravid Jul 1, 2020

Choose a reason for hiding this comment

Uh oh!

iravid Jul 2, 2020

Choose a reason for hiding this comment

Uh oh!

iravid commented Jul 15, 2020

Uh oh!

svranesevic commented Jul 16, 2020

Uh oh!

svranesevic commented Jul 17, 2020

Uh oh!

iravid commented Jul 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

svranesevic commented Jun 26, 2020 •

edited

Loading

CLAassistant commented Jun 26, 2020 •

edited

Loading

svranesevic Jun 29, 2020 •

edited

Loading

svranesevic Jun 29, 2020 •

edited

Loading