All committers have signed the CLA.

Taking a step back here, @borissmidt I'd like to see a benchmark of this first.

Building a list (and yes, reversing it) is very fast in Scala because the appends are O(1) and the reversal can happen with a pre-allocated buffer.

According to this: https://www.lihaoyi.com/post/BenchmarkingScalaCollections.html#construction-performance the only thing that is faster than prepending to a list is pre-allocating an array and filling it up, but we can't do that because we don't know up-front what is the size of the stream.

Another thing to consider when collecting to a chunk is the size limit: a ChunkBuilder will accumulate elements into an array, so this has a hard size limit at 2^31. Not sure if this is a showstopper or not.

Maybe it is a better idea to change it to Seq then as the return type then you can change it on to something that would give the best performance without breaking binary compatibility?

Since i can imagine that there are other better structures than an array that has to be copied every so often and a list that has a lot of indirection. Also in case of reversing it has to iterate over the whole list?

How could i best benchmark this?

looking at the scala 2.13 source for the reverse it doesn't use a pre-allocated buffer so you need to iterate the full list.

  final override def reverse: List[A] = {
    var result: List[A] = Nil
    var these = this
    while (!these.isEmpty) {
      result = these.head :: result
      these = these.tail
    }
    result
  }

In case of a ListBuilder it used mutible cons and replaces the 'next' each time you append to the ListBuilder.

Array builder pre allocates an Array and doubles the size each time it runs out of capacity, (it reminds me of the c++ vector class)

the benchmark doesn't use 'builders' in for construction i could extend that and give an update?
https://github.com/lihaoyi/scala-bench/blob/master/bench/src/main/scala/bench/Benchmark.scala

i'll rerun the constructor benchmarks with a list builder and array builder for scala 2.11/2.12/2.13 since these results are from scala 2.11.8.

as for the maximum size, list, seq and vector only index up to 2^32 elements so maybe use a long as index for chunk ;)

I've updated the scala-bench project to use builders and will run them overnight.
It could be used to benchmark the chunk collection?

https://github.com/borissmidt/scala-bench

@borissmidt I think it could be easier to just add another test case to ChunkBenchmarks.scala in this project.

bench-result.zip
I was just curios my self to see if the array builder is faster than the normal list add operation and indeed it is. results are the average runtime in nano seconds of iteration of a run.

So in these results i see that the arrayBuilder is signifcantly faster than the array :+ .
A list can be made faster then the arrayBuilder. But its always slower if you reverse it afterwards.

Can you post the results here?

You were testing list prepend, not append, right?

Yes list prepend! the :: operation. you can view the code and run it yourself i've posted it.
I also call the builder.result at the end of each run so its a fair comparison.

Another suprising fact, very big arrays become slow to allocate than list > 1M elements in size.

Right. I'm looking at the results of ListReverse and ArrayBuilder which is what we'd be using in practice. At 250k elements, the List becomes faster.

I think this is good evidence that ChunkBuilder and Chunk are good for the runCollect usecase!

ZSink is expected to be the final stage of stream processing. If a user decided to collect a huge stream into a List/Chunk, it opted-in for having abnormally large objects in its program, and will face problems anyway.
I don't think we should spend too much of an effort optimizing for this case.

@borissmidt you can get rid of linting errors by running sbt prepare (see https://github.com/zio/zio/blob/master/docs/about/contributing.md)

There is still the open decision to keep the old Api and add the chunk as separate function calls.
Or replace them with chunk as i've done now.

"size can be modified locally" at src/test/scala/zio/test/GenSpec.scala:607 fails and i don't understand why this is related to the changes i had made.

@adamgfraser Can you peek at the test failure here and give us a hint?

@iravid Yes let me take a look.

I fixed, it but i'm not sure if it is a clean way of doing it, i tried t make a 'lazy chunkbuilder' that creates a new instance on the first += operation. But it wasn't possible with the typing of the return type.
An alternative could have been using Option. But then i wonder what the impact of the extra allocations would have been.

I think it should be ok now? i've added the missing merge conflicts.

I ran the fmt and fix command but there are no more changes detected. so i don't know why the fmtCheck still fails. Is there any reason why fix and fmt aren't ran during the build?

Use the prepare command in sbt. I think the contributor guide mentions that.

The CI doesn't commit to Github, so it doesn't format the code there.

Aha you have to do sbt clean prepare otherwise it has chached the formatting and will not do anything. I've more experience with gradle.

Oke i fixed all issues

@iravid Could this be merged before there is another merge conflict?

Thank you @borissmidt!