Prototype for NonEmptyChunk #3241

ahoy-jon · 2020-03-30T04:29:42Z

Following a discussion on Discord with @adamgfraser
This PR is to add a NonEmptyChunk[A] to ZIO.

A Chunk[A] would be either an Empty or a NonEmptyChunk[A].

Some methods are being overloaded in the trait NonEmptyChunk[A], some are moving to an Ops.

Design decisions :

When possible, return a NonEmptyChunk[A]
All the subtype of Chunk[A], except Empty, extend NonEmptyChunk[A]
Minimize the change to existing code
Complete compatibility with Scala 2.11 is optional (overloaded method are not working well in 2.11)

So far it seems possible! 😃

CLAassistant · 2020-03-30T04:29:49Z

All committers have signed the CLA.

streams/shared/src/main/scala/zio/stream/ZSink.scala

ahoy-jon · 2020-03-30T05:13:33Z

Related issues/PR :

ahoy-jon · 2020-03-31T01:01:20Z

@adamgfraser if you can take a look to check it is going in the right direction!

On my list of things to add :

there are a couple of tests to add (reduce, flatMap, ... )
performance tests

core/shared/src/main/scala/zio/package.scala

core/shared/src/main/scala/zio/ZIO.scala

streams-tests/jvm/src/test/scala/zio/stream/ChunkSpec.scala

streams/shared/src/main/scala/zio/stream/ZSink.scala

core/shared/src/main/scala/zio/Chunk.scala

adamgfraser

Going in the right direction! Left a few more specific comments. Overall it looked like there were a couple of implementation changes that may not be necessary and then would go through and see if we can eliminate some of the uses of asInstanceOf. But if the performance checks out this is going to be a nice addition!

jdegoes · 2020-03-31T18:41:53Z

Organization-wise: think this should be Chunk.NonEmpty to mirror Chunk.Empty.

core/shared/src/main/scala/zio/Chunk.scala

ahoy-jon · 2020-04-01T03:33:48Z

@jdegoes @adamgfraser Thanks you for the code review!

I have a lot to clean before the next review.

0 asInstanceOf
reorder the methods in NonEmpty
...

As for the performance, it is similar to the previous version (thanks @Vilkina for quick help).

Old Version

[info] Benchmark                     (size)  Mode  Cnt      Score      Error  Units
[info] MixedChunkBenchmarks.find       1000  avgt    5     64.854 ±    0.452  ns/op
[info] MixedChunkBenchmarks.flatMap    1000  avgt    5  80139.933 ± 3931.659  ns/op
[info] MixedChunkBenchmarks.fold       1000  avgt    5  22705.831 ± 1053.015  ns/op
[info] MixedChunkBenchmarks.foldM      1000  avgt    5  31908.680 ± 3762.721  ns/op
[info] MixedChunkBenchmarks.map        1000  avgt    5  26212.104 ± 2510.529  ns/op
[info] MixedChunkBenchmarks.mapM       1000  avgt    5  28448.603 ± 3103.181  ns/op
zio/master

New Version

[info] Benchmark                     (size)  Mode  Cnt      Score      Error  Units
[info] MixedChunkBenchmarks.find       1000  avgt    5     61.773 ±    0.425  ns/op
[info] MixedChunkBenchmarks.flatMap    1000  avgt    5  80019.958 ±  201.439  ns/op
[info] MixedChunkBenchmarks.fold       1000  avgt    5  24802.410 ± 2749.465  ns/op
[info] MixedChunkBenchmarks.foldM      1000  avgt    5  29257.296 ± 1952.077  ns/op
[info] MixedChunkBenchmarks.map        1000  avgt    5  24998.218 ±  865.577  ns/op
[info] MixedChunkBenchmarks.mapM       1000  avgt    5  29055.468 ± 1118.978  ns/op
nonEmptyChunk

I don't find the difference conclusive, so more benchmarks!

* specialize if possible all the methods, duplicate when needed. * distribute the implementations between Empty, NonEmpty. * update ZIO, IO, Task, UIO, URIO, RIO with aliases.

bernit77 · 2020-04-02T10:54:00Z

working with @ahoy-jon , here are the performance test on my desktop
with the command ./sbt 'benchmarks/jmh:run -i 15 -wi 15 -f1 -t2 ".*ChunkBenchmarks.*"'

nonEmptyChunk

[info] Benchmark                     (size)  Mode  Cnt       Score      Error  Units
[info] MixedChunkBenchmarks.find       1000  avgt   15      81,038 ±    0,483  ns/op
[info] MixedChunkBenchmarks.flatMap    1000  avgt   15  110688,064 ± 1006,285  ns/op
[info] MixedChunkBenchmarks.fold       1000  avgt   15   36156,638 ±  178,179  ns/op
[info] MixedChunkBenchmarks.foldM      1000  avgt   15   38390,781 ±  242,665  ns/op
[info] MixedChunkBenchmarks.map        1000  avgt   15   36968,630 ±  247,083  ns/op
[info] MixedChunkBenchmarks.mapM       1000  avgt   15   38547,848 ±  195,323  ns/op
[success] Total time: 1885 s (31:25), completed 2 avr. 2020 11:09:29
ahoy-jon/master

master

[info] Benchmark                     (size)  Mode  Cnt       Score      Error  Units
[info] MixedChunkBenchmarks.find       1000  avgt   15      81,806 ±    1,835  ns/op
[info] MixedChunkBenchmarks.flatMap    1000  avgt   15  114643,764 ± 1404,866  ns/op
[info] MixedChunkBenchmarks.fold       1000  avgt   15   27936,776 ±  318,577  ns/op
[info] MixedChunkBenchmarks.foldM      1000  avgt   15   34992,621 ±  274,165  ns/op
[info] MixedChunkBenchmarks.map        1000  avgt   15   34528,973 ±  173,703  ns/op
[info] MixedChunkBenchmarks.mapM       1000  avgt   15   37150,904 ±  246,418  ns/op
[success] Total time: 1865 s (31:05), completed 2 avr. 2020 11:45:50
zio/ef148c12

ahoy-jon · 2020-04-02T12:23:45Z

Thanks!

I will take a look at fold.

ahoy-jon · 2020-04-02T13:18:16Z

I remove the code around slice, that could be a part of the effort in #3238

The code bring improvements but is at risk of raising StackOverflow errors with the current concat unbalancing.

💡Maybe we should upgrade Arr to integrate Slice, like:

private final case class Arr[A](private val array: Array[A], offset: Int, l: Int) extends NonEmpty[A] {

with the dispatch on slice with concat (39 nodes)

[info] Concat
[info] | Concat
[info] | | Concat
[info] | | | Concat
[info] | | | | Concat
[info] | | | | | Concat
[info] | | | | | | Concat
[info] | | | | | | | Slice 0 150
[info] | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | | Concat
[info] | | | | | | | | Concat
[info] | | | | | | | | | Slice 150 50
[info] | | | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | | | | Concat
[info] | | | | | | | | | | Slice 200 50
[info] | | | | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | | | | | Slice 250 150
[info] | | | | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | | | Slice 400 50
[info] | | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | Slice 450 50
[info] | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | Concat
[info] | | | | | | Slice 500 100
[info] | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | Slice 600 150
[info] | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | Concat
[info] | | | | | Slice 750 50
[info] | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | Slice 800 50
[info] | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | Singleton(802)
[info] | | Concat
[info] | | | Slice 752 48
[info] | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | Slice 800 50
[info] | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | Singleton(1000)

without (64 nodes)

[info] Concat
[info] | Concat
[info] | | Concat
[info] | | | Concat
[info] | | | | Concat
[info] | | | | | Concat
[info] | | | | | | Concat
[info] | | | | | | | Slice 0 150
[info] | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | | Concat
[info] | | | | | | | | Concat
[info] | | | | | | | | | Slice 150 50
[info] | | | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | | | | Concat
[info] | | | | | | | | | | Slice 200 50
[info] | | | | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | | | | | Slice 250 150
[info] | | | | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | | | Slice 0 50
[info] | | | | | | | | | Concat
[info] | | | | | | | | | | Slice 400 100
[info] | | | | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | | | | | Slice 500 100
[info] | | | | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | Slice 50 50
[info] | | | | | | | Concat
[info] | | | | | | | | Slice 400 100
[info] | | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | | | Slice 500 100
[info] | | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | Concat
[info] | | | | | | Slice 100 100
[info] | | | | | | | Concat
[info] | | | | | | | | Slice 400 100
[info] | | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | | | Slice 500 100
[info] | | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | Slice 0 150
[info] | | | | | | | Concat
[info] | | | | | | | | Slice 600 150
[info] | | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | | | Slice 750 50
[info] | | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | Concat
[info] | | | | | Slice 150 50
[info] | | | | | | Concat
[info] | | | | | | | Slice 600 150
[info] | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | | Slice 750 50
[info] | | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | Slice 800 50
[info] | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | Singleton(802)
[info] | | Slice 2 98
[info] | | | Concat
[info] | | | | Slice 150 50
[info] | | | | | Concat
[info] | | | | | | Slice 600 150
[info] | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | | | Slice 750 50
[info] | | | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | | | | Slice 800 50
[info] | | | | | Arr(1,2,3,4,5,6,7,8,9,10,11,12 ...
[info] | Singleton(1000)

(some code for the tree)

  final def tree[A](chunk: Chunk[A]): String = {

    def tree(chunk: Chunk[A], depth: Int): NonEmpty[String] = {
      val prefix = "| " * depth
      chunk match {
        case Concat(left, right)  => single(prefix + "Concat") ++ tree(left, depth + 1) ++ tree(right, depth + 1)
        case Slice(ch, offset, l) => single(prefix + s"Slice $offset $l") ++ tree(ch, depth + 1)
        case _ =>
          val str = {
            val str = chunk.toString
            val l   = 30
            if (str.length < l) str else str.take(l) + " ..."
          }
          single(prefix + str)
      }
    }

    tree(chunk, 0).mkString("\n")
  }

bernit77 · 2020-04-02T15:33:52Z

With the new version and :

sbt 'benchmarks/jmh:run -i 15 -wi 15 -f1 -t 2 “zio.chunks.*”'

[info] Benchmark                         (size)  Mode  Cnt      Score      Error  Units
[info] ArrayBenchmarks.find                1000  avgt   15     26,942 ±    0,818  ns/op
[info] ArrayBenchmarks.findOptimized       1000  avgt   15      5,427 ±    0,262  ns/op
[info] ArrayBenchmarks.flatMap             1000  avgt   15  22083,641 ±  208,726  ns/op
[info] ArrayBenchmarks.flatMapOptimized    1000  avgt   15  10755,606 ±  384,065  ns/op
[info] ArrayBenchmarks.fold                1000  avgt   15   9594,572 ±  297,517  ns/op
[info] ArrayBenchmarks.foldOptimized       1000  avgt   15    268,918 ±   18,812  ns/op
[info] ArrayBenchmarks.map                 1000  avgt   15  12291,378 ±   44,157  ns/op
[info] ArrayBenchmarks.mapOptimized        1000  avgt   15    608,422 ±    8,144  ns/op
[info] ChunkArrayBenchmarks.find           1000  avgt   15     26,264 ±    0,451  ns/op
[info] ChunkArrayBenchmarks.flatMap        1000  avgt   15  81095,690 ± 1168,859  ns/op
[info] ChunkArrayBenchmarks.fold           1000  avgt   15   6769,394 ±   20,036  ns/op
[info] ChunkArrayBenchmarks.foldM          1000  avgt   15  16551,973 ±   65,901  ns/op
[info] ChunkArrayBenchmarks.map            1000  avgt   15   6748,881 ±   24,788  ns/op
[info] ChunkArrayBenchmarks.mapM           1000  avgt   15  14987,576 ± 3682,394  ns/op
[info] MixedChunkBenchmarks.find           1000  avgt   15     53,163 ±    2,275  ns/op
[info] MixedChunkBenchmarks.flatMap        1000  avgt   15  81339,852 ± 1468,492  ns/op
[info] MixedChunkBenchmarks.fold           1000  avgt   15  26671,136 ±  966,332  ns/op
[info] MixedChunkBenchmarks.foldM          1000  avgt   15  33207,425 ± 1557,041  ns/op
[info] MixedChunkBenchmarks.map            1000  avgt   15  26464,614 ± 1327,751  ns/op
[info] MixedChunkBenchmarks.mapM           1000  avgt   15  31967,890 ± 1931,565  ns/op
[success] Total time: 6060 s (01:41:00), completed 2 avr. 2020 17:20:05
ahoy-jon/master

adamgfraser · 2020-04-02T21:31:18Z

Does supporting nonempty chunks make this worth? My suggestion would be that we focus this PR on adding nonempty chunks to the API and make as few changes to the underlying implementation as possible and then we can pursue optimizations in separate PRs.

ahoy-jon · 2020-04-02T22:38:43Z

on more (from @bernit77 )

info] Benchmark                         (size)  Mode  Cnt      Score     Error  Units
[info] ArrayBenchmarks.find                1000  avgt   15     26,443 ±   0,272  ns/op
[info] ArrayBenchmarks.findOptimized       1000  avgt   15      5,510 ±   0,095  ns/op
[info] ArrayBenchmarks.flatMap             1000  avgt   15  22281,623 ± 469,810  ns/op
[info] ArrayBenchmarks.flatMapOptimized    1000  avgt   15   9977,730 ± 290,703  ns/op
[info] ArrayBenchmarks.fold                1000  avgt   15   8852,843 ±  28,939  ns/op
[info] ArrayBenchmarks.foldOptimized       1000  avgt   15    235,250 ±   0,378  ns/op
[info] ArrayBenchmarks.map                 1000  avgt   15   6710,605 ±  23,257  ns/op
[info] ArrayBenchmarks.mapOptimized        1000  avgt   15    398,848 ±   0,503  ns/op
[info] ChunkArrayBenchmarks.find           1000  avgt   15     14,331 ±   0,076  ns/op
[info] ChunkArrayBenchmarks.flatMap        1000  avgt   15  43430,912 ± 294,322  ns/op
[info] ChunkArrayBenchmarks.fold           1000  avgt   15   4136,066 ±  15,416  ns/op
[info] ChunkArrayBenchmarks.foldM          1000  avgt   15   8944,188 ±  57,128  ns/op
[info] ChunkArrayBenchmarks.map            1000  avgt   15   4341,903 ±  22,704  ns/op
[info] ChunkArrayBenchmarks.mapM           1000  avgt   15  11449,996 ± 152,716  ns/op
[info] MixedChunkBenchmarks.find           1000  avgt   15     46,374 ±   0,189  ns/op
[info] MixedChunkBenchmarks.flatMap        1000  avgt   15  63203,012 ± 307,207  ns/op
[info] MixedChunkBenchmarks.fold           1000  avgt   15  20928,559 ± 420,732  ns/op
[info] MixedChunkBenchmarks.foldM          1000  avgt   15  25200,845 ± 132,856  ns/op
[info] MixedChunkBenchmarks.map            1000  avgt   15  22759,742 ± 187,125  ns/op
[info] MixedChunkBenchmarks.mapM           1000  avgt   15  26708,820 ±  98,812  ns/op
[success] Total time: 6024 s (01:40:24), completed 2 avr. 2020 19:24:07

ahoy-jon · 2020-04-02T22:55:32Z

Sure, I removed most of the not necessary changes, the rest will come to another PR to have a better +, ++, slice.

If you see anything not needed (like this), don't hesitate to review it.

Some of the array initializations have been changed :

before

var dest = null.asInstanceOf[Array[B]]
var i = 0
...
 if(dest == null) {
  implicit val B: ClassTag[B] = Chunk.Tags.fromValue(b)
  dest    = Array.ofDim[B](len)
}

after

val b     = f(first)
implicit val B: ClassTag[B] = Chunk.Tags.fromValue(b)
val dest    = Array.ofDim[B](len)

dest(0) = b
var i = 1

That can be changed back.

adamgfraser · 2020-04-02T22:59:06Z

Sounds good. Will take a look tonight.

adamgfraser

Looks great! Couple of minor comments and would be good to add some tests that operators retain the subtype information. Then I think this will be ready to go.

streams-tests/jvm/src/test/scala/zio/stream/ChunkSpec.scala

core/shared/src/main/scala/zio/Chunk.scala

streams-tests/jvm/src/test/scala/zio/stream/ChunkSpec.scala

core/shared/src/main/scala/zio/Chunk.scala

adamgfraser · 2020-04-04T16:56:02Z

Looks great! Can you fix the warning and resolve conflicts so we can merge?

ahoy-jon · 2020-04-04T18:35:43Z

Sure, I will do that as soon as possible!

…

On Sat, 4 Apr 2020 at 18:56, Adam Fraser ***@***.***> wrote: Looks great! Can you fix the warning and resolve conflicts so we can merge? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3241 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAUKL3FKCDLV66WKCSMINTRK5RC7ANCNFSM4LWITWYQ> .

adamgfraser

Would be good to figure out what we can do about concatenation. I have an idea there, but let's get this on so we can build on it.

adamgfraser · 2020-04-05T19:45:01Z

Hmmm, my idea for ++ didn't work. Maybe we should add another combinator like concatNonEmpty for that use case?

ahoy-jon · 2020-04-06T09:29:41Z

Yes! A couple of potential names :

concatNonEmpty
appendNonEmpty
++!
...
What is your choice? 😄

jdegoes · 2020-04-06T20:31:11Z

@adamgfraser Did you try:

trait Chunk[+A] {
  type Concat[A1 >: A] <: Chunk[A1]

  def ++ [A1 >: A](that: Chunk[A]): Concat[A1]
}
...
class NonEmpty[+A] {
  type Concat[A1 >: A] = NonEmpty[A1]
}

That gets you non empty ++ chunk, and to get chunk ++ non empty, you can probably rely on overloading since you will get the more specific overload (in theory).

ahoy-jon · 2020-04-07T02:03:51Z

@jdegoes Just tried it, look promising but didn't work yet. I will update Sink a bit and check if there are other parts that don't pass.

[error] /Users/jon/Projects/zio/streams/shared/src/main/scala/zio/stream/ZSink.scala:671:86: type mismatch;
[error]  found   : zio.NonEmptyChunk[A1]
[error]     (which expands to)  zio.Chunk.NonEmpty[A1]
[error]  required: zio.NonEmptyChunk[A00]
[error]     (which expands to)  zio.Chunk.NonEmpty[A00]
[error]             { case (c, leftover) => UIO.succeedNow(Right((c, leftover ++ Chunk.single(a)))) }

diff --git a/core/shared/src/main/scala/zio/Chunk.scala b/core/shared/src/main/scala/zio/Chunk.scala
index c63fbcdd..bb1822bb 100644
--- a/core/shared/src/main/scala/zio/Chunk.scala
+++ b/core/shared/src/main/scala/zio/Chunk.scala
@@ -19,7 +19,8 @@ package zio
 import java.nio._
 
 import scala.collection.mutable.Builder
-import scala.reflect.{ classTag, ClassTag }
+import scala.language.higherKinds
+import scala.reflect.{ClassTag, classTag}
 
 /**
  * A `Chunk[A]` represents a chunk of values of type `A`. Chunks are designed
@@ -38,10 +39,13 @@ sealed trait Chunk[+A] { self =>
    */
   val length: Int
 
+
+  type Concat[A1] <: Chunk[A1]
   /**
    * Returns the concatenation of this chunk with the specified chunk.
    */
-  def ++[A1 >: A](that: Chunk[A1]): Chunk[A1] = Chunk.concat(self, that)
+  def ++[A1 >: A](that: Chunk[A1]): Concat[A1]
+  final def ++[A1 >: A](that: NonEmptyChunk[A1]):NonEmptyChunk[A1] = Chunk.concat(self,that)
 
   /**
    * Appends an element to the chunk
@@ -1205,10 +1209,18 @@ object Chunk {
      * Zips this chunk with the specified chunk using the specified combiner.
      */
     override def zipWith[B, C](that: Chunk[B])(f: (Nothing, B) => C): Chunk[C] = Empty
+
+    override type Concat[A1] = Chunk[A1]
+
+    /**
+     * Returns the concatenation of this chunk with the specified chunk.
+     */
+    override def ++[A1](that: Chunk[A1]): Chunk[A1] = that
   }
 
   sealed trait NonEmpty[+A] extends Chunk[A] { self =>
 
+
     final def first: A = self(0)
     final def last: A  = self(length - 1)
 
@@ -1223,6 +1235,8 @@ object Chunk {
       res
     }
 
+    override type Concat[A1] = NonEmpty[A1]
+
     /**
      * Returns the concatenation of this chunk with the specified chunk.
      */

ahoy-jon · 2020-04-07T02:17:13Z

There are more errors after on simpler cases:

[error] /Users/jon/Projects/zio/core-tests/jvm/src/test/scala/zio/SerializableSpec.scala:157:35: ambiguous reference to overloaded definition,
[error] both method ++ in trait NonEmpty of type [A1 >: Int](that: zio.Chunk[A1])zio.NonEmptyChunk[A1]
[error] and  method ++ in trait Chunk of type [A1 >: Int](that: zio.NonEmptyChunk[A1])zio.NonEmptyChunk[A1]
[error] match argument types (zio.NonEmptyChunk[Int])
[error]       val chunk = Chunk.single(1) ++ Chunk.single(2)
[error]                                   ^
[error] /Users/jon/Projects/zio/core-tests/shared/src/test/scala/zio/ChunkSpec.scala:11:83: ambiguous reference to overloaded definition,
[error] both method ++ in trait NonEmpty of type [A1 >: Int](that: zio.Chunk[A1])zio.NonEmptyChunk[A1]
[error] and  method ++ in trait Chunk of type [A1 >: Int](that: zio.NonEmptyChunk[A1])zio.NonEmptyChunk[A1]
[error] match argument types (zio.NonEmptyChunk[Int])
[error]         val chunk = Chunk.empty ++ Chunk.fromArray(Array(1, 2)) ++ Chunk(3, 4, 5) ++ Chunk.single(6)
[error]

ahoy-jon · 2020-04-07T02:19:28Z

The initial solution with Ops still compile:

  implicit class ChunkOps[A](val self: Chunk[A]) extends AnyVal {
    def ++[A1 >: A](chunk: Chunk[A1]): Chunk[A1]                         = concat(self, chunk)
    def ++[A1 >: A](nonEmptyChunk: NonEmptyChunk[A1]): NonEmptyChunk[A1] = concat(self, chunk)
  }

But :

it decrease the visibility
don't compile in ZSink without some code change (xxx ++ single(a) => xxx + a)

diff --git a/core/shared/src/main/scala/zio/Chunk.scala b/core/shared/src/main/scala/zio/Chunk.scala
index c63fbcdd..4c4013b0 100644
--- a/core/shared/src/main/scala/zio/Chunk.scala
+++ b/core/shared/src/main/scala/zio/Chunk.scala
@@ -19,6 +19,7 @@ package zio
 import java.nio._
 
 import scala.collection.mutable.Builder
 import scala.reflect.{ classTag, ClassTag }
 
 /**
@@ -38,10 +39,13 @@ sealed trait Chunk[+A] { self =>
    */
   val length: Int
 
-  /**
-   * Returns the concatenation of this chunk with the specified chunk.
-   */
-  def ++[A1 >: A](that: Chunk[A1]): Chunk[A1] = Chunk.concat(self, that)
 
   /**
    * Appends an element to the chunk
@@ -597,6 +601,11 @@ sealed trait Chunk[+A] { self =>
 
 object Chunk {
 
+  implicit class ChunkOps[A](private val self: Chunk[A]) extends AnyVal {
+    def ++[A1 >: A](chunk: Chunk[A1]): Chunk[A1]                         = concat(self, chunk)
+    def ++[A1 >: A](nonEmptyChunk: NonEmptyChunk[A1]): NonEmptyChunk[A1] = concat(self, nonEmptyChunk)
+  }
+
   /**
    * Returns the empty chunk.
    */
@@ -1223,10 +1239,12 @@ object Chunk {
       res
     }

     /**
      * Returns the concatenation of this chunk with the specified chunk.
      */
-    final override def ++[A1 >: A](that: Chunk[A1]): NonEmptyChunk[A1] = Chunk.concat(self, that)
+    final def ++[A1 >: A](that: Chunk[A1]): NonEmptyChunk[A1] = Chunk.concat(self, that)
 
     /**
      * Materializes a chunk into a chunk backed by an array. This method can
diff --git a/streams-tests/jvm/src/test/scala/zio/stream/ChunkSpec.scala b/streams-tests/jvm/src/test/scala/zio/stream/ChunkSpec.scala
index 45be09cc..ac6530d6 100644
--- a/streams-tests/jvm/src/test/scala/zio/stream/ChunkSpec.scala
+++ b/streams-tests/jvm/src/test/scala/zio/stream/ChunkSpec.scala
@@ -307,7 +307,7 @@ object ChunkSpec extends ZIOBaseSpec {
         Chunk.single(x),
         Chunk.succeed(x),
         nonEmptyChunk ++ chunk,
-        //chunk ++ nonEmptyChunk, //won't work
+        chunk ++ nonEmptyChunk,
         nonEmptyChunk.flatMap(i => Chunk(i)),
         nonEmptyChunk.map(identity),
         nonEmptyChunk.zipAllWith(Chunk(0))(l => (l, l), r => (r, r))((l, r) => (l, r)),
diff --git a/streams/shared/src/main/scala/zio/stream/ZSink.scala b/streams/shared/src/main/scala/zio/stream/ZSink.scala
index cb976743..06a1bc73 100644
--- a/streams/shared/src/main/scala/zio/stream/ZSink.scala
+++ b/streams/shared/src/main/scala/zio/stream/ZSink.scala
@@ -658,7 +658,7 @@ trait ZSink[-R, +E, +A0, -A, +B] extends Serializable { self =>
                 if (self.cont(s2)) UIO.succeedNow(Left(s2))
                 else self.extract(s2).map(Right(_))
               },
-            { case (b, leftover) => UIO.succeedNow(Right((b, leftover ++ Chunk.single(a)))) }
+            { case (b, leftover) => UIO.succeedNow(Right((b, leftover + a))) }
           )
 
         val rightStep: ZIO[R1, E1, Either[that.State, (C, Chunk[A00])]] =
@@ -668,7 +668,7 @@ trait ZSink[-R, +E, +A0, -A, +B] extends Serializable { self =>
                 if (that.cont(s2)) UIO.succeedNow(Left(s2))
                 else that.extract(s2).map(Right(_))
               },
-            { case (c, leftover) => UIO.succeedNow(Right((c, leftover ++ Chunk.single(a)))) }
+            { case (c, leftover) => UIO.succeedNow(Right((c, leftover + a))) }
           )
 
         leftStep.zipPar(rightStep)

adamgfraser · 2020-04-07T02:29:27Z

@jdegoes The issue we are actually having is with getting chunk ++ nonEmpty to work.

@ahoy-jon That should work equally well if you move both of those to the trait itself instead of an implicit syntax class so I don't think the first point is an issue. I'm looking at the issue with ZSink now.

ahoy-jon commented Mar 30, 2020

View reviewed changes

streams/shared/src/main/scala/zio/stream/ZSink.scala Outdated Show resolved Hide resolved

ahoy-jon mentioned this pull request Mar 30, 2020

ZIO.Chunk classCast exception #3242

Closed

ahoy-jon changed the title ~~[WIP] Prototype for NEChunk~~ Prototype for NonEmptyChunk Mar 31, 2020