cc: @kyri-petrou @jdegoes

cc: @regiskuckaertz,
@ghostdogpr mentioned u in discord as a potential reviewer, I'd appreciate your input on this pr.

@kyri-petrou , this pr fixes dropWhileZIO as well.
notice the current behavior is flawed, feeding an infinite stream to this sink never ends, I suggest you compare that with ZSink.head's behavior as it's a bit easier to follow.

one way to 'wrap your head' about ZSink is considering what would you expect if it operated on element level (instead of chunk level). in this case ZSink.head consumes exactly one element (or none if upstream is empty) and completes. ZSink.dropWhile reads one element, determine if it's still dropping and then either complete without further reading from upstream or otherwise cycles this process again.

Enter chunks, ZSink.head now reads a chunk instead of a just a single element, when this chunk has more than one element it must expose the remaining leftovers in order to support proper ZSink composition, this is achieved by writing the leftovers to downstream before completing.

I think a good reference for sinks behavior is Zsink.fold which many sinks (such as head) are defined in terms of.

I think the part that I'm trying to wrap my head around is whether having ZSinks such as dropWhile even makes sense. In the case of dropWhile there is no success value (it's actually Unit even though the type is set to Any for some reason), and the remainder of the stream becomes a leftover.

As for ZStream#dropWhileZIO, currently it's implemented different than ZStream#dropWhile (one uses a ZSink, the other does not). Shouldn't both implementations use the same underlying mechanism?

@kyri-petrou , the dropWhile and dropWhileZIO sinks are useful when composed with other sinks, like the composition in @oridag 's original use case (combined with ZSink.head).

re. the inconsistency in ZStream's operator implementations, I can easily fix this well. lmk if you think it belongs in this pr.

are you convinced about the validity of the fixes introduced by this pr?

@kyri-petrou @ghostdogpr ,
any chance to finalize this review?

@kyri-petrou , thanks for pulling series/2.x into my branch,
I've looked into the failure and I think it just shows that implementing ZStream.dropWhileXXX inb terms of ZSink.dropWhileXXX is wrong.
I think the following test using ZSink.head and ZStream.pipeThrough (equivalent to the failing ZStream.dropWhileXXX implementation) shows this:

test("pipeThrough(ZSink.head)")(
            check(pureStreamOfInts) {ints =>
              for{
                res1 <- ints.pipeThrough(ZSink.head).runCollect
                res2 <- ints.chunks.filter(_.nonEmpty).run(ZSink.head)
                res3 = res2.toSeq.flatMap(_.drop(1))
              } yield {
                assert(res1)(equalTo(res3))
              }
            }
          )

in short: a sink stops consuming as soon as it can produce a result, at this time it'd emit the leftovers to downstream.
ZStream.pipeThrough runs the sink (once) and emits its leftovers, in the case of ZSink.head this is the tail of the first non empty chunk (or nothing in case the stream is empty).
ZSink.dropWhilXXX use to consume the entire upstream as leftovers which is a bug, it is also the reason the faulty implementation of ZStream.dropWhieXXX passed the tests.

thanks @kyri-petrou for adding this ZStream test show-casing your concern, I'll fix the ZStream.dropWhilXXX operators as well.

@kyri-petrou , btw seems like ZStream.dropWhile is properly implemented in term of ZPipeline.dropWhile, not really sure why ZStrea,dropWhilZIO was implemented in terms of the sink rather than the pipeline.

what's more alarming is that it seems ZSink.dropUntil is also implemented in terms of the pipeline, hence emitting the entire upstream as leftovers... I'll add a fix as part of this pr (assuming acceptable by u).

are you convinced about the validity of the fixes introduced by this pr?

There are things that I still don't understand with ZSink. e.g., why does this happen? Is this valid or not?

ZStream(0, 1, 2, 0, 2, 0, 2)
                .rechunk(3)
                .transduce {
                  ZSink.dropWhile[Int](_ == 0) *> ZSink.head[Int]
                }
                .take(10)
                .runCollect // produces Chunk(Some(1), Some(2), Some(2), Some(2), None)

Why does the 2nd Some(2) exist when we used ZSink.head? If it's valid that it exists, why did the 0 after 1 got removed? It feels like transduce is broken altogether 😕

In either way, the part that worries me the most about these changes is not their validity per se but the fact that it alters the underlying behaviour of ZSink.dropWhle. Effectively its behaviour changes entirely, so users that are currently using it will likely end up with broken code unknowingly. It's even worse if libraries are using it because users will not even be able to change its usage

re. transduce: it applies the sink multiple times until exhausting upstream, in this case u can think about it as if it breaks the stream into these chunks:

[0, 1] --> Some(1)
[2] --> Some(2)
[0, 2] --> Some(2)
[0, 2] --> Some(2)
[] --> None

re. change behavior, existing behavior is a bug. it may manifest itself in multiple ways:

1. non-completing streams
2. memory leaks as result of accumulation.
3. eager 'error consumption'
4. broken sinks composition

most of these won't happen when using sinks the 'normal' way, they appear only when using operators like transduce or ZPipeline.fromSink and alike.

is there a process to test this kind of change at least on the zio ecosystem? major libraries like zio-http, zio-grc, zio-kafka, zio-aws (feel free to add more), perhaps by releasing a side branch and then running test prs on these projects?

is there a process to test this kind of change at least on the zio ecosystem

No, not at the moment at least. I recently proposed to create a nightly build of sorts that tests the latest ZIO snapshot against all the zio-* libraries to ensure that there aren't any regressions. In the meantime, the only option will be to search for usages of dropWhile and dropWhileZIO in the zio org

and another fun fact, seems like ZPipeline.dropUntilZIO has a bug as well, I suspect it happens when the last dropped element is the last one in a chunk.

see this test:

test("dropUntilZIO") {
          check(pureStreamOfInts, Gen.function(Gen.boolean)) { (s, p) =>
            for {
              res1 <- (s >>> ZPipeline.dropUntilZIO(p.andThen(ZIO.succeed(_)))).runCollect
              res2 <- s.runCollect.map(_.dropWhile(!p(_)).drop(1))
            } yield assert(res1)(equalTo(res2))
          }
        }

fails with:

✗ Empty$[0] : expected '447334053' got 'null'

      res1 did not satisfy equalTo(res2)
      res1 = Chunk()

I'll take a stub at this as well

are you convinced about the validity of the fixes introduced by this pr?

There are things that I still don't understand with ZSink. e.g., why does this happen? Is this valid or not?

ZStream(0, 1, 2, 0, 2, 0, 2)
                .rechunk(3)
                .transduce {
                  ZSink.dropWhile[Int](_ == 0) *> ZSink.head[Int]
                }
                .take(10)
                .runCollect // produces Chunk(Some(1), Some(2), Some(2), Some(2), None)

Why does the 2nd Some(2) exist when we used ZSink.head? If it's valid that it exists, why did the 0 after 1 got removed? It feels like transduce is broken altogether 😕

In either way, the part that worries me the most about these changes is not their validity per se but the fact that it alters the underlying behaviour of ZSink.dropWhle. Effectively its behaviour changes entirely, so users that are currently using it will likely end up with broken code unknowingly. It's even worse if libraries are using it because users will not even be able to change its usage

Yes, this is the expected behavior (the trailing None is arguable). transduce takes a sink and runs the stream through it multiple times. Given a stream Stream[A] and a sink Sink[B], stream.transduce(sink) will transform the stream of A into a stream of B. In this case, the sink is composed from two sinks, one that consumes (and discards) all zeros, and then one that takes the first element. Running the stream into this sink multiple times should yield all non-zero elements.

@kyri-petrou , searched for ZSink in major zio org repos, scalapb's zio-grpc, caliban..
seems like ZSink are quite sparsely used, any ways no usage of any of the sinks mentioned in this pr.