👍

What exactly ended up being the issue with the implementation of concat based on ZManaged#switchable that you posted in discord? I though it was a bit nicer than introducing structure

Good question and good to have it written in this PR for posterity.

The previous ++ would do something like this (written minimally on purpose):

def ++(other: => ZStream) = 
  ZStream {
    for {
      switched <- Ref.make(false).toManaged_
      currPull <- Ref.make(...)
      pull = {
        // get the current pull from the Ref, and pull it
        // if it ended and !switched, switch over to `other`
        // otherwise end
      }
    } yield pull
  }

As you can see, it allocates a bunch of resources that are retained for the existence of the stream (self ++ other).

Now, let's consider what happens in self.forever (defined as forever = self ++ forever) - or any stream that concatenates recursively. When we evaluate ++, it allocates these refs and starts pulling self. Once self is done, the ZManaged for other is opened and that one starts being pulled.

But, other is actually another ++, so more Ref instances are allocated! This happens on every switch to the RHS, so every recursion allocates more Ref. That would be fine if we could somehow stop retaining the current Ref, but we can't "exit" the scope of the current ZManaged. This is the root of the problem. Once we're inside a scope, we can't exit it.

So this formulation is heap-unsafe. I tried pretty hard, but I couldn't work around it properly. I think that the way to solve it is to interpret the recursive structure "from outside", before entering the scope that is recursive.

I hope this makes sense!

Ah, I think I understand. But this is a very subtle issue O.o

Thanks for the explanation and nice fix!

Suggestion:

trait Structure[R, E, A] {
  def process: ZManaged[R, E, Pull[R, E, A]]
  def tail: Option[() => Structure[R, E, A]]
}
case class Iterator[R, E, A](process: ZManaged[R, E, Pull[R, E, A]]) extends Structure[R, E, A] {
  final val tail = None
} 
case class Concat[-R, +E, +A](hd: Structure[R, E, A], tl: () => Structure[R, E, A])
        extends Structure[R, E, A] {
  final def process = hd.process
  final def tail = Some(tl)
}

val structure = tl()
switchPull(structure.hd).mapError(Some(_)).tap(currPull.set) *>
  nextPull.set(structure.tl) *> go(doneRef, currPull, nextPull, switchPull)

I'm actually going to avoid pulling more things up into Structure because I think this is going to end up as a mini-algebra, with an additional FlatMap constructor.

Then I'll move the interpreter up from Concat (which is the only composite term currently requiring interpretation) into Structure.

Have you noticed that pattern is coming up all the time with switchable? I wonder if we could refactor without having to keep currPull and nextPull refs around.

Yeah, there's some convenience to be had here: we could have the switchable constructor return something like:

trait Switchable[R, E, A] {
  def switch(m: ZManaged[R, E, A]): ZIO[R, E, A]
  def get: ZIO[R, E, A]
}

So the Ref is embedded in that.

Ah it's actually not that simple. We would have to start requiring an initial resource, or get becomes UIO[Option[A]]. Beyond that, we need to think what happens if switch fails to acquire the resource. What would get return afterwards? That also pushes us towards UIO[Option[A]]. The problem is that sticking an Option in there can be a real pain for calling code.

Good idea to keep all the guts private so we can experiment throughout 1.x.