Thank you for this!

I won't be able to properly review this until July 12. I'm just leaving two high-level comments for now:

• There is no need for the tail of a LazyList to be lazy anymore. Indeed, it can be an eager LazyList that happens to be Suspended instead!
• I would highly recommend exploring a design where the subclasses of LazyList are private. The existence of Suspended should be more or less an implementation detail. And you don't want people to try and explicitly pattern-match on Cons/Empty/Suspended, because it will most likely not do what they think it's doing.

As added food for thought: I wonder whether it is still useful that the head be lazy. Are there use cases where one already knows that the lazy list is non-empty, but doesn't know yet what its head will be (and cannot synchronously compute it)? My experience from programming in Oz (where this lazier LazyList is basically the base data type used everywhere--and optimized by the VM) doesn't support any such use case. Combined with my first comment that tail need not be lazy anymore, I expect that LazyList.Cons can be fully eager.

@sjrd It is conceivable to me that there exists some situation where Iterator.hasNext is cheap, but .next() is not; in such a case, it would be valuable to have a lazy head (when creating a LazyList from the iterator). That being said, I can't think of such a situation.

@NthPortal Yes, that is definitely conceivable. But at call site, is it likely that you want to call hasNext and not follow up immediately with next(), in the same synchronous "step"? Because if you always follow hasNext with next(), then it doesn't matter whether the potentially expensive operation is performed during hasNext or during next().

Maybe it still makes sense to have a lazy head so that we can know whether the lazy list is empty without evaluating its next element:

def expensiveList(n: Int): LazyList[Int] =
  LazyList {
    if (n == 0) State.empty
    else State.nonEmpty(heavyComputation(n), expensiveList(n - 1))
  }

In this example, it might be useful in some cases to evaluate the fact that the underlying state is NonEmpty but still defer the computation of its head to a later point in the program?

Your example is again based on the definition site. It does not address what I said #6880 (comment) any more than what has previously been said.

@julienrf: I did originally have a design similar to @sjrd's, but I figured there was too much indirection calling this.state.foo, so this.

it prevents returning an eager LazyList by mistake (like we currently do in Process#lazyLines)

How do you mean? The relevant bit called by lazyLines looks now like

def next(): LazyList[T] = LazyList.suspend(q.take match {
  case Left(0)    => LazyList.empty
  case Left(code) => if (nonzeroException) scala.sys.error("Nonzero exit code: " + code) else LazyList.empty
  case Right(s)   => LazyList.cons(s, next())
})

(previously the suspend call wasn't there, of course). The overeager evaluation here is at the definition side; there's no way to drop the suspend and have a lazy-enough list.

I'm going to take Sébastien's advice and make the subclasses private; if we ensure that the Cons constructor isn't called with a strictly-evaluated tail, then overstrictness won't be a problem.

With reference to @sjrd's point about strictness in the head: iterator is always going to force the heads, but a sufficiently-lazy operation doesn't have to. drop is an easy example. foldLeft and foldRight could avoid evaluating heads, except the parameters to the combining function are strict, so in effect they won't. Personally I prefer the lazy head, but with the current strictness of the collections operations taking advantage of it may wind up being tough to use.

Would lazyFold{Right,Left}s be considered useful specialized operations?

it prevents returning an eager LazyList by mistake

How do you mean?

For instance when we implement transformation operations such as map or filter, we now have to carefully wrap the result in a defer call to not eagerly evaluate the receiver’s state.

@sjrd I guess there can be some algorithms that care about the fact that a lazy list is empty or not, and then defer when they actually evaluate the head to a later point. OK, I have no examples…

@hrhino What do you think about #6880 (comment) ?

@julienrf I think that it's a fair point (and I'm rewriting this to use Sébastien's design), but I think doesn't go far enough on its own; we also need to take care of those SeqOps implementations which don't go through views. If I can make all of the inherited operations that return a LazyList use fromSpecificIterable applied to a view, then I think your point is a good idea. I'll double-check that all the methods can be lazy enough (I don't have the source code with me right now) but there is always the danger that someone adds a new operation or an override in a superclass of LazyList and accidentally makes it strict. I'll put tests in, at least.

@hrhino For methods such as dropRight or groupBy we have no other choice than being strict. And these methods clearly document that they are strict. So I think it’s fine to inherit them in LazyList.

@hrhino Will you have time to continue this work? Do you want any help?

Yes, I'll pick it up this weekend. It's been really busy at work; sorry. I'll shout in the strawman gitter if I get confused. Thanks for the poke.

When's the feature freeze deadline?

The targeted deadline is the 10th of August.

Any update on this @hrhino? If you are too busy maybe you can push your current state somewhere and someone else can finish the work?

Yes, sorry, I may need to do that.

Status summary

We have decided it is best to go with an implementation like the one suggested by @sjrd; specifically, that LazyList should contain a lazy state, so that knowledge of whether or not it is empty does not need to be known at construction time.

With this new internal structure, significantly less is shared with Stream, and it doesn't make sense for them to share a trait with implementation (LazyListOps) anymore. Thus, in order to allow for a more sensible implementation of LazyList, the implementations need to be separated.

Thus, there are two major tasks to be accomplished to make a better LazyList a reality:

1. Modify LazyList to be fully lazy
   • use a lazy internal state, as mentioned above
   • make the implementation of all methods specific to LazyList, instead of inheriting from a shared trait
   • remove the implementation of Stream which inherits from LazyListOps
   • remove LazyListOps
2. Port the implementation of Stream from 2.12 and update it to work with the new collections design. It should still be deprecated in favour of LazyList (though it's possible the deprecation message will need to be tweaked slightly from what it is now).

I am currently working on (1). There are various aspects of the previous Stream implementation which make porting it non-trivial (at least for me), so it would be very helpful if someone more familiar could take care of (2).

Regarding (2), I’m not sure porting (again) the 2.12 implementation to the new design would be simpler than just inlining the LazyListOps inheritance relation in Stream.

@julienrf I'm open to that as well.

I'm closing this in favor of @NthPortal's work. Sorry for the false alarm.

See #7000