In the 1st commit, I used Chunk.range. But it's not available for dotty/2.13. I tried to change it to Chunk.fromIterable, but then got Stack Overflow from Chunk append. Issue created at #4065

I also ran into this issue at work. The stack points at line 1041 which is unfold, but I am not sure from which operator it came from.

It seems to crash after consuming events from Kafka for about 20 minutes (using zio-kafka cc @iravid ). I suspect it's the Schedule.forever used in ZStream#aggregateAsync which is itself used by zio-kafka.

Ouch!

@ghostdogpr I think we're only using aggregateAsync in Consumer.consumeWith, so potentially you could copy consumeWith to your project and replace the use of aggregateAsync with aggregateAsyncWithin(... Schedule.fixed(1.millis)). It's used for batching commits so that should probably be fine.

@jdegoes @adamgfraser Can you take a look at the fix?

Ah, actually I'm not using consumeWith but subscribeAnd so it must be somewhere else. It's hard to pinpoint, I just know that it happens only on my branch introducing zio-kafka 😄 Will investigate more tomorrow if there is no fix in between.

Fix looks fine assuming it works (it would have led to infinite thunks before).

@tuleism Thank you for the contribution and the test!

One issue here is we're evaluating the by name parameter immediately. What about something like?

  def unfold[A](a: => A)(f: A => A): Schedule[Any, Any, A] = {
    def loop(a: => A): StepFunction[Any, Any, A] = {
      lazy val a0 = a
      (now, _) => ZIO.succeed(Decision.Continue(a0, now, loop(f(a0))))
    }

    Schedule((now, _) => ZIO.succeed(Decision.Continue(a, now, loop(f(a)))))
  }

This resolves the stack overflow issue and preserves the laziness of the parameter.

Also I think we may want to scale back the size of the test. I was able to reproduce with 10,000 elements instead of 1,000,000.

Whoa, just hit this in production! Will upgrade asap. Thanks!