@mschuwalow Thanks for taking the lead on this! This is really great work and goes a long way to cleaning up a complicated part of the codebase with a lot of dependencies.

The main concern I have with the current implementation is that it pushes more work onto the user to specify whether a Schedule should do "sleeps" or not. For example, consider this snippet:

ZIO.effectTotal(println("Hello, World!")).repeat(ZSchedule.fixed(1.second))

In the current version of ZIO, this would print "Hello, World!" to the console repeatedly with a one second delay. In this PR, that line would instead immediately print "Hello, World!" an infinite number of times with no delay. To get an actual delay we need to instead do:

ZIO.effectTotal(println("Hello, World!")).repeat(ZSchedule.fixed(1.second).addDelay)

And the user in general has to do that every time they call a Schedule method. We generally can't do it within ZIO library combinators because addDelay adds a clock dependency so is only appropriate if we know that the schedule will always involve "sleeps" and not just for example be repeating while a condition is true. This is not great from a user perspective. Incidentally if we do go down this road I would suggest renaming addDelay to something like run to highlight its nature of adding real Clock effects and being irreversible (we can addDelay to any schedule but we can't do that reverse because at that point we have actual sleep effects that we can't introspect).

I think broadly this PR goes in the right direction by continuing to model a schedule as a "description" of a series of steps with delays rather than in each step doing the delay, at least until the schedule is "run" with addDelay. We generally benefit by describing effects versus actually doing them and this true here as well where if each step does a "sleep" in combinators like spaced we lose the ability to modify those delay in some way because by the time we "see" them they have already occurred, so for example we can add delays but can't reduce them and || would need to behavior more like ZSteam.merge. I think there is a consistent set of combinators there but they are different than the existing one and in some ways more restrictive, so I think you are right to model most combinators as descriptions and only introduce actually doing the "sleeping" as late as possible.

What I think we need to add is using the type system more to automatically "derive" whether a schedule should use Clock and do sleeping, versus the user having to specify that using addDelay. Conceptually, the compiler should "know" that a schedule using Schedule.fixed should do real sleeps and that one only using Schedule.doWhile shouldn't. To do that, every schedule needs to maintain one additional piece of information, which we could think of as the "ExecutionStrategy" for the schedule, which could either be "live" (do sleeping), or not. Then I think the composition rule is that if we combine schedules and either schedule uses "sleep", the combined combinator uses sleep. I think if we take that aspect from #1720 and incorporate it we could get schedules that automatically propagate the appropriate clock dependency, which I think is ultimately where we want to be.

If that is right then I think we conceptually end up at a similar place to #1720 (schedules are descriptions of a series of delays that include an instruction for how a client can "run" those delays). And then I think we have to see how different they. If schedules really are descriptions of a series of delays with a way for a client to effectually "run" them to potentially actually do a "sleep", then we can implement all downstream method with almost no change and remove the Clock dependency as in #1720.

In order to make this easier I split up all time-based schedules into functions that produce non-sleeping schedules that emit durations and a function addDelay that allows to 'materialize' this into sleeps.

I too am not a fan of this. It impacts user-land ergonomics enormously.

What is wrong with just sleeping? Seems like you almost had that working at the hackathon. Run into more problems there???

In order to make this easier I split up all time-based schedules into functions that produce non-sleeping schedules that emit durations and a function addDelay that allows to 'materialize' this into sleeps.

I too am not a fan of this. It impacts user-land ergonomics enormously.

What is wrong with just sleeping? Seems like you almost had that working at the hackathon. Run into more problems there???

I guess the one that is specifically proving challenging is jittered.
I believe having the option to at least randomly extend how long schedules are sleeping is super useful, but it's not quite clear to me how to express this on top of the new api. One option is see is
measuring how long schedules are taking with timed and extending it with a second sleep, but this feels quite dirty.
The second is the approach I've taken in the pr, but I agree that the ergonomics are bad.

@jdegoes Do you have an idea how to express this?

Isn’t the problem more fundamental? Take the example of:

ZSchedule.spaced(10.seconds).modifyDelay(_ / 2)

If a schedule just describes delays, so each step doesn’t actually sleep but returns some state indicating how long to sleep this is easy because we just divide the number by two.

But if a schedule “sleeps” then how do we implement this? We just have an opaque effect that we can either do nothing with or run, in which case we actually sleep for 10 seconds and it is already too late to achieve the goal of sleeping for only 5 seconds. By converting schedules from descriptions to actions we have lost some ability to compose them.

We could instead implement modifyDelay as only adding additional delays and jittered as increasing each delay by a random amount instead of potentially increasing or reducing the delay, but I think that is just one example of how the semantics of schedules that just sleep will be somewhat different than the existing behavior.

Just throwing something out here. What if we add a delay "instruction", represented by an ADT that is parameterized by R? This ADT can be translated to a ZIO but it keeps the R.

Something like ...

sealed trait Something[-R]
object Something {
  case object DoesNothing extends Something[Any]
  case object AddsDelay(delay: Duration) extends Something[Clock]
}

Keep this on the decision, and you can both introspect it and translate it to a ZIO while keeping the R precise. What do you think @mschuwalow @adamgfraser ?

@iravid Yes that is exactly the direction I think we need to go in. Here is what I had in #1720 that does this. Obviously we could change the names.

sealed trait Delay[-R] extends (Duration => ZIO[R, Nothing, Unit]) {
  def combine[R1 <: R](that: Delay[R1]): Delay[R1]
}
final case object Live extends Delay[Clock] {
  def apply(d: Duration): ZIO[Clock, Nothing, Unit] =
    clock.sleep(d)
  def combine[R1 <: Clock](that: Delay[R1]): Delay[R1] =
    this
}
final case object Mock extends Delay[Any] {
  def apply(d: Duration): ZIO[Any, Nothing, Unit] =
    ZIO.unit
  def combine[R1 <: Any](that: Delay[R1]): Delay[R1] =
    that
}

In that version the length of the delay is a separate field in Decision but it could easily be moved into Delay instead.

@adamgfraser It seems with that encoding, you will always end up writing decision.delay(decision.duration). Why not specialise Decision itself?

sealed abstract class Decision[R, A, B](cont: Boolean, state: A, finish: () => B) {
  def delay: ZIO[R, Nothing, Unit] = ZIO.unit
}
final case class Sleep(duration: Duration, cont: Boolean, state: A, finish: () => B) extends Decision[Clock, A, B](cont, state, finish) {
  def delay = clock.sleep(duration)
}
final case class At(instant: Instant, cont: Boolean, state: A, finish: () => B) extends Decision[Clock, A, B](cont, state, finish) {
  def delay = clock.now.map(instant.toEpochMillis - _).flatMap(n => if (n <= 0) ZIO.unit else clock.sleep(n))
}

@regiskuckaertz nice!!

@regiskuckaertz Yes you could definitely do that. You end up not having to write it very often because you do it in a few key combinators like retry and repeat and then from a user perspective it is completely transparent. In my original version I was trying to remove the Clock dependency with minimal other changes so it worked well to leave duration as a field on Decision.

One advantage of this approach is that it should be possible to implement all the methods built on top of Schedule with almost no changes. You should just have to change any references to clock.sleep(duration) to decision.delay.

I really like @regiskuckaertz encoding if we want to go with this style. Especially the At decision would be a very nice way to tackle the cron stuff

Hm maybe we can combine the two approaches. What if we keep the encoding in this pr, which allows nice fold-based usage instead of cont, but introduce an Delayed abstraction very close to what @regiskuckaertz proposed. Something like:

trait ZSchedule0[-R, -A, +B] {
  type State
  type RunningState <: State

  val initial: ZIO[R, Nothing, State]
  val extract: RunningState => B
  val update: (A, State) => ZIO[R, RunningState, Delayed[R, RunningState]]
}

object ZSchedule0 {

  sealed abstract class Delayed[-R, +A] {
    def delay: ZIO[R, Nothing, A]
  }
  final case class Sleep[A](duration: Duration, result: A) extends Delayed[Clock, A] {
    val delay = clock.sleep(duration).as(result)
  }
  final case class At[A](instant: Instant, result: A) extends Delayed[Clock, A] {
    val delay = clock.currentTime(TimeUnit.MILLISECONDS).map(instant.toEpochMilli - _).flatMap(n => if (n <= 0) ZIO.unit else clock.sleep(n.millis)).as(result)
  }
}

This also makes it much clearer that a schedule is not supposed to delay when it short circuits. We could even make this more polymorphic in order to make this more explicit, at the cost of another parameter:

trait ZSchedule0[-R0, -R, -A, +B] {
  type State
  type RunningState <: State

  val initial: ZIO[R0, Nothing, State]
  val extract: RunningState => B
  val update: (A, State) => ZIO[R0, RunningState, Delayed[R, RunningState]]
}

@mschuwalow

I guess the one that is specifically proving challenging is jittered.

If you make jittered only work on schedules with Clock with Random, it becomes trivial.

In general, modifyDelay could work only on Clock with Random.

We have seen in #676 that it is not sufficient to have a Duration. It's a leaky abstraction because it does not permit you to create schedules that run at specific times. The very concept of modifyDelay assumes all delays are relative, not absolute; which is true for some common cases but not other common cases (cron-style).

Essentially, I believe that losing modifyDelay (and friends) is a good thing. We throw away some structure (which has cost, of course!) and get something that is more general, which can satisfy more use cases. Yes, we have fewer operations, but then the data type becomes less constrained, which means we can use it for broader use cases.

@jdegoes Do you have an idea how to express this?

I would press forward with the generic approach. It results in less code, cleaner code, and yes, fewer operations, but a more capable data type. The alternative will be adding even more and highly irregular structure (you can see that emerging in #676).

For jitter in particular, you are right that it is useful. I would make jitter work only on Clock with Random, or you can easily make it work on R >: DefaultEnvironment#Environment <: Clock with Random, i.e. set of all ZIO built-in services, but at least Clock and Random for the jitter. That takes care of the common useful case in a way that's repeatable (we can do the same thing elsewhere, if we want), and we can then sacrifice modifyDelay and such as being too specialized and not compatible with scheduling at absolute date/times.

Alright, I'll update the pr and get back to you

True, you would probably end up special casing behavior for the different sleep styles all the time O.o
I've added this formulation of jittered. It's a bit sad that we can't write this fully generically without macros 😢

There is one further issue I ran into with this encoding.
We currently are unable to extract the output from a schedule without updating once, which is desired behavior. The problem arises when we want to interrupt the schedule, but still want to be able to extract the output from it. This happens whenever we race two schedules and require both of their outputs.

The prime candidates where this is necessary are && and ***.
I don't think it's possible to implement intersection without making the type signatures very unergonomic and Arrow's split at all.

I believe the following encoding works for all we want to do, but does not enforce the correct usage as strongly.

trait ZSchedule[-R, -A, +B] extends Serializable { self =>
  type State
  val initial: ZIO[R, Nothing, State]
  val extract: (A, State) => B
  val update: (A, State) => ZIO[R, Unit, State]
}

Where initial returns the state to be paired with the first element and update($a_1, $state_1) returns state_2 and so on. Failure with () signals that the schedule should stop and extract with the current a. Alternatively we might be able to do this, but I don't know whether this is any better and I haven't tried it out.

trait ZSchedule[-R, -A, +B] extends Serializable { self =>
  type State
  val initial: A => ZIO[R, Nothing, State]
  val extract: State => B
  val update: (A, State) => ZIO[R, Unit, State]
}

What do you think?

/cc @jdegoes

I admit that just by having a cursory look, there are quite a few semantic and signature changes than I feel comfortable with.

Are all of them necessitated by the changes required to support a specific R? Or can we roll back some of the encoding changes and save them for 2.0?

In particular I should note that the initial/update/extract style of usage, seen in ZSink, is very hard to get right across all combinators and constructors.

We could merge #1720, which would remove the Clock dependency with basically no changes to other combinators, and then use this PR as the basis for additional work. That would let us start implementing other combinators on ZIO in terms of Schedule, which we're already seeing more requests for, and potentially alleviate some of the pressure to get this in immediately.

@mschuwalow What limitations on #1720 led to exploring a different encoding?

@iravid @adamgfraser
I believe the main motivation for this is having a better basis to implement more sophisticated behavior on top of schedules (e.g. #676).
I agree though that this is shaping up to be a rather large change and it might make sense to delay this in favor of #1720. Especially as the design space for this seems quite large and we might need a few more iterations

Ok, I think this is at a point where it can be reviewed. I'm still unsure about the specific encoding of schedule, but I believe the current semantics work out nicely for everything we want to do.

I will take a detailed look this PR and the other one this week.

The main goal is allowing schedules that run at absolute times, so we can define compositional cron-like languages; secondary goal would be eliminating Clock requirement from repeat / retry combinators, if they use schedules that do not require Clock; third goal would be simplifying the core Schedule model, if possible.

I think the main goal (cron) is worth pushing in a "big enough" change to get this through (it is a long requested feature); ideally the reworking would result in something not harder to use than the old API.

The changes look good to me and I definitely like the new encoding! update(...).foldM is much nicer than the previous pattern of update, check the cont flag, delay manually...

Great work @mschuwalow!

Thanks for reviewing! I'll address your remarks tomorrow and also resolve conflicts with master

I checked out this branch and can confirm it solves the particular issue I described in #1909. Thanks @mschuwalow! Looking forward to this.

@jdegoes thanks for the review and the kind feedback 🙏
I believe I have addressed most remarks in your review, but there were a few question that arose. I have left the relevant comments unresolved. Would be great if you could take another look -- I'll finish the changes asap after.