Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Scheduler] Caching does not work for multiple messages in same schedule #51384

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
StanJansen opened this issue Aug 14, 2023 · 4 comments
Closed

[Scheduler] Caching does not work for multiple messages in same schedule #51384

StanJansen opened this issue Aug 14, 2023 · 4 comments
Labels
Bug Scheduler Status: Needs Review

Comments

@StanJansen
Copy link
Contributor

StanJansen commented Aug 14, 2023 •
edited
Loading

Symfony version(s) affected

6.3

Description

When you have multiple messages in the same schedule, only the timestamp of the latest message will be saved in the cache. When your consumer stops (by --time-limit or unexpected) the next run date for the other messages will be calculated again. So when you have recurring messages for 1 minute and 30 minutes, with a restart policy of 60 seconds, the 30 minutes message will never be executed when you do do not pass the from argument.

As a temporary workaround I disabled jitter and set the from argument to a date in the past on 00:00:00, which makes the message be calculated based on that (so always on 00:30, 01:00 etc.). However if the consumer is restarting or down due to a crash at that moment, they will not be catched up when booting up the consumer again as the last-ran-date is not cached. This is why I disabled jitter to reduce that risk to the specific time instead of adding the jitter time to that.

How to reproduce

  • Create a stateful schedule with a recurring message scheduled every minute (message A), and one message every 30 minutes (message B). Do not pass a from date.
  • Dump the next run dates in MessageGenerator.php
  • Start the consumer
  • Assert that message A is scheduled for +1 minute, assert that message B is scheduled for +30 minutes
  • Wait a minute for message A to be executed
  • Restart the consumer
  • Message A is scheduled for +1 minute (relative to the previous datetime), message B is scheduled for +30 minutes again (so ~1 minute later than the last next run date it dumped)

Possible Solution

Edit the caching so it works per message, not per schedule.

Additional Context

I created a PR that got merged in 6.3.2 that fixed the from argument. This fixed the scheduler from executing 30 minute messages every minutely restart for example, but due to this bug that fix now causes those messages to never get executed. So unless you pass a from date, those schedules are completely broken at the moment.
@StanJansen StanJansen added the Bug label Aug 14, 2023
@valtzu
Copy link
Contributor

valtzu commented Sep 2, 2023

I tried stateful schedule for the first time and it seems to have very strange behavior indeed.

        (new Schedule())
            ->stateful($this->cache)
            ->add(RecurringMessage::every(1, new TestMessage('every 1 seconds')))
            ->add(RecurringMessage::every(4, new TestMessage('every 4 seconds')))

When continuing on a cached state:

[18:44:11.000] every 1 seconds
[18:44:12.000] every 1 seconds
[18:44:13.000] every 1 seconds
[18:44:14.000] every 4 seconds
[18:44:14.000] every 1 seconds
[18:44:15.000] every 1 seconds
[18:44:15.000] every 4 seconds
[18:44:16.000] every 1 seconds
[18:44:17.000] every 1 seconds

"every 4 seconds" is one second apart at startup, then it goes back to "normal" though.

Edit the caching so it works per message, not per schedule.

That's one solution, other seems to be

  1. Use null as default value for from
  2. Store from=$now in a checkpoint (either add to the current or create separate checkpoint/cache entry)
  3. Pass from from the checkpoint to all triggers (getNextRunDate($run, $from))
  4. Now you have the possibility to make deterministic calculations in the trigger since $from stays the same between consumer restarts

I did manage to make it work, however, it's a lot of breaking changes and I'm not sure if that's the way to go.

About number 1, I think it should be done regardless. Now it's quite unexpected that messages in a Schedule will have different from value because for each message a new DateTime instance is created which can be demonstrated with the debug command if you add microseconds:

 ------------- ------------------------------------ ---------------------------------- 
  Message       Trigger                              Next Run                          
 ------------- ------------------------------------ ---------------------------------- 
  TestMessage   PeriodicalTrigger: every 2 seconds   2023-09-02 19:02:43.734667+00:00  
  TestMessage   PeriodicalTrigger: every 8 seconds   2023-09-02 19:02:49.734865+00:00  
 ------------- ------------------------------------ ----------------------------------

IMO microseconds in both messages should be the same since I have not defined a from.

@rodnaph rodnaph mentioned this issue Sep 13, 2023
Closed
@valtzu valtzu mentioned this issue Sep 13, 2023
Merged
@kbond
Copy link
Member

kbond commented Sep 14, 2023

@StanJansen, can you confirm if #51651 solves this issue?

fabpot added a commit that referenced this issue Sep 16, 2023
@fabpot
feature #51651 [Scheduler] Fix stateful scheduler (valtzu)
a90eca6 
This PR was merged into the 6.4 branch.

Discussion
----------

[Scheduler] Fix stateful scheduler

| Q             | A
| ------------- | ---
| Branch?       | 6.3
| Bug fix?      | yes
| New feature?  | no
| Deprecations? | no
| Tickets       | Fix #51646, #51384
| License       | MIT

Stateful scheduler seems rather broken at the moment, see #51384 (comment).

Let's fix it by storing the original first run start time, that way it's always possible to recalculate the state.

Catching-up works now:
```
[23:14:11.709710] Worker started
[23:14:11.759318] every 2 seconds
[23:14:11.760291] every 2 seconds
[23:14:11.761257] every 2 seconds
[23:14:11.763244] every 2 seconds
[23:14:12.637054] every 2 seconds
[23:14:14.620595] every 2 seconds
[23:14:16.632170] every 2 seconds
```

Whereas before it would only start on from the current item, possibly skipping previous items like stated in #51646. _(is this a bc break?)_

I will be waiting for input from authors of the related issues.

---

One test is failing because because `getNextRunDate` is called with `2020-02-20T01:59:00+02:00` and then next run date is expected at `2020-02-20T02:09:00+02:00` but we get `2020-02-20T02:00:00+02:00` because that's set as `from`. I don't quite get the logic, I would assume that it is expected to be run immediately on `from` :thinking:

Commits
-------

2d5856b Fix stateful scheduler
@fabpot
Copy link
Member

fabpot commented Sep 16, 2023

This bug has been fixed in 6.4, not 6.3, as we had to introduce a BC break to fix it properly.

@fabpot fabpot closed this as completed Sep 16, 2023
@StanJansen
Copy link
Contributor Author

Sorry for the late response but this indeed seems to fix it 🥳

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Scheduler Status: Needs Review
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@fabpot @kbond @valtzu @StanJansen @carsonbot