[Messenger] Add a scheduler component #47112

upyx · 2022-07-29T10:38:54Z

Q	A
Branch?	6.3
Bug fix?	no
New feature?	yes
Deprecations?	no
Tickets	no
License	MIT
Doc PR	TBD

Introdution

There is no easy way to schedule periodical tasks. There are few useful tools which I touched:

They are complicated. They doesn't allow to set time with precision in seconds (at least it isn't easy). They require difficult tricks to set not linear periods (like on sunset in Tokyo). ~~They are~~ Part of them inefficient with infrequent tasks because resources are needed on every run (e.g. Kubernetes CronJob creates and destroes an entire container).

Proposal

I use a custom transport of Messenger that generates messages periodically. It's a simple solution to run periodical jobs without external software. It's especially helpful in environments where additional runners are impossible or very painful (like Kubernetes). Configuration is very flexible and accurate, it's possible to configure any rules when to run or not to run.

Compared to crond: there is no need to install something external, precision of schedule in microseconds instead of minutes, it's possible to set any periods.

Simple example

# messenger.yaml
framework:
  messenger:
    transports:
      schedule_default: 'schedule://default'

Few types of messages:

class SomeJob {}
class OtherJob {}

A handlers:

#[AsMessageHandler]
class SomeJobHandler {
    public function __invoke(SomeJob $job) {
        // do job or delegate it to other service
    }
}

#[AsMessageHandler]
class OtherJobHandler {
    public function __invoke(OtherJob $job) {
        // do job or delegate it to other service
    }
}

A schedules are provided by locators. It's possible to create many locators and/or provide many schedules by the same locator:

class ExampleLocator implements ScheduleLocatorInterface
{
    public function get(string $id): ScheduleConfig
    {
        // once after an hour from now
        $deferForAnHour = new \DatePeriod(
            new \DateTimeImmutable('now'),
            new \DateInterval('PT1H'),
            1,
            \DatePeriod::EXCLUDE_START_DATE
        );

        return (new ScheduleConfig())
            ->add(new OnceTrigger(new \DateTimeImmutable()), new WarmUpJob()) // once on every worker's start
            ->add(PeriodicalTrigger::create('P1D', '2022-01-01T03:00:00Z'), new SomeJob()) // every night at 3 a.m. (UTC)
            ->add(PeriodicalTrigger::fromPeriod($deferForAnHour), new OtherJob())
        ;
    }

    public function has(string $id): bool
    {
        return 'default' === $id;
    }
}

To run schedule:

bin/console messenger:consume schedule_default

It's easy to run jobs manually:

#[AsCommand(name: 'some', description: 'Manually runs SomeJob')]
class SomeCommand extends Command {
    public function __construct(private MessageBusInterface $bus) {
    }

    protected function execute(InputInterface $input, OutputInterface $output): int {
        $this->bus->dispatch(new SomeJob());
    }
}

Example with returning a result

class GoodJob {
    public function __construct(public readonly ?LoggerInterface $logger) {
    }
}

#[AsMessageHandler]
class GoodHandler {
    public function __construct(private readonly LoggerInterface $logger) {
    }

    public function __invoke(GoodJob $job){
        // compute $result
        ($job->logger ?? $this->logger)->info('The result is: {result}', ['result' => $result])
    }
}

#[AsCommand(name: 'job', description: 'Manually runs job')]
class SomeCommand extends Command {
    public function __construct(private MessageBusInterface $bus) {
    }

    protected function execute(InputInterface $input, OutputInterface $output): int {
        $this->bus->dispatch(new GoodJob(new ConsoleLogger($output))); // result will be printed in console
    }
}

Configuring

A minimal configuration:

# messenger.yaml
framework:
  messenger:
    transports:
      schedule_default: 'schedule://default'

More complex example:

# messenger.yaml
framework:
  messenger:
    transports:
      schedule_default:
        dsn: 'schedule://default'
        options:
          cache: 'app.cache'
          lock: 'default'

Example HA configuration with redis:

framework:
  cache:
    default_redis_provider: '%env(REDIS_DSN)%'
  lock:
    redis: '%env(REDIS_DSN)%'
  messenger:
    transports:
      schedule_default:
        dsn: 'schedule://default'
        options:
          cache: 'cache.redis'
          lock:
            resource: 'redis'
            ttl: 60
            auto_release: true

Deprecations

None

Implementation

This PR contains an implementation.

ToDo

Guikingone · 2022-07-29T16:40:20Z

Hi 👋🏻

As SchedulerBundle and the related PR has been linked, some informations about the related "solutions" and some assertions that you made:

The existing solutions (#39719, 2, more?) depend on additional software like crond.

Regarding SchedulerBundle (and as the PR is just the discussion that trigger the creation of the bundle), it DOES NOT rely on crond, crond is A way to launch / consume tasks but the main logic behind SchedulerBundle is to be able to consume tasks no matter how you need to, it could be an HTTP call, using the command or just by calling the worker.

They doesn't allow to set time with precision in seconds.

Actually, it depends, talking about SchedulerBundle, you can set an expression via a custom ExpressionFactory, if precision like MonotonicClock is required, it could be extended when needed, the interface is autoconfigured.

They require difficult tricks to set not linear periods (like on sunset in Tokyo). They are inefficient with frequent tasks because resources are needed on every run (e.g. Kubernetes CronJob runs an entire container).

In the end, running the task via the worker of Messenger or via a dedicated worker is quite the same, a ressource is always consumed when you consume a task 🙂

upyx · 2022-07-30T05:12:14Z

Nice to meet you, @Guikingone 👋

I've mentioned SchedulerBundle because it's an awesome tool for cron like schedules, and it's a good for requests&tasks based applications (which are the most widespread... perhaps 🙂 )

Regarding SchedulerBundle (...), it DOES NOT rely on crond, crond is A way to launch / consume tasks ...

But it requires a way to launch tasks! Without running ScheduleRunner::__invoke() it won't work. This PR introduces yet another way to do that with Messenger. It's absolutely legal to use them together 🙂

e.g.

class ScheduleFactory {
    public static function createSchedule(ClockInterface $clock){
        return new \Symfony\Component\Messenger\Bridge\Scheduler\Transport\Schedule(
            $clock,
            PeriodicalJob::infinite('00:00:00', 'PT1M', new EveryMinuteJob()),
        );
    }
}

#[AsMessageHandler]
class EveryMinuteHandler {
    public function __construct(private readonly ScheduleRunner $scheduleRunner) {
    }

    public function __invoke(EveryMinuteJob $job){
        ($this->scheduleRunner)();
    }
}

In the end, running the task via the worker of Messenger or via a dedicated worker is quite the same, a ressource is always consumed when you consume a task 🙂

Yes, it's possible to write some custom worker with while loop. But it isn't necessary. The Messenger's Worker does a loop, and does it very well (with count and memory limits and other nice features).

UPDATE
I linked wrong bundle. https://github.com/Guikingone/SchedulerBundle already has a necessary command.

upyx · 2022-07-30T05:12:27Z

Nevertheless, there is a nuance... Hm... there are few ones 😁

Why is some Schedule/Cron/Jobs/Tasks/Etc bundle is needed when the crond/cronspec/CronJob/cronetc is already exists? Why do not configure a crontab/CloudSpec/Helm/Etc to run tasks?

The answer is: because it is necessary to control the application's tasks in the application itself, not by infrastructure.

The first nuance is the Messenger is in the application already (no needs to launch anything dependent on time).
The second one is the ScheduleTransport does the most of the ALotOfBundles features by less than 500 lines of code.

ChrisRiddell · 2022-08-02T09:46:08Z

I like this idea, been able to keep the application code together, having all "Schedule/Queue" tasks together helps with maintainability and without needing crond would be a very nice touch to have in some container setups.

fabpot · 2022-08-03T06:33:41Z

@upyx I like it a lot :)

upyx · 2022-08-03T15:37:14Z

@Guikingone I was cofused with zenstruck's bundle and yours, sorry for that. A broken link in the issue (#39719) supports doing mistakes like that.
Guikingone/SchedulerBundle has the --wait option on the scheduler:consume command that can be used without any crond or so on. However, the other values are still valid.

@fabpot I'll prepare PR for review in few days.

Guikingone · 2022-08-03T15:43:58Z

@Guikingone I was cofused with zenstruck's bundle and yours, sorry for that. A broken link in the issue (#39719) supports doing mistakes like that.
Guikingone/SchedulerBundle has the --wait option on the scheduler:consume command that can be used without any crond or so on. However, the other values are still valid.

No problem 😅

I tend to have a pretty different vision of why a scheduler worker is different from a message queue one (in particular when it comes to consuming tasks, handling failure, transports, limits, middleware and so on), in the end, the way the linux scheduler handles tasks is not the same as RabbitMQ or even Kafka but that's a complete separate topic from the current one 🙂

upyx · 2022-08-04T18:15:41Z

Ready for feedback ✍️

Few questions:

~~The Bridge directory is not the best place for the component. Where to place it?~~
Where to place the testMessengerScheduler?
~~I can't choose how to implement configuration of schedules: interfaces, attributes, factories?~~
Should I "fix" psalm warnings or can suppress them?
Is it possible to teach fabbot the word "kaz-info-teh"? 🤣

upyx · 2022-08-04T18:58:50Z

@Guikingone

I tend to have a pretty different vision of why a scheduler worker is different from a message queue one ...

I understand what you are saying about. Despite much in common, message processing and task scheduling are completely different things. It matters when we design an application that deals with realtime signals like a shooter game or an audio editor. It matters when we plan a distributed application like Telegram. When the only thing that is needed is to create a new table partition every week, I tend to save a few key presses and some space in the vendor directory.

fabpot · 2022-08-05T05:45:57Z

@upyx Bridges are only helpful when a feature depends on a third-party service/prodiver. Here, that's not the case, so I would move the code to the "core" of the component instead.

fabpot

I still need to play with it a bit more, but I like the simplicity of this approach.
I've made some renaming suggestions.
~~Would it sense to deprecate DelayStamp as it does something similar?~~

src/Symfony/Component/Messenger/Transport/Schedule/JobScheduleInterface.php

src/Symfony/Component/Messenger/Transport/Schedule/PeriodicalJob.php

upyx · 2022-08-12T13:40:32Z

I have used the new Clock component. How to properly add a dependency of that? "symfony/clock": "^6.2"?

fabpot · 2022-08-12T14:31:07Z

I have used the new Clock component. How to properly add a dependency of that? "symfony/clock": "^6.2"?

You should add it as a require-dev dep in the Messenger composer.json file.

src/Symfony/Component/Messenger/Transport/Schedule/PeriodicalMessage.php

src/Symfony/Component/Messenger/Transport/Schedule/ScheduleTransportFactory.php

src/Symfony/Component/Messenger/Transport/Schedule/Schedule.php

src/Symfony/Component/Messenger/Transport/Schedule/ScheduleTransport.php

src/Symfony/Component/Messenger/composer.json

fabpot · 2023-03-15T14:06:23Z

Still one issue we need to fix... WIP

src/Symfony/Component/Scheduler/RecurringMessage.php

src/Symfony/Component/Scheduler/Tests/Messenger/SchedulerTransportTest.php

nicolas-grekas · 2023-03-15T16:01:38Z

src/Symfony/Bundle/FrameworkBundle/Tests/Functional/SchedulerTest.php

+        $this->assertSame([$foo], $fetchMessages(1.0));
+        $this->assertSame([], $fetchMessages(1.0));
+        $this->assertSame([$bar], $fetchMessages(60.0));
+        $this->assertSame([$foo, $bar, $foo, $bar], $fetchMessages(600.0));


Tests are green but this result is suspicious, isn't it?

I'll check. It's funtional, and it's a sequence of "runs".

src/Symfony/Component/Scheduler/Generator/MessageGenerator.php

src/Symfony/Component/Messenger/Transport/TransportFactory.php

src/Symfony/Component/Scheduler/Messenger/SchedulerTransport.php

upyx · 2023-03-16T06:43:08Z

src/Symfony/Component/Scheduler/Messenger/SchedulerTransport.php

+    public function get(): iterable
+    {
+        foreach ($this->messageGenerator->getMessages() as $message) {
+            yield Envelope::wrap($message, [new ScheduledStamp()]);


Was better. All instances of ScheduledStamp are the same, so can be reused.

nicolas-grekas

Let's merge and iterate!

fabpot · 2023-03-16T20:49:07Z

Thank you @upyx.

upyx · 2023-03-21T12:22:24Z

Thank you Fabien!

I'm proud to have been working with you 🙂

carsonbot added Status: Needs Review Feature Messenger RFC RFC = Request For Comments (proposals about features that you want to be discussed) labels Jul 29, 2022

carsonbot added this to the 6.2 milestone Jul 29, 2022

carsonbot changed the title ~~[RFC][Messenger] Simple scheduler~~ [Messenger] Simple scheduler Jul 29, 2022

upyx force-pushed the messenger-scheduler-transport branch 3 times, most recently from 4fb1889 to b65a683 Compare August 4, 2022 17:58

upyx force-pushed the messenger-scheduler-transport branch from b65a683 to 5e9664d Compare August 5, 2022 12:39

fabpot changed the title ~~[Messenger] Simple scheduler~~ [Messenger] Add a scheduler transport Aug 10, 2022

fabpot requested changes Aug 10, 2022

View reviewed changes

carsonbot added Status: Needs Work and removed Status: Needs Review labels Aug 10, 2022

upyx force-pushed the messenger-scheduler-transport branch 2 times, most recently from 31f0f16 to 4be42d9 Compare August 12, 2022 13:29

fabpot requested changes Aug 12, 2022

View reviewed changes

fabpot reviewed Aug 12, 2022

View reviewed changes

src/Symfony/Component/Messenger/composer.json Outdated Show resolved Hide resolved

fabpot force-pushed the messenger-scheduler-transport branch from ca924f7 to 82a2682 Compare March 15, 2023 13:51

fabpot force-pushed the messenger-scheduler-transport branch from 82a2682 to a499641 Compare March 15, 2023 14:19

stof reviewed Mar 15, 2023

View reviewed changes

src/Symfony/Component/Scheduler/RecurringMessage.php Outdated Show resolved Hide resolved

nicolas-grekas reviewed Mar 15, 2023

View reviewed changes

src/Symfony/Component/Scheduler/Tests/Messenger/SchedulerTransportTest.php Outdated Show resolved Hide resolved

nicolas-grekas force-pushed the messenger-scheduler-transport branch 2 times, most recently from ae88cdd to 3b205a7 Compare March 15, 2023 15:59

nicolas-grekas reviewed Mar 15, 2023

View reviewed changes

nicolas-grekas force-pushed the messenger-scheduler-transport branch from 3b205a7 to a054802 Compare March 15, 2023 17:44

nicolas-grekas reviewed Mar 15, 2023

View reviewed changes

src/Symfony/Component/Scheduler/Generator/MessageGenerator.php Show resolved Hide resolved

upyx commented Mar 16, 2023

View reviewed changes

src/Symfony/Component/Messenger/Transport/TransportFactory.php Outdated Show resolved Hide resolved

upyx commented Mar 16, 2023

View reviewed changes

src/Symfony/Component/Scheduler/Messenger/SchedulerTransport.php Outdated Show resolved Hide resolved

upyx commented Mar 16, 2023

View reviewed changes

Add the Scheduler component

6d9311f

fabpot force-pushed the messenger-scheduler-transport branch 3 times, most recently from f1f9a90 to 32cbe20 Compare March 16, 2023 07:13

[Scheduler] Rework the component

a18127b

fabpot force-pushed the messenger-scheduler-transport branch from 32cbe20 to a18127b Compare March 16, 2023 07:19

fabpot mentioned this pull request Mar 16, 2023

[Notifier] Support for scheduled notifications #46738

Closed

nicolas-grekas approved these changes Mar 16, 2023

View reviewed changes

carsonbot added Status: Reviewed and removed Status: Needs Work labels Mar 16, 2023

carsonbot changed the title ~~Add a scheduler component~~ [Messenger] Add a scheduler component Mar 16, 2023

fabpot mentioned this pull request Mar 16, 2023

[Messenger] Add a scheduler component symfony/symfony-docs#18067

Closed

fabpot merged commit b42027b into symfony:6.3 Mar 16, 2023

upyx deleted the messenger-scheduler-transport branch March 26, 2023 15:05

fabpot mentioned this pull request May 1, 2023

Release v6.3.0-BETA1 #50199

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Messenger] Add a scheduler component #47112

[Messenger] Add a scheduler component #47112

upyx commented Jul 29, 2022 •

edited

Loading

Guikingone commented Jul 29, 2022 •

edited

Loading

upyx commented Jul 30, 2022 •

edited

Loading

upyx commented Jul 30, 2022

ChrisRiddell commented Aug 2, 2022

fabpot commented Aug 3, 2022

upyx commented Aug 3, 2022

Guikingone commented Aug 3, 2022

upyx commented Aug 4, 2022 •

edited

Loading

upyx commented Aug 4, 2022

fabpot commented Aug 5, 2022

fabpot left a comment •

edited

Loading

upyx commented Aug 12, 2022

fabpot commented Aug 12, 2022

fabpot commented Mar 15, 2023

nicolas-grekas Mar 15, 2023

upyx Mar 16, 2023

upyx Mar 16, 2023

nicolas-grekas left a comment

fabpot commented Mar 16, 2023

upyx commented Mar 21, 2023

[Messenger] Add a scheduler component #47112

[Messenger] Add a scheduler component #47112

Conversation

upyx commented Jul 29, 2022 • edited Loading

Introdution

Proposal

Simple example

Example with returning a result

Configuring

Deprecations

Implementation

ToDo

Guikingone commented Jul 29, 2022 • edited Loading

upyx commented Jul 30, 2022 • edited Loading

upyx commented Jul 30, 2022

ChrisRiddell commented Aug 2, 2022

fabpot commented Aug 3, 2022

upyx commented Aug 3, 2022

Guikingone commented Aug 3, 2022

upyx commented Aug 4, 2022 • edited Loading

upyx commented Aug 4, 2022

fabpot commented Aug 5, 2022

fabpot left a comment • edited Loading

Choose a reason for hiding this comment

upyx commented Aug 12, 2022

fabpot commented Aug 12, 2022

fabpot commented Mar 15, 2023

nicolas-grekas Mar 15, 2023

Choose a reason for hiding this comment

upyx Mar 16, 2023

Choose a reason for hiding this comment

upyx Mar 16, 2023

Choose a reason for hiding this comment

nicolas-grekas left a comment

Choose a reason for hiding this comment

fabpot commented Mar 16, 2023

upyx commented Mar 21, 2023

upyx commented Jul 29, 2022 •

edited

Loading

Guikingone commented Jul 29, 2022 •

edited

Loading

upyx commented Jul 30, 2022 •

edited

Loading

upyx commented Aug 4, 2022 •

edited

Loading

fabpot left a comment •

edited

Loading