Symfony version(s) affected

6.3.4

Description

When using the scheduler with a single daily recurring message we are often seeing days where the scheduler appears to have not run.

Here is the schedule config:

return (new Schedule())
    ->add(RecurringMessage::every('1 day', new OurDailyMessage(), from: '10:30'))
    ->stateful($this->cache)
;

We have configured caching for the scheduler provider using the database, to allow it to catchup if the task was to crash. The servers run in UTC.

We have added some middleware to attempt to give some visibility to scheduler activity, which just logs when each message is dispatched on the schedule.

class SchedulerLoggingMiddleware implements MiddlewareInterface
{
    public function __construct(
        private readonly Connection $connection,
    ) {
    }

    public function handle(Envelope $envelope, StackInterface $stack): Envelope
    {
        if (null !== $envelope->last(ScheduledStamp::class)) {
            $this->connection->insert('scheduler_log', [
                'date_created' => new \DateTimeImmutable(),
                'message' => $envelope->getMessage()::class,
            ], [
                Types::DATETIME_IMMUTABLE,
                \PDO::PARAM_STR,
            ]);
        }

        return $stack->next()->handle($envelope, $stack);
    }
}

This is then run in production alongside our other Messenger transports (eg. async another_transport) with an ECS task.

bin/console messenger:consume scheduler_default async another_transport etc --limit=1000 --time-limit=3600 

As you can see from the logs it appears some days are missing (ie. 10th and 7th in this example).

I know this is a tricky one, as I don't have an exact reproduction, or really know where to start debugging this. I'm hoping in raising an issue perhaps there's something obvious which is incorrectly configured, or someone can indicate an area to look at to attempt to debug or reproduce this.

How to reproduce

As indicated above I do not have a reproduction, running the scheduler locally on shorter timeframes seems to behave as intended, and debugging a daily job is hard so I'm hoping for some pointers before beginning on that.

Possible Solution

Unsure if this is related: #51384

Additional Context

1. Is running the scheduler in the same command as other messenger transports a valid use? I thought the cache should allow the scheduler to catch up if it happens to be busy at that moment... but should the scheduler always be run as the sole transport?