[Messenger] Add retry and failure transport support to the sync transport - #65902
[Messenger] Add retry and failure transport support to the sync transport#65902nicolas-grekas wants to merge 1 commit into
Conversation
ccdcef9 to
448ffb1
Compare
448ffb1 to
66d95f5
Compare
wachterjohannes
left a comment
There was a problem hiding this comment.
Correct and well-tested, the shared RetryDecider keeps the worker and this in sync on purpose. Verified locally, all green. Details inline. Also: is there a docs PR for this, and given the DispatchAfterCurrentBusMiddleware fix is a real bug independent of the rest, worth splitting it out for backport as you floated in the description?
| // attempts, an unbounded retry would loop in place until the failure goes away | ||
| $shouldRetry = $this->retryStrategy && false !== RetryDecider::decideFromException($e) && $this->retryStrategy->isRetryable($envelope, $e); | ||
|
|
||
| $this->eventDispatcher?->dispatch(new SyncMessageFailedEvent($envelope, $alias, $e, $shouldRetry)); |
There was a problem hiding this comment.
This fires on every failed attempt regardless of $this->retryStrategy/$this->failureSender, and messenger.php wires the dispatcher unconditionally. So the description's "without the flags, nothing changes" isn't quite right: a plain sync:// transport starts dispatching this event after the upgrade. Harmless today since nothing subscribes yet, but worth fixing the claim.
There was a problem hiding this comment.
Right, and it is deliberate. A failed handler is worth reporting whether or not the transport retries, and without the event the only way to observe a synchronous failure is to catch the exception at every call site.
What was wrong is the claim, now fixed in the description: without the flags the handling of a message is unchanged, it is dispatched once and the exception propagates, and the event is what the upgrade adds. Nothing in the framework subscribes to it.
6d9f33b to
45a968d
Compare
|
Both answers, thanks for asking. The No docs pull request yet, for this one or for the rest of the series. The public API is still moving under review, so I am waiting for these to be merged before writing them; the points each one has to carry are listed at the end of its description in the meantime. |
…spatch (nicolas-grekas)
This PR was merged into the 6.4 branch.
Discussion
----------
[Messenger] Drop the messages queued by a failed nested dispatch
| Q | A
| ------------- | ---
| Branch? | 6.4
| Bug fix? | yes
| New feature? | no
| Deprecations? | no
| Issues | -
| License | MIT
A message dispatched with a `DispatchAfterCurrentBusStamp` from inside a handler is queued and dispatched once the root dispatch is over. When the nested dispatch that queued it fails, the queued messages are dispatched anyway, although the work that queued them never completed.
The root dispatch already drops its queue when handling throws, with the reasoning that the queued messages were likely dependent on the work that queued them. The nested path did not, so the two halves of the same middleware disagreed.
```php
public function __invoke(RegisterUser $command): void
{
try {
// this dispatch queues UserRegistered, then throws
$this->bus->dispatch(new CreateProfile($command->getUuid()));
} catch (\Throwable) {
// the caller decides to carry on
}
// UserRegistered was dispatched anyway, for a profile that was never created
}
```
Found while working on #65902, which turns each attempt of a synchronous retry into a nested dispatch and therefore meets this on every failed attempt. It is a bug of its own, so it is proposed here separately for the lowest maintained branch.
## Checks
- `./phpunit src/Symfony/Component/Messenger/Tests` on 6.4: 425 tests, green.
- `testMessagesQueuedByAFailedNestedDispatchAreDropped` errors without the fix, because the event queued by the failed nested dispatch reaches the handling middleware a third time.
Commits
-------
c0ccae2 [Messenger] Drop the messages queued by a failed nested dispatch
…port A message handled through the sync transport had no retry and no dead letter: an exception was the only outcome, and the retry_strategy and failure_transport configured for the transport were ignored. Both features are now opt-in through the transport DSN or options. "sync://?retry=true" handles a failed message again, immediately and as many times as the retry strategy of the transport allows. "sync://?failure_transport=true" sends a message that still fails to the failure transport of the transport instead of throwing. The delays of the retry strategy are not honored, since the transport cannot wait inside the calling process, and a forced retry is bounded by the strategy for the same reason. SyncMessageFailedEvent and SyncMessageRetryingEvent are dispatched by the transport for each failed attempt and before each new attempt, so that listeners monitoring failures see synchronously handled messages too. The worker events are not reused because their listeners (retry, failure transport, service reset) would react to them. The rules deciding whether an exception can be retried move from SendFailedMessageForRetryListener to the internal RetryDecider class, shared with the transport. MessengerBundle passes the retry strategy locator, the failure senders locator, the event dispatcher and the logger to the sync transport factory.
45a968d to
51ddf38
Compare
This is a proposal, open for discussion. It started from Dariusz Gafka's article Symfony Messenger vs Ecotone: The Real Difference, which describes how Ecotone approaches this. What is proposed here is a free interpretation for Symfony, built on Messenger's own model rather than ported from Ecotone, so it departs from the article where the two models differ. The examples in this description are the article's.
Retries and the failure transport only exist in the worker's event listeners. A message handled synchronously through the
sync://transport, a payment webhook handled in the request for instance, has no retry and no dead letter: an exception is the only outcome, and the per-transportretry_strategyandfailure_transportconfiguration is silently ignored. This PR makes both available to synchronous handling, on an opt-in basis:retry=true: a failing message is handled again right away, as many times as the transport's retry strategy allows. Delays are not honored: the transport runs in the calling process and does not wait between attempts.UnrecoverableExceptionInterfaceis never retried, and a forced retry (RecoverableExceptionInterface) is bounded by the strategy too, since an unbounded retry without a wait would loop in place. Handlers that already succeeded are not run again.failure_transport=true: a message that still fails is sent to the transport's failure transport (or the global one) instead of throwing;dispatch()returns an envelope carryingSentToFailureTransportStamp,RedeliveryStampandErrorDetailsStamp, and the message shows up inmessenger:failed:showlike any other.This is the handling-phase half of what #64628 asks for webhooks: route the webhook messages to such a transport and they get retries and a dead letter without a queue. Parse failures happen before any message exists and stay out of scope.
Both options can also be given in the transport
optionsarray, which wins over the DSN query. Enabling a flag without a retry strategy or a failure transport configured for that transport fails when the transport is created.Without the flags, the handling of a message is unchanged: it is dispatched once and the exception propagates. The one thing that happens for every
sync://transport, flags or not, is the newSyncMessageFailedEventbelow, which reports a failure that was previously observable only by catching the exception at the call site.Public API
SyncTransport::__construct(MessageBusInterface $messageBus, ?RetryStrategyInterface $retryStrategy = null, ?SenderInterface $failureSender = null, ?EventDispatcherInterface $eventDispatcher = null, ?LoggerInterface $logger = null)SyncTransportFactory::__construct(MessageBusInterface $messageBus, ?ContainerInterface $retryStrategyLocator = null, ?ContainerInterface $failureSenderLocator = null, ?EventDispatcherInterface $eventDispatcher = null, ?LoggerInterface $logger = null), both locators keyed by transport name, and theretryandfailure_transportboolean options.Symfony\Component\Messenger\Event\SyncMessageFailedEventandSyncMessageRetryingEvent(see below).SendFailedMessageForRetryListenermoved to an@internalRetryDeciderhelper so the worker and the sync transport cannot diverge.Events
Synchronous handling now has the same failure and retry signals the worker has, for monitoring listeners:
SyncMessageFailedEvent(getEnvelope(),getTransportName(),getThrowable(),willRetry()) is dispatched for every failed attempt, after the retry decision and before it is acted on.SyncMessageRetryingEvent(getEnvelope(),getTransportName()) is dispatched right before the next attempt, with the envelope carrying the newRedeliveryStamp.Together with the per-handler
HandlerStartingEvent,HandlerSuccessEventandHandlerFailureEventalready dispatched on 8.2 whatever the transport, this covers #57623. The worker events keep theirWorkerprefix on purpose: their listeners (retry, failure transport, service reset) act on them, so reusing them for synchronous handling would double-process the message.Prerequisite, already merged
A failed attempt runs as a nested dispatch inside the calling request, so the messages it queued with
DispatchAfterCurrentBusStampwould still be dispatched after a later successful attempt, or after the message went to the failure transport. That rollback was split out of this pull request and merged on its own as #66032, so it is no longer part of this diff.Checks
./phpunit src/Symfony/Component/Messenger/Testsand./phpunit src/Symfony/Bundle/FrameworkBundle/Tests: green (usual missing-server skips). The existingSendFailedMessageForRetryListenerTestis unchanged and green.sync://?retry=true&failure_transport=truetransport with an in-memory failure transport) fails without the wiring.Documentation
max_retries(or a customisRetryable()) bounds them; delays, multiplier, max delay and jitter are ignored; forced retries stay bounded.messenger:failed:showandmessenger:failed:retryapply.DispatchAfterCurrentBusStampduring a failed attempt are dropped ([Messenger] Drop the messages queued by a failed nested dispatch #66032).WorkerMessageFailedEventneedSyncMessageFailedEventtoo to see synchronous handling.