[Messenger] Fix processing batches#62872
Conversation
Are messages really flushed when a worker shuts down? Last time I tried they were just lost 🤔 |
Could you please try again? I've never experienced this behavior. |
|
I observed this behavior after updating my handler so that it respects I can confirm this still happens but since this is what your PR fixes I tried applying its changes to a Symfony 7.4 app but the worker crashes on [$envelope, $transportName, $acked] = $unacks[$batchHandler];with
and I don’t understand why so I cannot say if it’s fixed 😅 |
|
@MatTheCat Are you sure you applied the fix correctly? This error would occur if line 168 wasn't applied ( |
|
Ah indeed missed this change… So now I can confirm messages are indeed processed when the worker shuts down 🎉 Thanks! On the other hand I can also see messages keep being logged as received until they’re handled. E.g. here are my worker container logs with a single message added in the queue: This is less important but maybe there’s something to improve here? |
|
@HypeMC just in case, did you have a look at @MatTheCat's comment? |
4eb07c6 to
014da25
Compare
014da25 to
4c56888
Compare
|
@nicolas-grekas @MatTheCat I believe this should solve the duplicate messages problem: https://github.com/symfony/symfony/compare/014da25e5583b21e1ebb63e0538d70c97aff9447..4c568885099e77eb3d847102a6160f452b0d4d59 |
|
@HypeMC can confirm it works 👍 |
|
Thank you @HypeMC. |
|
After our testing on 7.3.10, we've found that the default behavior is altered where the batch handling is now waiting for the batchsize to be filled before processing the batch. In the case described by @wazum this is unwanted behavior, but for our case that is very much wanted behavior. We have an fluctuating workload which we batch handle to improve efficiency, but when the worker is idle it should handle the message asap. Otherwise it would wait until the batch is filled or the worker is reaching it's timeout limit. This is also the behavior of the batch processing according to the manual: https://symfony.com/doc/current/messenger.html#process-messages-by-batches
I think this should be an opt-in feature, since batch sizes are most of the time random and waiting for a specific amount of messages to be handled can be very unstable and cause for other, unwanted behavior. |
…s nearly immediately - Since symfony/symfony#62872 the default batch handler waits for the batch size to be full before processing. That's problematic because when only a single message is dispatched, it waits a really long time to be processed - Introduced Time-based batch handler. It waits max for 2s and if no message is added to the batch, it procesess it immediately.
…s nearly immediately - Since symfony/symfony#62872 the default batch handler waits for the batch size to be full before processing. That's problematic because when only a single message is dispatched, it waits a really long time to be processed - Introduced Time-based batch handler. It waits max for 2s and if no message is added to the batch, it procesess it immediately.
This PR was merged into the 6.4 branch. Discussion ---------- [Messenger] Revert batch processing fix | Q | A | ------------- | --- | Branch? | 6.4 | Bug fix? | yes | New feature? | no | Deprecations? | no | Issues | Fix #63277 (comment) | License | MIT As mentioned in #63277 (comment), this will be reverted in 6.4 as it changes the existing behavior, and reimplemented in a better way in 8.1. Commits ------- 387394a Revert "bug #62872 [Messenger] Fix processing batches (HypeMC)"
* 6.4: [Console] Fix failing test Revert "bug #62872 [Messenger] Fix processing batches (HypeMC)" [FrameworkBundle] Fix BrowserKitAssertionsTrait compatibility with HttpBrowser Add support for Mailjet SMTP relay X-MJ-TemplateErrorReporting header format to MailjetApiTransport.
* 7.4: [Console] Fix failing test Revert "bug #62872 [Messenger] Fix processing batches (HypeMC)" [FrameworkBundle] Fix BrowserKitAssertionsTrait compatibility with HttpBrowser Add support for Mailjet SMTP relay X-MJ-TemplateErrorReporting header format to MailjetApiTransport.
* 8.0: [Console] Fix failing test Revert "bug #62872 [Messenger] Fix processing batches (HypeMC)" [FrameworkBundle] Fix BrowserKitAssertionsTrait compatibility with HttpBrowser Add support for Mailjet SMTP relay X-MJ-TemplateErrorReporting header format to MailjetApiTransport.
VersionDeleteHandler, AssetPreviewImageHandler and OptimizeImageHandler each process jobs independently with no cross-job aggregation, so there is no benefit to deferring a flush. With symfony/messenger >= v6.4.32 (symfony/symfony#62872), idle-time flushing of partial batches was removed, meaning Acknowledgers can now sit unresolved in memory until the batch threshold is reached. If the worker exits before that point the Acknowledger destructor throws a LogicException. Setting shouldFlush() to return true whenever jobs are pending ensures immediate processing and eliminates the exposure window entirely.
- the original issue is already solved, see symfony/symfony#62872
- the original issue is already solved, see symfony/symfony#62872
- the original issue is already solved, see symfony/symfony#62872
The reason batches can contain a random number of elements, as described in the issue, is that during idle periods (when no new messages arrive) the worker flushes all remaining batches regardless of whether they have reached the batch size limit. When messages are queued one by one, a race condition may occur where the worker checks for new messages before a message is queued and flushes the batch prematurely.
This PR changes the behavior so that the
$forceparameter is respected and batches are flushed only when$forceistrue.This change introduces a downside: if a batch has not reached its size limit and no new messages arrive, the existing messages will not be flushed until the batch is filled or the worker shuts down. This can cause messages to remain "stuck" in a batch indefinitely.
Because of this, I'm not sure whether the original behavior is a bug or intended.