Improvement of message consumption for messenger with AMQP #30454
Conversation
…onsumer to continue processing other messages without stopping * Added `UnrecoverableMessageExceptionInterface` and `RecoverableMessageExceptionInterface` into AMQP transport into exception to allow handler to nack messages with or without requeue * Standardized AMQP rejection with `nack` (`reject` is a `nack` without multiple rejections abilities which is not used in Messenger context).
consume_fatal and consume_requeue to allow c…| throw $e; | ||
| if ($connectionCredentials['consume_fatal']) { | ||
| throw $e; | ||
| } |
There was a problem hiding this comment.
I'm very interested in this part, and these two new options. I'd like to make sure we're thinking about a solution that could be generically applied to other transports and support a "retry" ability. Here's the behavior I'd like to see, which is built off of your ideas:
- If
RecoverableMessageExceptionInterfaceis thrown, we re-queue - If
UnrecoverableMessageExceptionInterfaceis thrown, we discard - If there is an
AMQPExceptionat any point, probably do nothing - because the message was neverack'ed, so it should still be on the queue? But, it should not kill the worker process (but that's another PR) - If any other
Throwableoccurs, I think we should requeue always and never throw the exception. However, a few caveats:
- it should only requeue the message a certain number of times before discarding it (that's the retry stuff)
- once it hits the retry max, the exception is still not thrown, but perhaps an event is dispatched to handle this.
Morever, I think one of your goals was to standardize a few things (even though we don't have multiple transports yet), which I love! Which is why I'm wondering if we should move some of this logic to the Worker class - #30557 is related. I'm proposing that this method should not handle any exceptions or requeue - it should just handle the message and try to ack it on success. We would move the exception-handling logic into Worker (so we can standardize behavior) and add a few new methods to ReceiverInterface, like requeu() or discard(). Worker would call this under the appropriate situations.
WDYT?
There was a problem hiding this comment.
Yes, could be a goog idea to move this into the worker ! +1 :)
There was a problem hiding this comment.
For your point 4, I wanted to keep the compatibility. As you wish :)
| @@ -206,6 +206,9 @@ public function channel(): \AMQPChannel | |||
| } | |||
|
|
|||
| $this->amqpChannel = $this->amqpFactory->createChannel($connection); | |||
| if (isset($this->connectionCredentials['prefetch_count'])) { | |||
| $this->amqpChannel->setPrefetchCount($this->connectionCredentials['prefetch_count']); | |||
| } | |||
There was a problem hiding this comment.
This is an extra feature unrelated to the main purpose of this PR, right?
There was a problem hiding this comment.
It's related to AMQP rejection process. By default, RabbitMQ uses a prefetch of 3. If you requeue a message, it comes behind the prefetched messages but not in the queue itself. If consumption speed is not important, you can choose a prefetch of 1, then the messages go back to the queue.
| } | ||
| } catch (UnrecoverableMessageExceptionInterface $e) { | ||
| try { | ||
| $this->connection->nack($AMQPEnvelope, AMQP_NOPARAM); |
There was a problem hiding this comment.
Why wouldn't we reject here? 🤔
|
With global retry and UnrecoverableMessageHandlingException, my PR is no more necessary :) |
consume_fatalandconsume_requeueto allow consumer to continue processing other messages without stoppingUnrecoverableMessageExceptionInterfaceandRecoverableMessageExceptionInterfaceinto * AMQP transport into exception to allow handler to nack messages with or without requeuenack(rejectis anackwithout multiple rejections abilities which is not used in Messenger context).