@Tobion I'm not sure if this fully explains/covers your situation, but a serializer is supposed to throw a MessageDecodingFailedException message if deserialization specifically fails:

Then, receivers/transports are supposed to catch this and reject the message -

This is something that didn't happen in 4.2, so it was fixed in 4.3. It does, however, rely on two "shoulds" that are mentioned on two interfaces (the serializer "should" throw that exception and the receiver "should" catch it and reject).

Is this your issue? Do you have a custom serializer or transport that's not following these "shoulds"?

Update: And we decided to "reject" instead of retry as a deserialization error is not one that seems "temporary"

@weaverryan you are right. Our custom serializer didn't throw a MessageDecodingFailedException. Using that, errors in serialization are handled correctly. But there is still room for improvments:

1. Serialization errors could use the failed transport to keep track of them
2. We should still handle \AMQPEnvelope::isRedelivery as explained above. There can be alot of reasons for this to happen, e.g. connection lost or processing taking too long or like in my case serialization errors without using MessageDecodingFailedException. So that can still result in blocked queues and redelivery loops.

https://www.rabbitmq.com/confirms.html

This means that if all consumers requeue because they cannot process a delivery due to a transient condition, they will create a requeue/redelivery loop. Such loops can be costly in terms of network bandwidth and CPU resources. Consumer implementations can track the number of redeliveries and reject messages for good (discard them) or schedule requeueing after a delay.

So we should also handle retry for those.

3. Why do serialization errors (any error outside the bus) result in worker termination but exceptions in handlers don't? I think it's not a problem because you need something like supervisor anyway, but it feels arbitrary.

Ok, let's break this down so we can see what actionable things we can do.

1. Serialization errors could use the failed transport to keep track of them

That seems reasonable... we would basically "fail" in the same way that an exception from a handler. So, by default, it would fail 3 times, then go to the failure queue. This also would solve item (3) I believe: if exceptions from deserialization are handled the same as exceptions from handlers, then the worker would not exit in both situations.

2. We should still handle \AMQPEnvelope::isRedelivery as explained above

I'd appreciate a separate issue or PR on this... as I'm still far from a RabbitMQ expert. I don't quite understand the flow/problem:

A) Handler takes longer than Rabbit connections heartbeat or timeout
B) Rabbit puts back in queue with a Redelievered header
C) Our app sees the message with AMQPEnvelope::isRedelivery() and so we ack immediately and trigger a retry?
D) Our app handles the retried message

I'm fuzzy about a few things:

• In (A), if the handler took really long... does it mean it wasn't handled? Or did our app actually handle it and so we should not handle the redelivered one?
• In (C) Why do we ack and then trigger a retry? By "trigger a retry" do you mean use the same redelivery/retry functionality we currently have?

Cheers!

Yes the flow is described correctly. Trying to answer your questions:

• (A) That's a tough question. We don't really know if it was handled. It could be handled but just the ack to rabbitmq didn't work. Or it was handled partly and then failed and then the nack didn't work either. This undecidable problem is also in the retry. We retry the full message handling. But maybe only part of it failed and other parts/handlers actually succeeded. So some handlers will be executed twice. This is why you usually want to implement message queues with idempotence in mind. This is a general problem and not what I'm trying to solve here.
• (C) We ack first to break a potential redelivery loop. Say a handler just always takes too much time and loses the connection so the ack doesn't work. So it will get redelivered everytime and fail to ack every time. By acknowledging first (or using auto-ack), we break the loop. By retrying then (yes the normal retry functionality we already have), we still make sure the message get's handled at some point with a max retry counter to break the retry loop as well.

To keep this bumping, I think I see two actionable things:

A) On deserialization errors (MessageDecodingFailedException), sent to the failure transport so those messages can be dealt with later.

B) Handle \AMQPEnvelope::isRedelivery(), which would be: if \AMQPEnvelope::isRedelivery(), then ack() immediately and then redeliver (using our normal redelivery mechanism).

Correct?

We just had a different case where the messsage get's redelivered again and again blocking the queue.
We have a message containing SimpleXMLElement and when it fails and should be sent to the failed transport, the PhpSerializer errors with Serialization of 'SimpleXMLElement' is not allowed.
The problem is that it happens as a listener (SendFailedMessageToFailureTransportListener) in

I fixed problem B) in #34107

Feature A) (sent deserialization errors to the failure transport) is a nice-to-have but also not straight forward because the sending to failure transport relies on \Symfony\Component\Messenger\Event\WorkerMessageFailedEvent which requires an evelope. But when the deserialization fails, you obviously have no envelope to use. So let's keep that separate and does not have much prio to me.