[Messenger] [Amqp] Handle AMQPConnectionException when publishing a message.#54167
Conversation
|
Hey! Thanks for your PR. You are targeting branch "7.1" but it seems your PR description refers to branch "6.4". Cheers! Carsonbot |
| if (++$retries <= $maxRetries) { | ||
| $this->clear(); | ||
|
|
||
| goto retry; |
There was a problem hiding this comment.
This will cost years of public bashing, but I guess we can afford it 🥲
There was a problem hiding this comment.
hehehe I like it too. I feel like imo it reads better than a loop specifically for retry logic.
|
Thank you @jwage. |
|
Lovely. Thank you |
|
@nicolas-grekas @chalasr @Nyholm thinking more about this. Does this retry open up the possibility for a double publish? What if the write succeeds on the server but the client loses the connection and exception is thrown before getting the response to the write? Then we retry and double publish. Since this isn't idempotent and we have no guarantees, I don't think we can generically retry. @dunglas see the above. This is related to our discussion about not being able to safely retry non idempotent queries automatically in Doctrine. Asked here as well php-amqp/php-amqp#548 |
|
@chalasr @nicolas-grekas is there a way to add this fix into the LTS 5.4 Version? Many thanks! |
|
Hey @xabbuh thank you very much!! 🍻 |
…ng a message (jwage) This PR was merged into the 5.4 branch. Discussion ---------- [Messenger] Handle `AMQPConnectionException` when publishing a message | Q | A | ------------- | --- | Branch? | 5.4 | Bug fix? | yes | New feature? | no | Deprecations? | no | Issues | backport of #54167 | License | MIT Commits ------- e10aa0e [Messenger] [Amqp] Handle AMQPConnectionException when publishing a message.
Ports the upstream Symfony 5.4/6.4 fix for symfony/symfony#48241 into this 4.4 fork. Wraps Connection::publish() in withConnectionExceptionRetry() which catches \AMQPConnectionException, drops cached channel/queues/exchanges via clear(), and retries up to 3 times with no backoff. Fixes intermittent AMQPS publish failures against AWS AmazonMQ caused by AWS NLB's 350s idle TCP timeout silently killing connections — ext-amqp's isConnected() doesn't detect the dead socket so the next publish surfaces as "Library error: a SSL error occurred". Upstream PR: symfony/symfony#54167 Upstream issue: symfony/symfony#48241
channel() catches \AMQPConnectionException from $connection->connect() and re-wraps it as \AMQPException with a friendlier message. This discards the exception type, which means withConnectionExceptionRetry() (backported from symfony/symfony#54167) never matches — its catch clause targets \AMQPConnectionException specifically. Change the rethrow to \AMQPConnectionException so the retry wrapper can catch it and retry with a fresh connection, as intended. Production logs confirm the issue: "Library error: a SSL error occurred" propagates as AMQPException through the retry wrapper unmatched, despite the wrapper being correctly positioned around publish().
If you have a message handler that dispatches messages to another queue, you can encounter
AMQPConnectionExceptionwith the message "Library error: a SSL error occurred" or "a socket error occurred" depending on if you are using tls or not or if you are running behind a load balancer or not.You can manually reproduce this issue by dispatching a message where the handler then dispatches another message to a different queue, then go to rabbitmq admin and close the connection manually, then dispatch another message and when the message handler goes to dispatch the other message, you will get this exception:
TODO: