Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Messenger] Messages with AmqpStamp($routingKey) and direct RabbitMQ exchange are not retry-able #32994

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
srigi opened this issue Aug 6, 2019 · 9 comments

Comments

@srigi
Copy link

srigi commented Aug 6, 2019

Symfony version(s) affected: 4.3.3
RabbitMQ version(s) affected: 3.7.17

Description
Symfony Messenger by default creates fanout exchange Messages and all queues are receiving any messages published to the bus. In our project this is not acceptable. We thought that Messenger routing would be enough to place particular messages into corresponding queues. It was not.

To route some particular message into (and only to) configured queue, you need to configure Messenger to create a direct exchange, bind queues with routing key and finally to dispatch message with AmqpStamp() with said routing key:

framework:
    messenger:
        transports:
            auth_signup:
                dsn: '%rabbitmq_connection%'
                options:
                    exchange: { type: direct }
                    queues:
                        auth_signup:
                            binding_keys: [!php/const App\RoutingKeysDirect::AUTH_SIGNUP]
            emails_transactional:
                dsn: '%rabbitmq_connection%'
                options:
                    exchange: { type: direct }
                    queues:
                        emails_transactional:
                            binding_keys: [!php/const App\RoutingKeysDirect::EMAILS_TRANSACTIONAL]

        routing:
            App\RegistrationFinishedMessage: auth_signup
            App\EmailMessage: emails_transactional

With this setup we defined two queues bound to direct default exchange with routing key. We can inspect setup via management:

exchanges

queues

We can inspect binding of the queue in detail, we see it is really bound to exchange with our routing-key:

sm_qas

Now we dispatch message with routing-key like this:

$message = new App\RegistrationFinishedMessage('bla bla');
$messageRoutingKey = App\RoutingKeysDirect::AUTH_SIGNUP;

$bus->dispatch($message, [new AmqpStamp($messageRoutingKey)]);

Problem is, this setup will not correctly handle message retrying with selected retryPolicy. I step-debugged code today and found that for correct enqueue to delayed retry queue there must by routing-key present in form of AmqpStamp in Envelope object.

This Stamp is however not serialized into message because it is implementing NonSendableStampInterface. I don't know exact reasons behind this, but changing interface to StampInterface will solve this problem!

However a new problem will now emerge - shoveling the message into failed transport after message reaches configured retryCount in retryPolicy (default is 3).
After that, message should be enqueued to "failed" transport if configured. In our setup (see above) it will fail with error

Server channel error: 406, message: PRECONDITION_FAILED - inequivalent arg 'type' for exchange 'm
  essages' in vhost '/': received 'fanout' but current is 'direct'

This is solvable pretty easily, again by defining exchange type for failed transport to direct

framework:
    messenger:
        failure_transport: failed
        transports:
            failed:
                dsn: '%rabbitmq_connection%'
                options:
                    exchange: { type: direct }
                    queues:
                        failed: ~

            ... other transports

However our message will still not be enqueued to failed transport after reaching retryCount. Thats because our previous fix (changing interface of AmqpStamp) is now doing bad things when enqueuing failed message - message have routing-key attached to the Envelope which is not wanted!

To solve this second problem we must strip this Stamp from Envelope in SendFailedMessageToFailureTransportListener. Code should look like this:

        $envelope = $envelope->withoutAll(ReceivedStamp::class)
            ->withoutAll(TransportMessageIdStamp::class)
            ->withoutAll(AmqpStamp::class)  // THIS LINE ADDED
            ->with(new SentToFailureTransportStamp($event->getReceiverName()))
            ->with(new DelayStamp(0))
            ->with(new RedeliveryStamp(0, $this->failureSenderAlias, $throwable->getMessage(), $flattenedException));

Now everything is working as expected!

To sumarize

  • Symfony Messenger creates by default fanout exchange
  • that eliminates routing to specified queues
  • to achieve above we must configure direct default exchange
  • now we need to attach AmqpStamp($routingKey) with message when dispatching
  • for messages in this setup retrying is not working
  • it can be solved by changing interface of AmpqStamp from NonSendableStampInterface to StampInterface
  • second problem will emerge with this setup - failed transport is not working when configured
  • we must define direct exchange for failed transport as well
  • we must strip AmqpStamp from envelope in listener from message to be correctly enqueued to failed transport

Hope this bug report is explaining our problems in understable way.
Wish all the best.

@srigi
Copy link
Author

srigi commented Aug 27, 2019

Hello, can anybody please give some kind of feedback? Are there any deadlines/processes set? Can I help in any way?

This is starting to limit us as we will need this functionality to work correctly very soon.

@michaelbagrov
Copy link

Faced the same issue, is critical for us as well. Would be very appreciate for update when this could be fixed.

@tmbdrogba
Copy link

Same issue here :)

@weaverryan
Copy link
Member

Excellent bug report!

  1. The NonSendableStampInterface was added to in [Messenger] fix publishing headers set on AmqpStamp #32413 by @Tobion. I now think that was a mistake - you DO want this stamp to "stay" with the message so that it those same headers/routing key are sent when the message is retried.

  2. Adding ->withoutAll(AmqpStamp::class) to SendFailedMessageToFailureTransportListener then seems reasonable.. as we are trying to sort of "reset" some settings on the Envelope.

  3. Finally, about the "precondition" failed part on the failure transport, for now, I think that is not a bug... you just need to make sure that your exchange is configured as a "direct" type on all your transports. I think this is a bit inconvenient... but it's kind of a separate issue and one that you can work around anyways.

I would welcome a PR for this :).

@Tobion
Copy link
Contributor

Tobion commented Sep 16, 2019

Thanks for the detailed report.

To 1.
Indeed the routing key gets lost. Sending the AmqpStamp with the message will only solve the retry problem for routed messages also sent by Symfony (ref. #32049). So instead the better solution is to read the original routing key the message was received from with https://github.com/pdezwart/php-amqp/blob/a531e2861d45b999beafc2c04f68ca9d299e63b3/stubs/AMQPEnvelope.php#L26
that is available via https://github.com/symfony/messenger/blob/ea2a2f12841403b357f4740978b04cf2bba84859/Transport/AmqpExt/AmqpReceivedStamp.php#L30
We can use that as a fallback in https://github.com/symfony/messenger/blob/ea2a2f12841403b357f4740978b04cf2bba84859/Transport/AmqpExt/Connection.php#L460 that you referenced corectly.

Edit: Since the Connection does not know about the Envelope, the AmqpReceiver should just create an AmqpStamp based on the \AMQPEnvelope. Then it's similar to serializing the AmqpStamp but the retry will also work for routed, external messages.

@JulienMoreau
Copy link

Same issue here :)

@weaverryan
Copy link
Member

Someone just needs to create a PR based on @Tobion’s thoughts :)

@Tobion
Copy link
Contributor

Tobion commented Oct 26, 2019

See #34134 for a fix

@Tobion
Copy link
Contributor

Tobion commented Oct 31, 2019

After that, message should be enqueued to "failed" transport if configured. In our setup (see above) it will fail with error Server channel error: 406, message: PRECONDITION_FAILED - inequivalent arg 'type' for exchange 'messages' in vhost '/': received 'fanout' but current is 'direct'
This is solvable pretty easily, again by defining exchange type for failed transport to direct

This is caused because you use the same exchange for the failed messages as for the normal ones. Instead of making the failed transport direct, you can just change the exchange name:

framework:
    messenger:
        failure_transport: failed
        transports:
            failed:
                dsn: '%rabbitmq_connection%'
                options:
                    exchange: { name: failed}
                    queues:
                        failed: ~

Then you don't have the problem with the routing key when publishing to the failed transport. You don't want to change the routing key of those messages so they can easily be retried and keep the same routing key.

@fabpot fabpot closed this as completed Nov 4, 2019
fabpot added a commit that referenced this issue Nov 4, 2019
…nd properties (Tobion)

This PR was merged into the 4.3 branch.

Discussion
----------

[Messenger] fix retry of messages losing the routing key and properties

| Q             | A
| ------------- | ---
| Branch?       | 4.3
| Bug fix?      | yes
| New feature?  | no <!-- please update src/**/CHANGELOG.md files -->
| Deprecations? | no <!-- please update UPGRADE-*.md and src/**/CHANGELOG.md files -->
| Tickets       | Fix #32994 <!-- prefix each issue number with "Fix #", if any -->
| License       | MIT
| Doc PR        |

Messages sent for retry in rabbitmq lost the routing key and properties like the priority. Now we read those original properties and sent the retry message with the same properties (unless those properties have already been set manually before).

Commits
-------

75c674d [Messenger] fix retry of messages losing the routing key and properties
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants