[Messenger] configure bus on transport #34247

Tobion · 2019-11-06T05:37:22Z

Q	A
Branch?	4.4
Bug fix?	no
New feature?	yes
Deprecations?	no
Tickets	Fix #32030 and part 3) of #32049
License	MIT
Doc PR

Changed implementation of RoutableMessage bus to use the existing ReceivedStamp (and SentToFailureTransportStamp) to do the routing. This way we can get rid of BusNameStamp and several middlewares.
The BusNameStamp was not a good solution because it does not respect that message handling is async. If you dispatch a message with bus A and delay (or it goes into retry), then rename the bus and redeploy, the message cannot be consumed suddenly because bus A does not exist anymore.
This is why explicitly setting the bus on the transport seems like the best solution.
Added bus option on the transport which fixes part 3) of [Messenger] make it easier to consume third-party messages #32049. The only thing that changes for people: If you defined more than one bus, you must set the "bus" config option on the transport. Messages received from this transport will be dispatched to this bus automatically. This replaces the --bus option of messenger:consume command and the BusNameStamp.
Removed --bus option of messenger:consume which was not good: The worker config is usually in supervisor or some other tool independent of messenger code. So you could easily forget to update the worker config there if you changed a bus name and it will not work at all. Or you accidentally dispatch messages to the wrong bus. With the "bus" option it's always in sync and validated on container build instead of at runtime of the worker.

With the removal of BusNameStamp we don't need to fallback bus inside RoutableMessageBus which was only for external messages or messages dispatched before its introduction for BC. With that we can make the default_busconfig optional even when having multiple buses which fixes [Messenger] Why are you forced to specify a "default bus"? #32030 and prevents unwanted autowiring. You can still use it though and is only a requirement for

symfony/src/Symfony/Bundle/FrameworkBundle/Controller/ControllerTrait.php

Lines 406 to 409 in 68d1b3f

    
           if (!$this->container->has('messenger.default_bus')) { 
        
               $message = class_exists(Envelope::class) ? 'You need to define the "messenger.default_bus" configuration option.' : 'Try running "composer require symfony/messenger".'; 
        
               throw new \LogicException('The message bus is not enabled in your application. '.$message); 
        
           }

The only thing not handled is [Messenger] Why are you forced to specify a "default bus"? #32030 (comment) but that should be a separate issue. If you used named autowiring there is no problem. It highly depends on the situation and to me we should try to enhance the error message there when autowiring fails due to MessageBusInterface

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/Configuration.php

Tobion · 2019-11-06T13:48:50Z

src/Symfony/Component/Messenger/RoutableMessageBus.php


-            return $this->fallbackBus->dispatch($envelope, $stamps);
+        /** @var SentToFailureTransportStamp|null $sentToFailureStamp */
+        $sentToFailureStamp = $envelope->last(SentToFailureTransportStamp::class);


This part replaces the FailedMessageProcessingMiddleware and makes sure the correct bus is used for messages consumed from the failure transport (directly or via messenger:failed:retry)

Tobion · 2019-11-06T19:09:20Z

I've only deprecated BusNameStamp so existing messages can be deserialized without an error.

weaverryan

Thanks for another thoughtful PR :).

I've started to review, but I'm hitting an immediate doubt. Currently, I could have 2 buses (command.bus and event.bus) and safely dispatch both of those to the same transport... because I don't care about priority and my app is simple and that's good enough for me. But with this PR, wouldn't I need to be careful to make sure that all "commands" are routed to a transport where bus is set to command.bus and all events are routed to a transport where bus is set to event.bus?

Attaching a bus name to each message is quite powerful: a worker can consume many different messages and each will definitely go through the correct bus. If the main motivation for the PR is to make the BusNameStamp optional when consuming from an external transport, could we instead leave the BusNameStamp idea, but add a new default_receive_bus option to each transport? This would be the transport that would be use if no explicit BusNameStamp is added. We could even change the behavior to throw an exception of no BusNameStamp or default_receive_bus is present (and there is more than one bus) so that you don't accidentally route all external messages to the "default bus".

We can also, of course, make the default_bus config optional as this PR does - that's kind of a separate "thing" and it seems sensible to me.

Thanks!

weaverryan · 2019-11-06T19:44:43Z

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/Configuration.php

-                        ->scalarNode('default_bus')->defaultNull()->end()
+                        ->scalarNode('default_bus')
+                            ->defaultNull()
+                            ->info('Bus name that will be used for autowiring MessageBusInterface. When there is only a single bus this happens automatically. With multiple buses you can set this option or use named autowiring based on the bus name or explicit wiring.')


Suggested change

->info('Bus name that will be used for autowiring MessageBusInterface. When there is only a single bus this happens automatically. With multiple buses you can set this option or use named autowiring based on the bus name or explicit wiring.')

->info('Bus name that will be used for autowiring MessageBusInterface. When there is only a single bus this happens automatically. With multiple buses, you can set this option or use named autowiring based on the bus name (or explicit wiring).')

Mostly, the last part of the sentence was confusing because it was "you can do X, or Y or Z" when really it's mean to read like "you can do X or Y (and Y is really 2 things). Hopefully this is clearer - it's minor regardless.

weaverryan · 2019-11-06T19:54:23Z

src/Symfony/Component/Messenger/RoutableMessageBus.php

            }

-            return $this->fallbackBus->dispatch($envelope, $stamps);
+            $envelope = $envelope->with(new ReceivedStamp($originalReceiver));


This logic does feel a bit unnatural here. For example, all the "sent to failure" stuff is done via an event listener, but then we wire this part of the logic right inside this class. It could still be done as an event listener (to look for the SentToFailureTransportStamp and then add the ReceivedStamp, couldn't it? I think the complexity is this doing this part of the process if it were in a different listener:

// in case the original receiver does not exist anymore, use the bus configured for failure transport

But I'm still not convinced this is the right place.

This is actually fixing a bug and a requirement for messenger:failed:retry to work with multiple buses.
If messages end in the failure transport because messages are missing the BusNameStamp (external) or because developers renamed the bus, then those messages currently cannot be consumed at all from the failure transport because they can never go to the right bus.
By doing these checks in the RoutableMessageBus we can use the configured bus of the original transport.

Tobion · 2019-11-06T20:35:20Z

Currently, I could have 2 buses (command.bus and event.bus) and safely dispatch both of those to the same transport... because I don't care about priority and my app is simple and that's good enough for me. But with this PR, wouldn't I need to be careful to make sure that all "commands" are routed to a transport where bus is set to command.bus and all events are routed to a transport where bus is set to event.bus?

That is true. But there is actually a use-case for not receiving from the same bus used for sending: #31848
He worked around the BusNameStamp by using the --bus option.

If we keep the BusNameStamp how do you suggest to deal with my argument 2) above? People loose all messages just by renaming the bus and you cannot warn them. Or all messages get dispatched to the default bus which is likely not the right one (and we don't want to force specifying one).
I don't really see a solution for this except requiring to chose the bus explicitly as in this PR which can be validated at compile time.

weaverryan · 2019-11-07T01:08:18Z

That is true. But there is actually a use-case for not receiving from the same bus used for sending: #31848
He worked around the BusNameStamp by using the --bus option.

Hmm, instead of a default_bus option, it could be called receive_bus and (most importantly) take precedent over BusNameStamp. Using receive_bus would be an advanced option (used for people consuming from external transports or people that want to handle via a different bus).

Or... the user in #31848 might be doing something we don't want to support - and maybe they should handle the received message in the same bus. And then, inside the handler, dispatch another message (which could be handled synchronously) into the "correct" bus.

If we keep the BusNameStamp how do you suggest to deal with my argument 2) above?

It's an edge case, but one that, indeed, we should worry about. To be fair, doesn't this PR have the same limitation, at least for the failure transport? From RoutableMessageBus:

// in case the original receiver does not exist anymore, use the bus configured for failure transport

That's less of an evil (because it affects less use-cases), but it still seems like we would have the same hole.

I could argue that removing a bus, while there are messages in the queue that are "scheduled" to be sent to it is simply something you should not do. After all, users already cannot remove a message class / handler from their app if there are more messages associated with that class that are in the queue. This is maybe the same situation as that? It seems to me that if a bus is missing, handling should fail and it should go into the failure transport so the user can look into it and deal with it.

teohhanhui · 2019-11-07T08:55:14Z

BusNameStamp feels like a hack... A message (Envelope) that knows about the message bus seems like a symptom of leaky abstraction.

weaverryan · 2019-11-08T02:39:49Z

BusNameStamp feels like a hack... A message (Envelope) that knows about the message bus seems like a symptom of leaky abstraction.

I'm fine with a better implementation if you can suggest one ;). The goal is simple: if I dispatch a message to command.bus, the message should absolutely be handled by command.bus when it is consumed. I can see no other way of doing this than for the message itself to know which bus it should go into.

Btw, I'm totally cool with adding an additional way for the system to know which bus to use (like an option on the transport that defines the bus to use if no BusNameStamp is present). The original issue was about making it easier to consume "external" messages. It seems that would accomplish this.

nicolas-grekas · 2019-11-09T21:53:51Z

What's the status here? I'm sorry I'm not into the topic much so I don't know how to help.
Is this PR on the critical path for the beta or can we consider it later?

Tobion · 2019-11-09T22:20:17Z

I'm confident the Symfony Messenger should not promote features that only work (and not even reliable as I pointed out) when dispatching with messenger and consuming with messenger. For anyone who has experience with message queues, such things are just not reasonable and you face big surprises.
I would be fine if the BusNameStamp logic would be something people can opt-in to explicitly, e.g. by setting the "bus" option on the transport, that I added here, to something like "dynamic" or "same-as-sent-from". But I would not be ok to have this as default solution.

smoench · 2019-11-11T09:35:41Z

I already faced a problem with BusNameStamp in a client project. Lets assume we have some services. Service A is dispatching and Service B consuming from A plus dispatching messages. B saves those message information via ORM with doctrine_ping_connection and doctrine_close_connection middlewares registered. B also dispatching messages in entity listeners (🙊). But now we need another bus otherwise doctrine's UOW will crash as the connection has been closed before the flush/commit thanks to doctrine_close_connection-middleware. I ended up defining a service_a-bus as default in A plus adding this bus in B for consuming.

It would be great having this in 4.4. It would improve the DX.

nicolas-grekas · 2019-11-11T14:14:00Z

Talking with @weaverryan on Slack, I think we should keep the BusNameStamp. Symfony Messenger talking to Symfony Messenger can legitimately provide a better/safer experience by default. This is what this stamp provides: traceability of the message in a way Symfony Messenger understands. Then, for interop with third-party apps, it's nice to not force them to use Symfony Messenger boilerplate.

Following this reasoning, it would make sense to me to allow binding a bus to a receiver - and this should be optional. When the stamp and the receiver don't match for the bus, we would need to decide for the best behavior: receiver wins, stamp wins, log, exception?

Anyway, I think it's too late to resolve this completly before 4.4/5.0
If there are things to deprecate in 5.1, let it be.

Tobion · 2019-11-11T17:11:12Z

The main problem of the BusNameStamp to me is

If you dispatch a message with bus A and delay (or it goes into retry), then rename the bus and redeploy, the message cannot be consumed suddenly because bus A does not exist anymore.

IMO there should not be a default solution that allows people to fall into this unexpected and non-obvious trap. And if we keep this as default default behavior, I also don't see how to fix #32030.

As I don't have time to work on this PR to make a potential compromise, let's just postpone it. Maybe we can some up with a solution later.

fabpot · 2020-08-11T10:38:09Z

Closing then for now.

biforb4 · 2021-04-15T05:22:23Z

any plans to continue with this?

Tobion · 2021-04-27T20:01:21Z

Not at the moment

Tobion requested a review from sroze as a code owner November 6, 2019 05:37

carsonbot added Status: Needs Review Messenger Feature labels Nov 6, 2019

Tobion force-pushed the messenger-no-bus-name-stamp branch from 7fcaa29 to 08bdab1 Compare November 6, 2019 05:49

nicolas-grekas added this to the 4.4 milestone Nov 6, 2019

teohhanhui reviewed Nov 6, 2019

View reviewed changes

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/Configuration.php Outdated Show resolved Hide resolved

Tobion force-pushed the messenger-no-bus-name-stamp branch from 08bdab1 to 13fbf63 Compare November 6, 2019 13:09

Tobion commented Nov 6, 2019

View reviewed changes

Tobion force-pushed the messenger-no-bus-name-stamp branch 2 times, most recently from 4541e3f to 083af0a Compare November 6, 2019 17:29

Tobion changed the title ~~[Messenger] WIP: configure bus on transport~~ [Messenger] configure bus on transport Nov 6, 2019

Tobion force-pushed the messenger-no-bus-name-stamp branch 2 times, most recently from d5535bf to 0361526 Compare November 6, 2019 18:11

Tobion mentioned this pull request Nov 6, 2019

[Messenger] make it easier to consume third-party messages #32049

Closed

Tobion force-pushed the messenger-no-bus-name-stamp branch from 0361526 to a930c6f Compare November 6, 2019 19:07

[Messenger] configure bus on transport

bbb5cb3

Tobion force-pushed the messenger-no-bus-name-stamp branch from a930c6f to bbb5cb3 Compare November 6, 2019 19:08

weaverryan reviewed Nov 6, 2019

View reviewed changes

nicolas-grekas removed this from the 4.4 milestone Nov 12, 2019

nicolas-grekas added this to the next milestone Nov 12, 2019

fabpot closed this Aug 11, 2020

nicolas-grekas modified the milestones: next, 5.2 Oct 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Messenger] configure bus on transport #34247

[Messenger] configure bus on transport #34247

Tobion commented Nov 6, 2019 •

edited

Loading

Tobion Nov 6, 2019

Tobion commented Nov 6, 2019

weaverryan left a comment

weaverryan Nov 6, 2019

weaverryan Nov 6, 2019

Tobion Feb 10, 2020

Tobion commented Nov 6, 2019 •

edited

Loading

weaverryan commented Nov 7, 2019

teohhanhui commented Nov 7, 2019 •

edited

Loading

weaverryan commented Nov 8, 2019

nicolas-grekas commented Nov 9, 2019

Tobion commented Nov 9, 2019 •

edited

Loading

smoench commented Nov 11, 2019

nicolas-grekas commented Nov 11, 2019

Tobion commented Nov 11, 2019 •

edited

Loading

fabpot commented Aug 11, 2020

biforb4 commented Apr 15, 2021

Tobion commented Apr 27, 2021

	if (!$this->container->has('messenger.default_bus')) {
	$message = class_exists(Envelope::class) ? 'You need to define the "messenger.default_bus" configuration option.' : 'Try running "composer require symfony/messenger".';
	throw new \LogicException('The message bus is not enabled in your application. '.$message);
	}

	->info('Bus name that will be used for autowiring MessageBusInterface. When there is only a single bus this happens automatically. With multiple buses you can set this option or use named autowiring based on the bus name or explicit wiring.')
	->info('Bus name that will be used for autowiring MessageBusInterface. When there is only a single bus this happens automatically. With multiple buses, you can set this option or use named autowiring based on the bus name (or explicit wiring).')

[Messenger] configure bus on transport #34247

[Messenger] configure bus on transport #34247

Conversation

Tobion commented Nov 6, 2019 • edited Loading

Tobion Nov 6, 2019

Choose a reason for hiding this comment

Tobion commented Nov 6, 2019

weaverryan left a comment

Choose a reason for hiding this comment

weaverryan Nov 6, 2019

Choose a reason for hiding this comment

weaverryan Nov 6, 2019

Choose a reason for hiding this comment

Tobion Feb 10, 2020

Choose a reason for hiding this comment

Tobion commented Nov 6, 2019 • edited Loading

weaverryan commented Nov 7, 2019

teohhanhui commented Nov 7, 2019 • edited Loading

weaverryan commented Nov 8, 2019

nicolas-grekas commented Nov 9, 2019

Tobion commented Nov 9, 2019 • edited Loading

smoench commented Nov 11, 2019

nicolas-grekas commented Nov 11, 2019

Tobion commented Nov 11, 2019 • edited Loading

fabpot commented Aug 11, 2020

biforb4 commented Apr 15, 2021

Tobion commented Apr 27, 2021

Tobion commented Nov 6, 2019 •

edited

Loading

Tobion commented Nov 6, 2019 •

edited

Loading

teohhanhui commented Nov 7, 2019 •

edited

Loading

Tobion commented Nov 9, 2019 •

edited

Loading

Tobion commented Nov 11, 2019 •

edited

Loading