[Messenger] add a transport that processes messages on kernel.terminate #28746

gonzalovilaseca · 2018-10-05T21:09:23Z

Q	A
Branch?	master
Bug fix?	no
New feature?	yes
BC breaks?	no
Deprecations?	no
Tests pass?	yes
Fixed tickets	#28646
License	MIT
Doc PR

I'm not sure if this is what Nicolas was asking for in this issue: #28646 (comment)
If it is, there are some questions that need general consensus:

Is it worth splitting this file into a listener and a handler and inject the handler in the listener?
Should the listener be in messenger component, in framework bundle or where?
Should the service definition: eg: tags for being processed as a handler and as an event subscriber be documented and configured manually by the user or should we put some kind of automation in place?

gonzalovilaseca · 2018-10-05T21:13:46Z

(Deprecation label should be removed BTW)

src/Symfony/Component/Messenger/Handler/KernelTerminationHandler.php

sroze · 2018-10-06T17:10:19Z

I think the approach of having a decorated handler might not be the best. What about using a transport for this?

gonzalovilaseca · 2018-10-11T20:03:38Z

@sroze Not sure if I got it right....is this the right approach? If it is, I'll create tests, documentation etc.

src/Symfony/Component/Messenger/Transport/KernelTerminate/KernelTerminateTransport.php

ogizanagi · 2018-10-11T21:50:46Z

Just for the sake of completeness, here is the whole workflow to me:

A message routed to the kernel.terminate transport is dispatched.
The SendMessageMiddleware gets the associated transport and send the message
KernelTerminateTransport::send() stores the message to be treated later.
If an exception occurs during the request, a kernel.exception listener clears the spool of messages, so none are handled.
On kernel.terminate, KernelTerminateTransport::flush() is executed,
which calls KernelTerminateTransport::receive() with the callable $handler(?Envelope $envelope) argument:
- adding a ReceivedMessage item to the envelope so the message is not sent again by the SendMessageMiddleware
- dispatching it to the bus.
KernelTerminateTransport::receive() loops over stored messages and call $handler for each of them. Thus, each message is dispatched to be finally handled.

I'm just not sure about 4. Perhaps it should be configurable at the transport level as an option.

sroze · 2018-10-21T13:02:58Z

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php

@@ -1553,6 +1554,11 @@ private function registerMessengerConfiguration(array $config, ContainerBuilder
                ->addTag('messenger.receiver', array('alias' => $name))
                ->addTag('messenger.sender', array('alias' => $name))
            ;
+
+            if ('symfony://kernel.terminate' === $transport) {
+                $transportDefinition->addTag('kernel.event_listener', array('event' => KernelEvents::TERMINATE));


Can't you take the event from what is after symfony:// instead of hardcoding it?

Does it make sense if we're only going to listen to that event?

See comments after #28646 (comment).
I'm not convinced another use-case should be provided out-of-the-box by the Framework bundle.

src/Symfony/Component/Messenger/Transport/KernelTerminate/KernelTerminateTransport.php

src/Symfony/Component/Messenger/Transport/KernelTerminate/KernelTerminateTransportFactory.php

nicolas-grekas · 2018-10-24T14:10:41Z

See #28646 (comment) for more thoughts on this.

gonzalovilaseca · 2018-10-25T19:48:35Z

I've updated the PR as per the comments (hope I understood all comments correctly...).
I'd like to ask for feedback on this issues:

I'd split MemoryTransport into 2 classes MemoryTransport and SymfonyKernelEventTransport, the second one will contain MemoryTransport plus the 2 kernel listener methods.
I also need to pass the bus to the memory transport but I'm not sure what's the best way to achieve it, it could be defined in the transport options, but then I'd need to inject it into the factory in src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php, the factory would then inject it in the memory transport. Any better way?

gonzalovilaseca · 2018-10-25T19:51:41Z

BTW does anyone have a sandbox project I could use to test this functionality?

gonzalovilaseca · 2018-10-28T19:03:35Z

@sroze @nicolas-grekas @ogizanagi Have updated the PR with suggestions given, it's not ready to review yet as it's not complete, but would like some feedback to check if I'm in the right direction before doing the final work.
Thanks!

ogizanagi · 2018-10-28T19:12:01Z

@gonzalovilaseca : Sorry for the lack of answer.

I also need to pass the bus to the memory transport but I'm not sure what's the best way to achieve it

I think the factory should be aware of all the available buses, by injecting a service locator. Then, a transport option should allow the factory to select the right bus by its key.

gonzalovilaseca · 2018-10-28T19:16:44Z

@gonzalovilaseca : Sorry for the lack of answer.

I also need to pass the bus to the memory transport but I'm not sure what's the best way to achieve it

I think the factory should be aware of all the available buses, by injecting a service locator. Then, a transport option should allow the factory to select the right bus by its key.

Can you check what I've done in src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php? I've just defined an alias for the chosen bus, it would make sense to inject a service locator if the transport was able to use more than one, I think it's easier WDYT?

ogizanagi · 2018-10-28T19:20:05Z

You may have more than one of such transport. For instance one per bus. The factory would be used to create each of these then (so it needs to know every buses).

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php

src/Symfony/Component/Messenger/Transport/MemoryTransport.php

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php

gonzalovilaseca · 2018-10-29T10:39:03Z

updated!

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php

gonzalovilaseca · 2018-10-30T16:04:37Z

@sroze @nicolas-grekas @ogizanagi Are we happy with this as it is, so I can start writing tests?

ogizanagi

Looks good (to me) yes :)

...Symfony/Component/Messenger/Transport/KernelTerminate/SymfonyKernelEventTransportFactory.php

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php

src/Symfony/Component/Messenger/Transport/MemoryTransport.php

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php

gonzalovilaseca · 2018-10-30T21:18:53Z

Updated!

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php

ogizanagi · 2018-10-31T19:08:09Z

@gonzalovilaseca: I hope you won't mind me, but I've rebased and pushed some changes to your branch hopefully fixing @sroze's review and some other tweaks I've spotted during this operation :)

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php

src/Symfony/Component/Messenger/Transport/KernelTerminateTransportFactory.php

ogizanagi · 2018-10-31T22:55:26Z

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php

@@ -1568,6 +1576,21 @@ private function registerMessengerConfiguration(array $config, ContainerBuilder
                ->setArguments(array($transport['dsn'], $transport['options']))
                ->addTag('messenger.receiver', array('alias' => $name))
            ;
+
+            if (0 === strpos($transport['dsn'], 'symfony://kernel.terminate')) {


Hmm. I just realized this won't work with env vars.
As I don't see any use-cases for it, it may be fine, but we need extra pov.
If we want to support it, the kernel terminate transport factory will need the event dispatcher and add listeners itself (still, that would be a better implementation and give the factory even more sense?).

(Note the same goes for the amqp:// check above, but it's only about a hint for the developper)

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php

gonzalovilaseca · 2018-11-09T22:05:37Z

@ogizanagi @sroze I've updated the PR with tests

ogizanagi · 2018-11-10T10:07:31Z

@gonzalovilaseca : Great! :)

But I think we need to treat #28746 (comment). Consistency is the key, this transport dsn should work as an env var as any other.
The default recipe suggests a MESSENGER_TRANSPORT_DSN env var that one may use as MESSENGER_TRANSPORT_DSN=symfony://kernel.terminate.
This means the factory must register the event listener itself at runtime, not the extension.

Regarding #29097 (comment), I also think we should extract the re-dispatching logic out of the MemoryTransport, which would just keep messages in memory, no more. So the null:// transport would just be a different factory.
But this last point could be done in a next PR, as we have time to iterate over this feature.

ogizanagi · 2018-11-10T10:13:55Z

src/Symfony/Component/Messenger/Transport/MemoryTransport.php

+     */
+    public function stop(): void
+    {
+        $this->envelopes = array();


Instead of emptying, we may just use a $stopped flag used in receive to skip handling as in

symfony/src/Symfony/Component/Messenger/Transport/NullTransport.php

Line 40 in 52f0a2c

if (!$this->stopped) {

so we keep envelopes list on exception, but don't re-dispatch them.

gonzalovilaseca · 2018-11-10T10:34:31Z

Can an event be registered at runtime? (I thought that wasnt possible as the container is already compiled)

ogizanagi · 2018-11-10T10:48:15Z

Sure. You'll need to inject the @event_dispatcher in the factory, and use EventDispatcher::addListener.
The container being already compiled just means we won't see the tagged service definition using debug:container (nor debug:event-dispatcher unless the factory is called before).

gonzalovilaseca · 2018-11-10T10:52:52Z

Ok cool, will do that then.

… kernel.terminate

gonzalovilaseca · 2018-12-13T20:39:27Z

@ogizanagi updated following suggestions

gharlan · 2019-01-23T09:05:12Z

src/Symfony/Component/Messenger/Transport/MemoryTransport.php

+    public function flush(): void
+    {
+        $this->receive(function (Envelope $envelope) {
+            if (null === $envelope) {


$envelope can not be null here, because of the type hint.

gharlan · 2019-01-23T09:25:47Z

src/Symfony/Component/Messenger/Transport/KernelTerminateTransportFactory.php

+        $transport = new MemoryTransport($this->busLocator->get($busId));
+
+        $this->eventDispatcher->addListener(KernelEvents::TERMINATE, array($transport, 'flush'));
+        $this->eventDispatcher->addListener(KernelEvents::EXCEPTION, array($transport, 'stop'));


Do we also have to add listeners to the corresponding console events? Like in swiftmailer-bundle for the memory spool?

gharlan · 2019-03-14T10:12:37Z

Any chance to get this merged into 4.3? (feature freeze is in two weeks)
How can I help?

gonzalovilaseca · 2019-03-14T10:26:03Z

It's just waiting for review..

sroze · 2019-04-06T10:25:07Z

Thank you very much for your efforts on this pull-request @gonzalovilaseca. Now that we have other transports (Doctrine especially) I think that the risks associated with such transport are too high: as mentioned by @fabpot ealier, running things on kernel.terminate are making things very hard to debug because you won't be able to see the errors (if some happen), even in development, nor be able to benefit from the retry mechanism that has been introduced.

Let's close this so we push everybody to use more reliable solutions 🚀

nicolas-grekas · 2019-04-06T10:42:19Z

Sorry for making you work on this, but even if it wasn't merged, it helped a lot moving things forward! So thank you!

carsonbot added Status: Needs Review Messenger Feature Deprecation labels Oct 5, 2018

chalasr removed the Deprecation label Oct 5, 2018

OskarStark requested changes Oct 6, 2018

View reviewed changes

src/Symfony/Component/Messenger/Handler/KernelTerminationHandler.php Outdated Show resolved Hide resolved

carsonbot added Status: Needs Work and removed Status: Needs Review labels Oct 6, 2018

chalasr changed the title ~~#28646 [Messenger] add a handler that processes messages on kernel.terminate~~ [Messenger] add a handler that processes messages on kernel.terminate Oct 8, 2018

ogizanagi reviewed Oct 11, 2018

View reviewed changes

nicolas-grekas added this to the next milestone Oct 14, 2018

sroze suggested changes Oct 21, 2018

View reviewed changes

gonzalovilaseca changed the title ~~[Messenger] add a handler that processes messages on kernel.terminate~~ [Messenger] add a transport that processes messages on kernel.terminate Oct 28, 2018

ogizanagi reviewed Oct 28, 2018

View reviewed changes

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php Outdated Show resolved Hide resolved

ogizanagi reviewed Oct 28, 2018

View reviewed changes

ogizanagi reviewed Oct 29, 2018

View reviewed changes

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php Outdated Show resolved Hide resolved

ogizanagi reviewed Oct 30, 2018

View reviewed changes

...Symfony/Component/Messenger/Transport/KernelTerminate/SymfonyKernelEventTransportFactory.php Outdated Show resolved Hide resolved

sroze suggested changes Oct 30, 2018

View reviewed changes

ogizanagi suggested changes Oct 31, 2018

View reviewed changes

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php Outdated Show resolved Hide resolved

gonzalovilaseca commented Oct 31, 2018

View reviewed changes

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php Outdated Show resolved Hide resolved

gonzalovilaseca commented Oct 31, 2018

View reviewed changes

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php Outdated Show resolved Hide resolved

gonzalovilaseca commented Oct 31, 2018

View reviewed changes

src/Symfony/Component/Messenger/Transport/KernelTerminateTransportFactory.php Outdated Show resolved Hide resolved

ogizanagi reviewed Oct 31, 2018

View reviewed changes

ogizanagi mentioned this pull request Nov 5, 2018

[Messenger] Add a "in-memory://" transport #29097

Merged

stloyd suggested changes Nov 5, 2018

View reviewed changes

src/Symfony/Bundle/FrameworkBundle/DependencyInjection/FrameworkExtension.php Show resolved Hide resolved

ogizanagi reviewed Nov 10, 2018

View reviewed changes

[Messenger][FrameworkBundle] add a handler that processes messages on…

b7671d6

… kernel.terminate

gharlan reviewed Jan 23, 2019

View reviewed changes

sroze closed this Apr 6, 2019

sroze added the ⭐️ EUFOSSA Hackathon label Apr 6, 2019

nicolas-grekas modified the milestones: next, 4.3 Apr 30, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Messenger] add a transport that processes messages on kernel.terminate #28746

[Messenger] add a transport that processes messages on kernel.terminate #28746

gonzalovilaseca commented Oct 5, 2018 •

edited

Loading

gonzalovilaseca commented Oct 5, 2018

sroze commented Oct 6, 2018

gonzalovilaseca commented Oct 11, 2018

ogizanagi commented Oct 11, 2018 •

edited

Loading

sroze Oct 21, 2018

gonzalovilaseca Oct 21, 2018

ogizanagi Oct 21, 2018

nicolas-grekas commented Oct 24, 2018

gonzalovilaseca commented Oct 25, 2018

gonzalovilaseca commented Oct 25, 2018

gonzalovilaseca commented Oct 28, 2018

ogizanagi commented Oct 28, 2018

gonzalovilaseca commented Oct 28, 2018

ogizanagi commented Oct 28, 2018 •

edited

Loading

gonzalovilaseca commented Oct 29, 2018

gonzalovilaseca commented Oct 30, 2018

ogizanagi left a comment

gonzalovilaseca commented Oct 30, 2018

ogizanagi commented Oct 31, 2018 •

edited

Loading

ogizanagi Oct 31, 2018 •

edited

Loading

gonzalovilaseca commented Nov 9, 2018

ogizanagi commented Nov 10, 2018

ogizanagi Nov 10, 2018

gonzalovilaseca commented Nov 10, 2018

ogizanagi commented Nov 10, 2018

gonzalovilaseca commented Nov 10, 2018

gonzalovilaseca commented Dec 13, 2018

gharlan Jan 23, 2019

gharlan Jan 23, 2019

gharlan commented Mar 14, 2019

gonzalovilaseca commented Mar 14, 2019

sroze commented Apr 6, 2019

nicolas-grekas commented Apr 6, 2019 •

edited

Loading

[Messenger] add a transport that processes messages on kernel.terminate #28746

[Messenger] add a transport that processes messages on kernel.terminate #28746

Conversation

gonzalovilaseca commented Oct 5, 2018 • edited Loading

gonzalovilaseca commented Oct 5, 2018

sroze commented Oct 6, 2018

gonzalovilaseca commented Oct 11, 2018

ogizanagi commented Oct 11, 2018 • edited Loading

sroze Oct 21, 2018

Choose a reason for hiding this comment

gonzalovilaseca Oct 21, 2018

Choose a reason for hiding this comment

ogizanagi Oct 21, 2018

Choose a reason for hiding this comment

nicolas-grekas commented Oct 24, 2018

gonzalovilaseca commented Oct 25, 2018

gonzalovilaseca commented Oct 25, 2018

gonzalovilaseca commented Oct 28, 2018

ogizanagi commented Oct 28, 2018

gonzalovilaseca commented Oct 28, 2018

ogizanagi commented Oct 28, 2018 • edited Loading

gonzalovilaseca commented Oct 29, 2018

gonzalovilaseca commented Oct 30, 2018

ogizanagi left a comment

Choose a reason for hiding this comment

gonzalovilaseca commented Oct 30, 2018

ogizanagi commented Oct 31, 2018 • edited Loading

ogizanagi Oct 31, 2018 • edited Loading

Choose a reason for hiding this comment

gonzalovilaseca commented Nov 9, 2018

ogizanagi commented Nov 10, 2018

ogizanagi Nov 10, 2018

Choose a reason for hiding this comment

gonzalovilaseca commented Nov 10, 2018

ogizanagi commented Nov 10, 2018

gonzalovilaseca commented Nov 10, 2018

gonzalovilaseca commented Dec 13, 2018

gharlan Jan 23, 2019

Choose a reason for hiding this comment

gharlan Jan 23, 2019

Choose a reason for hiding this comment

gharlan commented Mar 14, 2019

gonzalovilaseca commented Mar 14, 2019

sroze commented Apr 6, 2019

nicolas-grekas commented Apr 6, 2019 • edited Loading

gonzalovilaseca commented Oct 5, 2018 •

edited

Loading

ogizanagi commented Oct 11, 2018 •

edited

Loading

ogizanagi commented Oct 28, 2018 •

edited

Loading

ogizanagi commented Oct 31, 2018 •

edited

Loading

ogizanagi Oct 31, 2018 •

edited

Loading

nicolas-grekas commented Apr 6, 2019 •

edited

Loading