-
-
Notifications
You must be signed in to change notification settings - Fork 9.6k
[Messenger] Add a command to setup transports #29476
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Messenger] Add a command to setup transports #29476
Conversation
a54ec40
to
7ab7d9b
Compare
I don't get what this would allow. What does it mean to "setup a transport"? |
@nicolas-grekas In that case "setup a transport" means to declare the queue on AMQP. Actually it is done only if we use debug : https://github.com/symfony/symfony/blob/master/src/Symfony/Component/Messenger/Transport/AmqpExt/Connection.php#L132 The goal of having this is to be able to easily configure the AMQP broker (RabbitMQ) without to switch environment. I encounter this problem when we added a new routing configuration to dispatch a new message on a production environment. The queue wasn't declared and the message was not routed to the Broker (I don't remember if the message was lost or an error was raised 🤔 ). To fix this we had to create a command that we use during the deployment. |
4fedfc8
to
1d33d2b
Compare
1d33d2b
to
d1fb482
Compare
…(weaverryan) This PR was merged into the 4.3-dev branch. Discussion ---------- Using AMQP auto-setup in all cases, not just in debug | Q | A | ------------- | --- | Branch? | master | Bug fix? | yes and no | New feature? | no | BC breaks? | yes | Deprecations? | no-> | Tests pass? | yes | Fixed tickets | Related to #29476 | License | MIT | Doc PR | TODO Currently AMQP does auto-setup of queues/exchanges in dev-mode only. That's a problem for 2 reasons: 1) Behavior in prod is drastically different... and actually... there's not currently a way I know of (easily) to set things up on prod. 2) One of the properties of AMQP is that you typically DO want things to be set up at runtime, as you need them - you usually *do* want auto-setup. This changes the behavior to auto-setup true always. Commits ------- 503c209 Using AMQP auto-setup in all cases, not just in debug
d1fb482
to
6c39a64
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me (except the change in logging). Can you also:
- Add a note in the FrameworkBundle's CHANGELOG?
- Add some tests for the command?
Thank you!
src/Symfony/Component/Messenger/Command/SetupTransportsCommand.php
Outdated
Show resolved
Hide resolved
6c39a64
to
1df3c02
Compare
@sroze I've made the requested changes, rebased and squashed to have one commit |
src/Symfony/Component/Messenger/Command/SetupTransportsCommand.php
Outdated
Show resolved
Hide resolved
src/Symfony/Component/Messenger/Command/SetupTransportsCommand.php
Outdated
Show resolved
Hide resolved
src/Symfony/Component/Messenger/Command/SetupTransportsCommand.php
Outdated
Show resolved
Hide resolved
src/Symfony/Component/Messenger/Command/SetupTransportsCommand.php
Outdated
Show resolved
Hide resolved
src/Symfony/Component/Messenger/DependencyInjection/MessengerPass.php
Outdated
Show resolved
Hide resolved
0445e8e
to
99af3b7
Compare
@weaverryan @sroze I've just push to fix tests and rebase. Also made the requested changes :) |
src/Symfony/Component/Messenger/Command/SetupTransportsCommand.php
Outdated
Show resolved
Hide resolved
src/Symfony/Component/Messenger/Command/SetupTransportsCommand.php
Outdated
Show resolved
Hide resolved
7c07612
to
a271c39
Compare
src/Symfony/Component/Messenger/Command/SetupTransportsCommand.php
Outdated
Show resolved
Hide resolved
src/Symfony/Component/Messenger/Tests/Command/SetupTransportsCommandTest.php
Outdated
Show resolved
Hide resolved
a271c39
to
cada294
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 after the test change for me :)
src/Symfony/Component/Messenger/Tests/Command/SetupTransportsCommandTest.php
Outdated
Show resolved
Hide resolved
cada294
to
b578603
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 From me
Shouldn't really be used for AMQP - we need auto-setup for retry or delay. But, for Doctrine, though that PR also supports auto-setup, if people don't want that, or have DB users on production that can't create tables (reasonable), this gives them a way to create it.
b578603
to
fbb534a
Compare
For AMQP, it would actually be useful for the following issue: #30218. 👍 |
Thank you @vincenttouzet. |
…touzet) This PR was merged into the 4.3-dev branch. Discussion ---------- [Messenger] Add a command to setup transports | Q | A | ------------- | --- | Branch? | master | Bug fix? | no | New feature? | yes | BC breaks? | no | Deprecations? | no | Tests pass? | yes | License | MIT This PR add a `SetupTransportsCommand` that allow to setup the transports. Actually the `AMQPTransport` is setup only if debug is enabled. With this PR the new `messenger:setup-transports` will setup all declared transports. Commits ------- fbb534a [Messenger] Add a command to setup transports
This PR was merged into the 4.3-dev branch. Discussion ---------- [Messenger] Add a Doctrine transport | Q | A | ------------- | --- | Branch? | master | Bug fix? | no | New feature? | yes | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | | License | MIT | Doc PR | symfony/symfony-docs#10616 | DoctrineBundle PR | doctrine/DoctrineBundle#868 As discussed with @sroze at PHPForum in Paris I've worked on adding a Doctrine transport to the Messenger component. Actually `AMQP` is the only supported transport and it could be a good thing to support multiple transports. Having a Doctrine transport could help users to start using the component IMHO (Almost all projects use a database). # How it works The code is splitted betwwen this PR and the one on the DoctrineBundle : doctrine/DoctrineBundle#868 ## Configuration To configure a Doctrine transport the dsn MUST have the format `doctrine://<entity_manager_name>` where `<entity_manager_name>` is the name of the entity manager (usually `default`) ```yml # config/packages/messenger.yaml framework: messenger: transports: my_transport: "doctrine://default?queue=important" ``` ## Table schema Dispatched messages are stored into a database table with the following schema: | Column | Type | Options | Description | |--------------|----------|--------------------------|-------------------------------------------------------------------| | id | bigint | AUTO_INCREMENT, NOT NULL | Primary key | | body | text | NOT NULL | Body of the message | | headers | text | NOT NULL | Headers of the message | | queue | varchar(32) | NOT NULL | Headers of the message | | created_at | datetime | NOT NULL | When the message was inserted onto the table. (automatically set) | | available_at | datetime | NOT NULL | When the message is available to be handled | | delivered_at | datetime | NULL | When the message was delivered to a worker | ## Message dispatching When dispatching a message a new row is inserted into the table. See `Symfony\Component\Messenger\Transport\Doctrine::publish` ## Message consuming The message is retrieved by the `Symfony\Component\Messenger\Transport\Doctrine\DoctrineReceiver`. It calls the `Symfony\Component\Messenger\Transport\Doctrine::get` method to get the next message to handle. ### Getting the next message * Start a transaction * Lock the table to get the first message to handle (The lock is done with the `SELECT ... FOR UPDATE` query) * Update the message in database to update the delivered_at columns * Commit the transaction ### Handling the message The retrieved message is then passed to the handler. If the message is correctly handled the receiver call the `Symfony\Component\Messenger\Transport\Doctrine::ack` which delete the message from the table. If an error occured the receiver call the `Symfony\Component\Messenger\Transport\Doctrine::nack` method which update the message to set the delivered_at column to `null`. ## Message requeueing It may happen that a message is stuck in `delivered` state but the handler does not really handle the message (Database connection error, server crash, ...). To requeue messages the `DoctrineReceiver` call the `Symfony\Component\Messenger\Transport\Doctrine::requeueMessages`. This method update all the message with a `delivered_at` not null since more than the "redeliver timeout" (default to 3600 seconds) # TODO - [x] Add tests - [x] Create DOC PR - [x] PR on doctrine-bundle for transport factory - [x] Add a `available_at` column - [x] Add a `queue` column - [x] Implement the retry functionnality : See #30557 - [x] Rebase after #29476 Commits ------- 88d008c [Messenger] Add a Doctrine transport
This PR add a
SetupTransportsCommand
that allow to setup the transports.Actually the
AMQPTransport
is setup only if debug is enabled. With this PR the newmessenger:setup-transports
will setup all declared transports.