Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Messenger] Add a command to setup transports #29476

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

vincenttouzet
Copy link
Contributor

Q A
Branch? master
Bug fix? no
New feature? yes
BC breaks? no
Deprecations? no
Tests pass? yes
License MIT

This PR add a SetupTransportsCommand that allow to setup the transports.

Actually the AMQPTransport is setup only if debug is enabled. With this PR the new messenger:setup-transports will setup all declared transports.

@nicolas-grekas
Copy link
Member

I don't get what this would allow. What does it mean to "setup a transport"?

@vincenttouzet
Copy link
Contributor Author

vincenttouzet commented Jan 28, 2019

@nicolas-grekas In that case "setup a transport" means to declare the queue on AMQP. Actually it is done only if we use debug : https://github.com/symfony/symfony/blob/master/src/Symfony/Component/Messenger/Transport/AmqpExt/Connection.php#L132

The goal of having this is to be able to easily configure the AMQP broker (RabbitMQ) without to switch environment.

I encounter this problem when we added a new routing configuration to dispatch a new message on a production environment. The queue wasn't declared and the message was not routed to the Broker (I don't remember if the message was lost or an error was raised 🤔 ). To fix this we had to create a command that we use during the deployment.

@vincenttouzet vincenttouzet force-pushed the messenger-setup-transport-command branch 2 times, most recently from 4fedfc8 to 1d33d2b Compare March 14, 2019 21:17
@vincenttouzet vincenttouzet force-pushed the messenger-setup-transport-command branch from 1d33d2b to d1fb482 Compare March 15, 2019 21:15
fabpot added a commit that referenced this pull request Mar 17, 2019
…(weaverryan)

This PR was merged into the 4.3-dev branch.

Discussion
----------

Using AMQP auto-setup in all cases, not just in debug

| Q             | A
| ------------- | ---
| Branch?       | master
| Bug fix?      | yes and no
| New feature?  | no
| BC breaks?    | yes
| Deprecations? | no->
| Tests pass?   | yes
| Fixed tickets | Related to #29476
| License       | MIT
| Doc PR        | TODO

Currently AMQP does auto-setup of queues/exchanges in dev-mode only. That's a problem for 2 reasons:

1) Behavior in prod is drastically different... and actually... there's not currently a way I know of (easily) to set things up on prod.

2) One of the properties of AMQP is that you typically DO want things to be set up at runtime, as you need them - you usually *do* want auto-setup.

This changes the behavior to auto-setup true always.

Commits
-------

503c209 Using AMQP auto-setup in all cases, not just in debug
@vincenttouzet vincenttouzet force-pushed the messenger-setup-transport-command branch from d1fb482 to 6c39a64 Compare March 18, 2019 21:26
Copy link
Contributor

@sroze sroze left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me (except the change in logging). Can you also:

  • Add a note in the FrameworkBundle's CHANGELOG?
  • Add some tests for the command?

Thank you!

@vincenttouzet
Copy link
Contributor Author

@sroze I've made the requested changes, rebased and squashed to have one commit

@vincenttouzet vincenttouzet force-pushed the messenger-setup-transport-command branch 2 times, most recently from 0445e8e to 99af3b7 Compare March 21, 2019 18:42
@vincenttouzet
Copy link
Contributor Author

@weaverryan @sroze I've just push to fix tests and rebase. Also made the requested changes :)

@vincenttouzet vincenttouzet force-pushed the messenger-setup-transport-command branch 3 times, most recently from 7c07612 to a271c39 Compare March 23, 2019 09:05
@vincenttouzet vincenttouzet force-pushed the messenger-setup-transport-command branch from a271c39 to cada294 Compare March 23, 2019 15:05
Copy link
Contributor

@sroze sroze left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 after the test change for me :)

Copy link
Member

@weaverryan weaverryan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 From me

Shouldn't really be used for AMQP - we need auto-setup for retry or delay. But, for Doctrine, though that PR also supports auto-setup, if people don't want that, or have DB users on production that can't create tables (reasonable), this gives them a way to create it.

@vincenttouzet vincenttouzet force-pushed the messenger-setup-transport-command branch from b578603 to fbb534a Compare March 26, 2019 20:08
@sroze
Copy link
Contributor

sroze commented Mar 27, 2019

Shouldn't really be used for AMQP - we need auto-setup for retry or delay

For AMQP, it would actually be useful for the following issue: #30218. 👍

@sroze sroze dismissed fabpot’s stale review March 27, 2019 11:25

Changes made 👍

@sroze
Copy link
Contributor

sroze commented Mar 27, 2019

Thank you @vincenttouzet.

@sroze sroze merged commit fbb534a into symfony:master Mar 27, 2019
sroze added a commit that referenced this pull request Mar 27, 2019
…touzet)

This PR was merged into the 4.3-dev branch.

Discussion
----------

[Messenger] Add a command to setup transports

| Q             | A
| ------------- | ---
| Branch?       | master
| Bug fix?      | no
| New feature?  | yes
| BC breaks?    | no
| Deprecations? | no
| Tests pass?   | yes
| License       | MIT

This PR add a `SetupTransportsCommand` that allow to setup the transports.

Actually the `AMQPTransport` is setup only if debug is enabled. With this PR the new `messenger:setup-transports` will setup all declared transports.

Commits
-------

fbb534a [Messenger] Add a command to setup transports
sroze added a commit that referenced this pull request Mar 31, 2019
This PR was merged into the 4.3-dev branch.

Discussion
----------

[Messenger] Add a Doctrine transport

| Q             | A
| ------------- | ---
| Branch?       | master
| Bug fix?      | no
| New feature?  | yes
| BC breaks?    | no
| Deprecations? | no
| Tests pass?   | yes
| Fixed tickets |
| License       | MIT
| Doc PR        | symfony/symfony-docs#10616
| DoctrineBundle PR | doctrine/DoctrineBundle#868

As discussed with @sroze at PHPForum in Paris I've worked on adding a Doctrine transport to the Messenger component.

Actually `AMQP` is the only supported transport and it could be a good thing to support multiple transports. Having a Doctrine transport could help users to start using the component IMHO (Almost all projects use a database).

# How it works

The code is splitted betwwen this PR and the one on the DoctrineBundle : doctrine/DoctrineBundle#868

## Configuration

To configure a Doctrine transport the dsn MUST have the format `doctrine://<entity_manager_name>` where `<entity_manager_name>` is the name of the entity manager (usually `default`)
```yml
        # config/packages/messenger.yaml
        framework:
            messenger:
                transports:
                    my_transport: "doctrine://default?queue=important"
```

## Table schema

Dispatched messages are stored into a database table with the following schema:

| Column       | Type     | Options                  | Description                                                       |
|--------------|----------|--------------------------|-------------------------------------------------------------------|
| id           | bigint   | AUTO_INCREMENT, NOT NULL | Primary key                                                       |
| body         | text     | NOT NULL                 | Body of the message                                               |
| headers      | text     | NOT NULL                 | Headers of the message                                            |
| queue      | varchar(32)     | NOT NULL                 | Headers of the message                                            |
| created_at   | datetime | NOT NULL                 | When the message was inserted onto the table. (automatically set) |
| available_at       | datetime   | NOT NULL                 | When the message is available to be handled                      |
| delivered_at | datetime | NULL                     | When the message was delivered to a worker                        |

## Message dispatching

When dispatching a message a new row is inserted into the table. See `Symfony\Component\Messenger\Transport\Doctrine::publish`

## Message consuming

The message is retrieved by the `Symfony\Component\Messenger\Transport\Doctrine\DoctrineReceiver`. It calls the `Symfony\Component\Messenger\Transport\Doctrine::get` method to get the next message to handle.

### Getting the next message

* Start a transaction
* Lock the table to get the first message to handle (The lock is done with the `SELECT ... FOR UPDATE` query)
* Update the message in database to update the delivered_at columns
* Commit the transaction

### Handling the message

The retrieved message is then passed to the handler. If the message is correctly handled the receiver call the `Symfony\Component\Messenger\Transport\Doctrine::ack` which delete the message from the table.

If an error occured the receiver call the `Symfony\Component\Messenger\Transport\Doctrine::nack` method which update the message to set the delivered_at column to `null`.

## Message requeueing

It may happen that a message is stuck in `delivered` state but the handler does not really handle the message (Database connection error, server crash, ...). To requeue messages the `DoctrineReceiver` call the `Symfony\Component\Messenger\Transport\Doctrine::requeueMessages`. This method update all the message with a  `delivered_at` not null since more than the "redeliver timeout" (default to 3600 seconds)

# TODO

- [x] Add tests
- [x] Create DOC PR
- [x] PR on doctrine-bundle for transport factory
- [x] Add a `available_at` column
- [x] Add a `queue` column
- [x] Implement the retry functionnality : See #30557
- [x] Rebase after #29476

Commits
-------

88d008c [Messenger] Add a Doctrine transport
@vincenttouzet vincenttouzet deleted the messenger-setup-transport-command branch April 7, 2019 07:44
@nicolas-grekas nicolas-grekas modified the milestones: next, 4.3 Apr 30, 2019
@fabpot fabpot mentioned this pull request May 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants