[Doctrine][Messenger] Make the messenger doctrine transport scalable #50972

hgraca · 2023-07-13T21:40:53Z

Q	A
Branch?	6.4
Bug fix?	no
New feature?	yes
Deprecations?	no
Tickets	Fix #47828
License	MIT
Doc PR	symfony/symfony-docs#18566

Relevant changes

This PR has a few commits with small improvements and two commits with relevant changes.

Here are the most significant changes:

Replacement and removal of the DoctrineReceivedStamp

It seems to be redundant because there is already another stamp there being used for the same purpose. Thus, in a commit I replace its usage and in the other commit I remove it. This might be considered a breaking change because the DoctrineReceivedStamp is not marked final, Im not sure, you tell me, these commits can be easily dropped without impact to the rest :)

Add "SKIP LOCKED" to the Mysql8 and Postgres query that retrieves messages

The current SELECT ... FOR UPDATE retrieves rows, using the index to find
and lock the retrieved rows.
This means that when we have more than one consumer, while
one consumer is locking those rows, the whole index is locked thus
other queries (consumers) will be put on hold.
While with a small table this might not be noticeable, as the table
grows the meddling with the index becomes slower and the other
consumers have to wait more time, eventually making the MQ inoperable.

The SKIP LOCKED addition will allow other consumers to query the
table and get messages immediately, ignoring the rows that other
consumers are locking.

Add batch delivery

Currently, the doctrine transport only delivers one message at a time,
resulting in low performance and a very high amount of hits to the DB.
(one hit per message)

With batch delivery, the transport will retrieve several messages in a single query.

Testing

I adjusted the tests, and added a few more cases.

I also tested this in a playground project, sending and receiving 400000 asynchronous messages, which were handled by 2 worker containers, each with 2 consumers.
All messages were processed and there were no duplicates.
These tests were done with mysql, postgres and mssql.

Maria DB was not updated with the SKIP LOCKED because the engine version that supports it, is not yet supported by Doctrine.

Oracle was not updated with SKIP LOCKED either because I could not even connect to the damn thing using my db client.

This allows for projects to set up the messenger config using this constant, making it deterministic, and therefore less prone to errors.

We already have the TransportMessageIdStamp, so the DoctrineReceivedStamp is redundant, as it contains the same data and its usage is semantically more correct by using the TransportMessageIdStamp.

The current `SELECT ... FOR UPDATE` retrieves rows, using the index to find and lock the retrieved rows. This means that when we have more than one consumer, while one consumer is locking those rows, the whole index is locked thus other queries (consumers) will be put on hold. While with a small table this might not be noticeable, as the table grows the meddling with the index becomes slower and the other consumers have to wait more time, eventually making the MQ inoperable. The `SKIP LOCKED` addition will allow other consumers to query the table and get messages immediately, ignoring the rows that other consumers are locking. Co-authored-by: Alexander Malyk <[email protected]>

Currently, the doctrine transport only delivers one message at a time, resulting in low performance and a very high amount of hits to the DB. (one hit per message) With batch delivery, the transport will retrieve several messages in a single query. Co-authored-by: Alexander Malyk <[email protected]>

derrabus · 2023-07-14T06:42:27Z

Thank you very much for this PR.

This PR has a few commits with small improvements and two commits with relevant changes.

I think, this should've been three PRs. Because you've changed multiple aspects of the Doctrine transport, the review might take a while. Smaller isolated changes are usually easier to review.

Replacement and removal of the DoctrineReceivedStamp

It seems to be redundant because there is already another stamp there being used for the same purpose. Thus, in a commit I replace its usage and in the other commit I remove it. This might be considered a breaking change because the DoctrineReceivedStamp is not marked final, Im not sure, you tell me, these commits can be easily dropped without impact to the rest :)

Changing the stamps that the transport attaches to the message is a breaking change. If you can split this into a separate PR, please do that and we discuss the change there.

Add "SKIP LOCKED" to the Mysql8 and Postgres query that retrieves messages

Very nice. I wonder if it makes sense to contribute support for SKIP LOCKED to DBAL. This kind of query manipulation feels like it belongs there.

Add batch delivery

Do we have other transports that support batches? Would it makes sense to generalize this feature?

Maria DB was not updated with the SKIP LOCKED because the engine version that supports it, is not yet supported by Doctrine.

And what version would that be? DBAL supports all MariaDB versions since 10.0, up to 11.0 currently.

Oracle was not updated with SKIP LOCKED either because I could not even connect to the damn thing using my db client.

😄

hgraca · 2023-07-18T14:46:08Z

Hi @derrabus tkx for the review.

I think, this should've been three PRs.

How would you separate the commits per PR?
Would this

1. 
Add DoctrineTransportFactory DSN_PREFIX constant 
Add type hint for $driverConnection in DoctrineTransportFactory
Ensure messages flagged for deletion have clear flag 

2.
Replace usage of DoctrineReceivedStamp
Remove the unused DoctrineReceivedStamp

3.
Add "SKIP LOCKED" to the query that retrieves messages

4.
Add batch delivery

make sense?

Changing the stamps that the transport attaches to the message is a breaking change. If you can split this into a separate PR, please do that and we discuss the change there.

Will do

I wonder if it makes sense to contribute support for SKIP LOCKED to DBAL.

Yes, I believe it makes sense, I was eager to get this functionality in the messenger though.
Do you want me to do that update in DBAL first?

Do we have other transports that support batches? Would it makes sense to generalize this feature?

The transport was already prepared for this, cozz the ReceiverInterface::get(): iterable; returns an iterable (batch of messages), and it was in fact returning [Envelope] and then consumer is handling all envelopes in the array, so I guess it is already the case that it is generalised, just the doctrine transport was not making use of it.

And what version would that be? DBAL supports all MariaDB versions since 10.0, up to 11.0 currently.

The SKIP LOCKED clause was introduced in MariaDB 10.6.0, but DBAL supports from 10.2.7 GA and there's no way to choose a higher version (AFAIK) so I assumed we needed to support the lowest version.
Let me know if that is not the case and I can add support for MariaDB.

derrabus · 2023-07-19T11:37:08Z

I think, this should've been three PRs.

How would you separate the commits per PR? Would this

We usually squash PRs before merging. Since you deliver multiple features with this PR, multiple PRs would make it easier to discuss the implementation of one feature without blocking the merge of another feature.

I wonder if it makes sense to contribute support for SKIP LOCKED to DBAL.

Yes, I believe it makes sense, I was eager to get this functionality in the messenger though. Do you want me to do that update in DBAL first?

Since we're targeting 6.4 which is due in November, we're not in a hurry. If you have an idea how to contribute this to DBAL, please go ahead.

I guess it is already the case that it is generalised, just the doctrine transport was not making use of it.

I know, but unless I've missed something, this PR would be the first that makes the batch size configurable. But that was just an idea/observation from my side. We can go with your approach and think about generalizing that functionality later.

DBAL supports from 10.2.7 GA and there's no way to choose a higher version (AFAIK)

DBAL does not introduce new platform classes unless they're needed. DBAL 3.7 will ship new ones for MariaDB, but still not the one you need. You can query the database version from the connection though. However, the fact that you'll need a new versioned platform class probably indicates that adding that feature to DBAL is a good idea.

hgraca · 2023-07-19T12:39:38Z

@derrabus
Cool, I will break this up in separate PRs, open a PR in DBAL and when that's merged I will open the PR for the SKIP LOCKED commit here.

hgraca · 2023-07-19T22:39:44Z

Opened up this PR
#51034

The others will follow up after the DBAL PR is merged

Herberto Graca and others added 7 commits July 9, 2023 23:13

Ensure messages flagged for deletion have clear flag

2f5375f

Add type hint for $driverConnection in DoctrineTransportFactory

a2f2741

Add DoctrineTransportFactory DSN_PREFIX constant

84cc536

This allows for projects to set up the messenger config using this constant, making it deterministic, and therefore less prone to errors.

Replace usage of DoctrineReceivedStamp

1863a80

We already have the TransportMessageIdStamp, so the DoctrineReceivedStamp is redundant, as it contains the same data and its usage is semantically more correct by using the TransportMessageIdStamp.

Remove the unused DoctrineReceivedStamp

ab468d1

carsonbot added Status: Needs Review Feature labels Jul 13, 2023

carsonbot added this to the 6.4 milestone Jul 13, 2023

Herberto Graca added 4 commits July 14, 2023 00:08

fixup! Replace usage of DoctrineReceivedStamp

54360a3

fixup! Add "SKIP LOCKED" to the query that retrieves messages

0f126f9

fixup! Add batch delivery

2013bdc

Remove obsolete comment

8a4e1a7

derrabus added Doctrine Messenger labels Jul 14, 2023

carsonbot changed the title ~~Make the messenger doctrine transport scalable~~ [Doctrine][Messenger] Make the messenger doctrine transport scalable Jul 14, 2023

derrabus mentioned this pull request Jul 19, 2023

[Messenger] support mysql 8.0 skip locked #51013

Closed

hgraca closed this Jul 19, 2023

hgraca deleted the scalable_doctrine_transport branch July 27, 2023 08:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Doctrine][Messenger] Make the messenger doctrine transport scalable #50972

[Doctrine][Messenger] Make the messenger doctrine transport scalable #50972

Uh oh!

hgraca commented Jul 13, 2023

Uh oh!

derrabus commented Jul 14, 2023

Replacement and removal of the DoctrineReceivedStamp

Add "SKIP LOCKED" to the Mysql8 and Postgres query that retrieves messages

Add batch delivery

Uh oh!

hgraca commented Jul 18, 2023

Uh oh!

derrabus commented Jul 19, 2023

Uh oh!

hgraca commented Jul 19, 2023

Uh oh!

hgraca commented Jul 19, 2023

Uh oh!

Uh oh!

Uh oh!

[Doctrine][Messenger] Make the messenger doctrine transport scalable #50972

[Doctrine][Messenger] Make the messenger doctrine transport scalable #50972

Uh oh!

Conversation

hgraca commented Jul 13, 2023

Relevant changes

Replacement and removal of the DoctrineReceivedStamp

Add "SKIP LOCKED" to the Mysql8 and Postgres query that retrieves messages

Add batch delivery

Testing

Uh oh!

derrabus commented Jul 14, 2023

Replacement and removal of the DoctrineReceivedStamp

Add "SKIP LOCKED" to the Mysql8 and Postgres query that retrieves messages

Add batch delivery

Uh oh!

hgraca commented Jul 18, 2023

Uh oh!

derrabus commented Jul 19, 2023

Uh oh!

hgraca commented Jul 19, 2023

Uh oh!

hgraca commented Jul 19, 2023

Uh oh!

Uh oh!