Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[doctrine-messenger] DB table locks on messenger_messages with many failures #52273

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
bn-jdcook opened this issue Oct 24, 2023 · 2 comments
Closed

[doctrine-messenger] DB table locks on messenger_messages with many failures #52273

bn-jdcook opened this issue Oct 24, 2023 · 2 comments
Labels
Bug Messenger Status: Needs Review

Comments

@bn-jdcook
Copy link
Contributor

Symfony version(s) affected

5.4.25

Description

We are using doctrine-messenger in our project and have noticed a DB lock on the messenger_messages if we have many failed messages due to this query: SELECT m.* FROM messenger_messages m WHERE (m.delivered_at is null OR m.delivered_at < '2023-10-24 07:36:16') AND (m.available_at <= '2023-10-24 08:36:16') AND (m.queue_name = 'default') ORDER BY available_at ASC LIMIT 1 and m.delivered_at is set to null when a message is moved to the failed queue.

How to reproduce

Messenger Config:
framework:
messenger:
# Uncomment this (and the failed transport below) to send failed messages to this transport for later handling.
failure_transport: failed
reset_on_message: true

    transports:
        # https://symfony.com/doc/current/messenger.html#transport-configuration
        async:
            dsn: 'doctrine://default'
            retry_strategy:
                max_retries: 5
                delay: 60000

        failed: 'doctrine://default?queue_name=failed'
        sync: 'sync://'

    routing:
        # Route your messages to the transports
        'App\Core\Message\AsyncMessageInterface': async

Failed data:
Screenshot 2023-10-24 at 10 54 48

Query:
SELECT m.* FROM messenger_messages m WHERE (m.delivered_at is null OR m.delivered_at < '2023-10-24 07:36:16') AND (m.available_at <= '2023-10-24 08:36:16') AND (m.queue_name = 'default') ORDER BY available_at ASC LIMIT 1

Notice the query will first include all with m.delivered_at is null which will include all in the failed queue. Even if it does not always cause a DB lock on that table, it makes the query unnecessarily slow.

Possible Solution

Move the queue_name to higher in the where clause.

Additional Context

No response
@bn-jdcook bn-jdcook added the Bug label Oct 24, 2023
@bn-jdcook
Copy link
Contributor Author

It appears it would affect 6.x as well.

@bn-nchen
Copy link

Here is the body size of failure status data example:
image

@bn-jdcook bn-jdcook mentioned this issue Oct 24, 2023
Closed
bn-jdcook added a commit to bn-jdcook/symfony that referenced this issue Oct 24, 2023
@bn-jdcook
fix symfony#52273 [doctrine-messenger] DB table locks on messenger_me…
bdaabc0 
…ssages with many failures
@bn-jdcook bn-jdcook mentioned this issue Oct 24, 2023
Closed
bn-jdcook added a commit to bn-jdcook/symfony that referenced this issue Oct 25, 2023
@bn-jdcook
fix symfony#52273 [doctrine-messenger] DB table locks on messenger_me…
3245280 
…ssages with many failures
@nicolas-grekas nicolas-grekas mentioned this issue Oct 25, 2023
Merged
fabpot added a commit that referenced this issue Oct 25, 2023
@fabpot
bug #52276 DB table locks on messenger_messages with many failures (b…
b34c4c7 
…n-jdcook)

This PR was merged into the 5.4 branch.

Discussion
----------

DB table locks on messenger_messages with many failures

| Q             | A
| ------------- | ---
| Branch?       | 5.4
| Bug fix?      | yes
| New feature?  | no
| Deprecations? | no
| Issues        | Fix #52273
| License       | MIT

This reorders the query to target the `queue_name` first and not include any `failed` messages in the first part of the `WHERE` clause to help performance.

Commits
-------

3245280 fix #52273 [doctrine-messenger] DB table locks on messenger_messages with many failures
@fabpot fabpot closed this as completed Oct 25, 2023
fabpot added a commit that referenced this issue Oct 26, 2023
@fabpot
Merge branch '5.4' into 6.3
a9b9e4e 
* 5.4:
  [5.4] Remove unused test fixtures
  [Dotent] Add PHPDoc for `$overrideExistingVars`
  [SecurityBundle] Fix missing login-link element in xsd schema
  [Validator] Add missing Chinese translations #51934
  [Serializer] Fix using `DateIntervalNormalizer` with union types
  [Validator] fix: add missing translations for for Thai (th)
  fix #52273 [doctrine-messenger] DB table locks on messenger_messages with many failures
  [Serializer] Handle defaultContext for DateTimeNormalizer
  [CI] Add step to verify symfony/deprecation-contracts requirements
fabpot added a commit that referenced this issue Oct 26, 2023
@fabpot
Merge branch '6.3' into 6.4
3d078d1 
* 6.3:
  [6.3] Remove unused test fixture
  [5.4] Remove unused test fixtures
  [Dotent] Add PHPDoc for `$overrideExistingVars`
  [SecurityBundle] Fix missing login-link element in xsd schema
  [Validator] Add missing Chinese translations #51934
  [Serializer] Fix using `DateIntervalNormalizer` with union types
  [Validator] fix: add missing translations for for Thai (th)
  fix #52273 [doctrine-messenger] DB table locks on messenger_messages with many failures
  [Serializer] Handle defaultContext for DateTimeNormalizer
  declare constructor argument as optional for backwards compatibility
  [CI] Add step to verify symfony/deprecation-contracts requirements
fabpot added a commit that referenced this issue Oct 26, 2023
@fabpot
Merge branch '6.4' into 7.0
c0d4fde 
* 6.4:
  [6.3] Remove unused test fixture
  [5.4] Remove unused test fixtures
  [Dotent] Add PHPDoc for `$overrideExistingVars`
  [SecurityBundle] Fix missing login-link element in xsd schema
  [Validator] Add missing Chinese translations #51934
  replace a not-existing virtual request stack with the real one
  [Messenger] add handler description as array key to `HandlerFailedException::getWrappedExceptions()`
  [Serializer] Fix using `DateIntervalNormalizer` with union types
  [Validator] fix: add missing translations for for Thai (th)
  fix #52273 [doctrine-messenger] DB table locks on messenger_messages with many failures
  [Serializer] Handle defaultContext for DateTimeNormalizer
  declare constructor argument as optional for backwards compatibility
  [CI] Add step to verify symfony/deprecation-contracts requirements
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Messenger Status: Needs Review
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@fabpot @xabbuh @carsonbot @bn-jdcook @bn-nchen