[Messenger] allow Redis transport to claim multiple messages sequentially #49028

AdamKatzDev · 2023-01-18T18:12:43Z

Q	A
Branch?	5.4
Bug fix?	yes
New feature?	no
Deprecations?	no
Tickets	Fix #49023
License	MIT

Fix for Redis transport, allows to claim multiple messages sequentially.
Before the fix a worker could claim only a single message per claim_interval from another consumer. In some cases this could lead to problems like processing delays or messages piling up.

carsonbot · 2023-01-18T18:12:49Z

Hey!

I see that this is your first PR. That is great! Welcome!

Symfony has a contribution guide which I suggest you to read.

In short:

Always add tests
Keep backward compatibility (see https://symfony.com/bc).
Bug fixes must be submitted against the lowest maintained branch where they apply (see https://symfony.com/releases)
Features and deprecations must be submitted against the 6.3 branch.

Review the GitHub status checks of your pull request and try to solve the reported issues. If some tests are failing, try to see if they are failing because of this change.

When two Symfony core team members approve this change, it will be merged and you will become an official Symfony contributor!
If this PR is merged in a lower version branch, it will be merged up to all maintained branches within a few days.

I am going to sit back now and wait for the reviews.

Cheers!

Carsonbot

carsonbot · 2023-01-19T12:46:37Z

Hey!

I see that this is your first PR. That is great! Welcome!

Symfony has a contribution guide which I suggest you to read.

In short:

Always add tests
Keep backward compatibility (see https://symfony.com/bc).
Bug fixes must be submitted against the lowest maintained branch where they apply (see https://symfony.com/releases)
Features and deprecations must be submitted against the 6.3 branch.

Review the GitHub status checks of your pull request and try to solve the reported issues. If some tests are failing, try to see if they are failing because of this change.

When two Symfony core team members approve this change, it will be merged and you will become an official Symfony contributor!
If this PR is merged in a lower version branch, it will be merged up to all maintained branches within a few days.

I am going to sit back now and wait for the reviews.

Cheers!

Carsonbot

carsonbot · 2023-01-19T13:39:34Z

Hey!

I see that this is your first PR. That is great! Welcome!

Symfony has a contribution guide which I suggest you to read.

In short:

Always add tests
Keep backward compatibility (see https://symfony.com/bc).
Bug fixes must be submitted against the lowest maintained branch where they apply (see https://symfony.com/releases)
Features and deprecations must be submitted against the 6.3 branch.

Review the GitHub status checks of your pull request and try to solve the reported issues. If some tests are failing, try to see if they are failing because of this change.

When two Symfony core team members approve this change, it will be merged and you will become an official Symfony contributor!
If this PR is merged in a lower version branch, it will be merged up to all maintained branches within a few days.

I am going to sit back now and wait for the reviews.

Cheers!

Carsonbot

AdamKatzDev · 2023-01-19T15:16:03Z

Not sure why the integration test keeps failing, it seems unrelated to the changes I've made.

AdamKatzDev · 2023-01-19T18:02:29Z

Looks like the tests in AppVeyor are a bit flaky, the code between builds is different only in a comment text:
https://ci.appveyor.com/project/fabpot/symfony/builds/45988646 - RedisExtIntegrationTest::testGetAfterReject crashes
https://ci.appveyor.com/project/fabpot/symfony/builds/45989058 - RedisExtIntegrationTest::testLazy crashes

I suppose that the PR is alright. Though it is a bit suspicious that the Connection tests are failing.

carsonbot · 2023-03-02T12:36:54Z

Hey!

I think @ostrolucky has recently worked with this code. Maybe they can help review this?

Cheers!

Carsonbot

…ally

nicolas-grekas

Thanks for the PR. I rebased and made a few changes to it, please fetch before working on your branch. Here are some comments to help move this forward.

nicolas-grekas · 2023-04-18T10:21:02Z

src/Symfony/Component/Messenger/Bridge/Redis/Tests/Transport/ConnectionTest.php

-            1 => 'consumer-2', // consumer-name
-            2 => 3600001, // idle
-        ]]);
+        $redis->expects($this->exactly(2))->method('xpending')->willReturnOnConsecutiveCalls(


I forgot to replace willReturnOnConsecutiveCalls with the $series trick (see removed code)

nicolas-grekas · 2023-04-18T10:21:31Z

src/Symfony/Component/Messenger/Bridge/Redis/Tests/Transport/ConnectionTest.php

            ->willReturn([]);

+        $redis->expects($this->exactly(3))->method('xpending')->willReturnOnConsecutiveCalls(


here also, willReturnOnConsecutiveCalls should be replaced, PHPUnit deprecated it.

nicolas-grekas · 2023-04-18T10:21:48Z

src/Symfony/Component/Messenger/Bridge/Redis/Tests/Transport/ConnectionTest.php

+                ['symfony', 'consumer', ['queue' => '0'], 1, null], // claim and get message redisid-123
+                ['symfony', 'consumer', ['queue' => '0'], 1, null]  // claim and get message redisid-234
+            )
+            ->willReturnOnConsecutiveCalls(


nicolas-grekas · 2023-04-18T10:21:54Z

src/Symfony/Component/Messenger/Bridge/Redis/Tests/Transport/ConnectionTest.php

+                ['queue' => [['message' => '{"body":"2","headers":[]}']]]
+            );
+
+        $redis->expects($this->exactly(2))->method('xpending')->willReturnOnConsecutiveCalls(


nicolas-grekas · 2023-04-18T10:22:52Z

src/Symfony/Component/Messenger/Bridge/Redis/Transport/Connection.php

@@ -329,13 +325,13 @@ private function claimOldPendingMessages()
                    ['JUSTID']
                );

-                $this->couldHavePendingMessages = true;
+                $couldHavePendingMessages = $couldHavePendingMessages || $claimResult || $try < 3 && $this->claimOldPendingMessages(++$try);


I added the $couldHavePendingMessages || part, to account for the variable possibly being set to true on L311, does that make sense?

nicolas-grekas · 2023-04-18T10:24:52Z

src/Symfony/Component/Messenger/Bridge/Redis/Transport/Connection.php

@@ -308,19 +308,15 @@ private function claimOldPendingMessages()
        $claimableIds = [];
        foreach ($pendingMessages as $pendingMessage) {


on L303, we call
$this->connection->xpending($this->stream, $this->group, '-', '+', 1);

Doesn't the 1 means we ask for only one item? Of not, how much iterations can this loop have? If the number is possibly high, do we need to do anything about that?

Sorry if this question is dumb, I'm not into this topic much.

AdamKatzDev · 2023-04-18T11:51:12Z

Hi, @nicolas-grekas. Thank you for your time.

The original intent of this PR was to be able to claim multiple stale messages for batch workers, but I've realized that this code won't work for that use case. I've described a possible implementation for new claim algorithm here #44400 (comment). But it is also very complicated and introduces a state into the Connection class, so I wasn't confident in the approach.

This PR can improve processing of stale messages when batches are not used, but in that case the queue unlikely to have a lot of stale messages to begin with. I guess the PR simplifies some parts of the code a little bit.

nicolas-grekas · 2023-04-18T12:05:07Z

Thanks for the feedback. Let's close, since this doesn't change the behavior but only the internal implementation, isn't it?

AdamKatzDev · 2023-04-18T13:25:43Z

@nicolas-grekas

Right.

Redis 6.2.0 introduced IDLE parameter to XPENDING and added XAUTOCLAIM command, it looks like these can help to improve the algorithm significantly, fix the issue that I've tried to tackle here and make the transport usable for batch handlers by also partially fixing #44400.

I will make another PR if I have some free time.

carsonbot added Status: Needs Review Bug Messenger labels Jan 18, 2023

carsonbot added this to the 5.4 milestone Jan 18, 2023

AdamKatzDev marked this pull request as draft January 18, 2023 18:40

AdamKatzDev marked this pull request as ready for review January 19, 2023 12:46

AdamKatzDev marked this pull request as draft January 19, 2023 13:05

AdamKatzDev marked this pull request as ready for review January 19, 2023 13:39

AdamKatzDev mentioned this pull request Jan 20, 2023

[Messenger] Could not acknowledge redis message with BatchHandlerInterface #44400

Open

nicolas-grekas changed the title ~~[Messenger] allow Redis transport to claim multiple messages sequenti…~~ [Messenger] allow Redis transport to claim multiple messages sequentially Apr 18, 2023

[Messenger] allow Redis transport to claim multiple messages sequenti…

79ae82d

…ally

nicolas-grekas force-pushed the messenger_redis_claim_fix_5.4 branch from 6f92983 to 79ae82d Compare April 18, 2023 10:19

nicolas-grekas reviewed Apr 18, 2023

View reviewed changes

nicolas-grekas closed this Apr 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Messenger] allow Redis transport to claim multiple messages sequentially #49028

[Messenger] allow Redis transport to claim multiple messages sequentially #49028

AdamKatzDev commented Jan 18, 2023

carsonbot commented Jan 18, 2023

carsonbot commented Jan 19, 2023

carsonbot commented Jan 19, 2023

AdamKatzDev commented Jan 19, 2023

AdamKatzDev commented Jan 19, 2023

carsonbot commented Mar 2, 2023

nicolas-grekas left a comment

nicolas-grekas Apr 18, 2023

nicolas-grekas Apr 18, 2023

nicolas-grekas Apr 18, 2023

nicolas-grekas Apr 18, 2023

nicolas-grekas Apr 18, 2023

nicolas-grekas Apr 18, 2023

AdamKatzDev commented Apr 18, 2023

nicolas-grekas commented Apr 18, 2023

AdamKatzDev commented Apr 18, 2023

		->willReturn([]);

		$redis->expects($this->exactly(3))->method('xpending')->willReturnOnConsecutiveCalls(

		@@ -308,19 +308,15 @@ private function claimOldPendingMessages()
		$claimableIds = [];
		foreach ($pendingMessages as $pendingMessage) {

[Messenger] allow Redis transport to claim multiple messages sequentially #49028

[Messenger] allow Redis transport to claim multiple messages sequentially #49028

Conversation

AdamKatzDev commented Jan 18, 2023

carsonbot commented Jan 18, 2023

carsonbot commented Jan 19, 2023

carsonbot commented Jan 19, 2023

AdamKatzDev commented Jan 19, 2023

AdamKatzDev commented Jan 19, 2023

carsonbot commented Mar 2, 2023

nicolas-grekas left a comment

Choose a reason for hiding this comment

nicolas-grekas Apr 18, 2023

Choose a reason for hiding this comment

nicolas-grekas Apr 18, 2023

Choose a reason for hiding this comment

nicolas-grekas Apr 18, 2023

Choose a reason for hiding this comment

nicolas-grekas Apr 18, 2023

Choose a reason for hiding this comment

nicolas-grekas Apr 18, 2023

Choose a reason for hiding this comment

nicolas-grekas Apr 18, 2023

Choose a reason for hiding this comment

AdamKatzDev commented Apr 18, 2023

nicolas-grekas commented Apr 18, 2023

AdamKatzDev commented Apr 18, 2023