Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Messenger][Redis] Transport can claim only single message per claim_interval #49023

Open
@AdamKatzDev

Description

@AdamKatzDev

Symfony version(s) affected

5.4, 6.3

Description

https://github.com/symfony/symfony/blob/6.3/src/Symfony/Component/Messenger/Bridge/Redis/Transport/Connection.php

After a message is claimed nextClaim field is reseted:

if (\count($claimableIds) > 0) {
try {
$this->getRedis()->xclaim(
$this->stream,
$this->group,
$this->consumer,
$this->redeliverTimeout,
$claimableIds,
['JUSTID']
);
$this->couldHavePendingMessages = true;
} catch (\RedisException $e) {
throw new TransportException($e->getMessage(), 0, $e);
}
}
$this->nextClaim = microtime(true) + $this->claimInterval;
}

which does not allow to claim next messages:
if (!$this->couldHavePendingMessages && $this->nextClaim <= microtime(true)) {
$this->claimOldPendingMessages();
}

This logic might be intended and works in many cases but it doesn't work well for batch workers with large batch size or large claim_interval setting. It can potentially cause messages to pile up in worst cases or to be processed far too late.

How to reproduce

Possible to reproduce this by SIGKILL'ing a batch worker with messages stored. I hope that my explanation is good enough and a reproducer is not needed.

Possible Solution

Should be possible to fix this by moving nextClaim update to a different place. The field should not be updated if there is a pending message with the same consumer (already does) or a message from another consumer was claimed (this bug).

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions