Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Messenger] Batch consumer: don't flush unless jobs amount is equal or greater than batch size #54600

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ozahorulia opened this issue Apr 14, 2024 · 6 comments
Open

[Messenger] Batch consumer: don't flush unless jobs amount is equal or greater than batch size #54600

ozahorulia opened this issue Apr 14, 2024 · 6 comments
Labels
Messenger Stalled

Comments

@ozahorulia
Copy link

Description

When defining a batch consumer with a batch size bigger than current amount of messages in queue, it still flushes the queue no matter what by adding FlushBatchHandlersStamp to the envelope. I need it to wait until the amount of jobs would be greater or equal to my batch size OR flush it every X minutes otherwise. I tried to redeclare flush method like this:

    public function flush(bool $force): void
    {
        if (!$force) {
            // Here I was thinking to add an internal timer of some sort
            return;
        }

        if ($jobs = $this->jobs) {
            $this->jobs = [];
            $this->process($jobs);
        }
    }

But it doesn't really work, because flush() will not be called unless new message in the queue appears.

Then, according to the docs, "... pending batches are flushed when the worker is stopped.", so I tried to add --time-limit to the consumer call. But in this case I'm just getting this error:

The acknowledger was not called by the "..." batch handler

I'm not completely sure whether this is a bug or expected behavior, but it's clear that messenger component is not able to∂ handle batch messages waiting for the needed amount of messages in the queue.

Am I missing something?

Example

Nothing to add here to be honest. Probably it could be additional methods for the BatchHandlerInterface to define some kind of a timeout before force flush or flag if jobs must be flushed if their amount is less than batch size
@carsonbot
Copy link

Thank you for this issue.
There has not been a lot of activity here for a while. Has this been resolved?

@carsonbot
Copy link

Friendly reminder that this issue exists. If I don't hear anything I'll close this.

@carsonbot
Copy link

Hey,

I didn't hear anything so I'm going to close it. Feel free to comment if this is still relevant, I can always reopen!

@constantin-gordienko-quarks-tech
Copy link

constantin-gordienko-quarks-tech commented Jan 9, 2025 •
edited
Loading

I had the same problem, adding $force == true flag check in flush method, in order to wait until my queue is filled with right amount of messages. What did trick for me in case of stopping is to implementing __destruct() method in trait (somewhere close to redeclared flush method) and calling nack like this:
public function __destruct() { foreach ($this->jobs as [$message, $ack]) { $ack->nack(new \LogicException('The message was not handled before the messenger stopped')); } }
This way "The acknowledger was not called by the "..." batch handler" message is not appear.

@constantin-gordienko-quarks-tech
Copy link

constantin-gordienko-quarks-tech commented Jan 10, 2025 •
edited
Loading

One more thing to add - I eventually left flush() method untouched. Because I haven't found the way to gracefully ack/nack every acknowledger. There were still situations when it was not acknowledged, and I've got an exception on worker stop. In my case, batch helped me to speed up processing. And also process smaller batches, when it's not enough for 1 full bunch. (e.g. shouldFlush() == false), or when command stops or restarts.
In this topic #44400 I saw that someone advice to patch/edit vendor and comment __destruct method of Acknowledger. But for me this was overhead for the purpose of strict number in bunch to process.

@MatTheCat
Copy link
Contributor

Just hit this one as well; could this be reopened @xabbuh? 🙏

@xabbuh xabbuh reopened this Jan 13, 2025
@MartkCz MartkCz mentioned this issue Feb 8, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Messenger Stalled
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@MatTheCat @xabbuh @ozahorulia @carsonbot @constantin-gordienko-quarks-tech