feature: non-blocking consume added - #627
Conversation
nicolas-grekas
left a comment
There was a problem hiding this comment.
I needed exactly this while adding push consumption to Symfony's AMQP transport (symfony/symfony#65924), so I built this branch against PHP 8.4 and put it through the cases I care about. It does what it says:
1. empty buffer, JUST_CONSUME|NB_CONSUME: returned in 0.0002s, no exception
same call without the flag: 1.0003s, AMQPQueueException "Consumer timeout exceed"
2. 8 messages already pushed: all 8 delivered in 0.0004s
3. 100 bodies of 10B to 700KB (many
frames each), drained through NB: intact and in order
4. after disconnect(): AMQPChannelException still raised
Point 4 matters: only AMQP_STATUS_TIMEOUT is swallowed, so a genuine failure is still surfaced. And 32768 is the next free bit after AMQP_REQUEUE, with no overlap with the flags basic.consume itself reads (AMQP_NOLOCAL, AMQP_AUTOACK, AMQP_EXCLUSIVE).
Why this is worth having
Without it, "the buffer is empty" can only be learned by catching an exception, and that turns out to cost more than the ugliness. Until php/php-src#23624, the engine drops every signal queued while an internal function was running when that function returns with an exception pending: ZEND_VM_FCALL_INTERRUPT_CHECK() sits right after an internal call returns, before the exception is handled, so pcntl_signal_dispatch() runs with EG(exception) set, zend_call_function() no-ops, and the queued signals are recycled without a single handler running.
A worker looping on consume() spends essentially all of its wall time inside that call, so it misses every SIGTERM:
read_timeout=0.20s, SIGTERM at 1.70s: 31 rounds, never stopped
read_timeout=0.05s, SIGTERM at 2.13s: 121 rounds, never stopped
read_timeout=0.01s, SIGTERM at 2.57s: 653 rounds, never stopped
653 iterations of a 10 ms read timeout and the signal never arrives, not even through an explicit pcntl_signal_dispatch() afterwards. Not blocked and not ignored: SigBlk, SigIgn and SigCgt read from inside the consume callback confirm the handler is installed, and the same signal during sleep() or stream_select() is delivered normally. What makes consume() different is only that it throws, which is why a no-throw drain is not just cosmetic.
One doc suggestion
AMQP_NB_CONSUME is non-blocking with respect to the next message, not mid-message: once the method frame is read, amqp_read_message() waits for the header and body frames with no timeout. That is the right behaviour, and it is what keeps the 700KB bodies in my test 3 intact, but the stub currently reads as an unqualified "returns immediately", which could surprise someone on a slow link. Something like "returns as soon as no further message is buffered; a message already being received is still read to completion" would set the expectation better.
Worth noting too that with a callback returning true the flag does not mean "drain what is buffered right now and stop": it keeps going while frames keep arriving. All 100 messages in my test 3 came back in a single call. That is fine, just not obvious from the name.
Nothing blocking from me. Happy to see this land, it would let the Symfony bridge drop a setReadTimeout(0.001) workaround.
7e1cca7 to
f6d1377
Compare
|
Thanks a lot for the thorough review, @nicolas-grekas! Really appreciate you putting the branch through such a rigorous set of tests, including the signal-handling investigation — that's a great catch and explains a real-world pain point. I looked into you work here. I had the same intention behind this PR: I actively use Symfony's AMQP Messenger transport myself and wanted to improve its performance, and I can see (and value) how much work you've already put into this, both in testing this PR and in tracking down the SIGTERM/exception-swallowing bug. I've updated the documentation in the stubs based on your comments — it should now be more precise about the "non-blocking with respect to the next message" semantics and the fact that a callback returning true will keep draining as long as frames keep arriving. @lstrojny, would you mind taking a look when you have a moment? Given @nicolas-grekas's comment above, this seems to have real, concrete value for a very large consumer of this extension (Symfony), so a review would be much appreciated. Ping me if any changes required. |
…me messages instead of fetching them one by one (nicolas-grekas) This PR was squashed before being merged into the 8.2 branch. Discussion ---------- [Messenger][Amqp] Add a prefetch_count option to consume messages instead of fetching them one by one | Q | A | ------------- | --- | Branch? | 8.2 | Bug fix? | no | New feature? | yes | Deprecations? | no | Issues | Fix #65920, Fix #30259 | License | MIT The AMQP transport asks the broker for one message at a time: `Connection::get()` calls `AMQPQueue::get()`, one `basic.get` per message, including when the worker wants a batch. RabbitMQ documents this as the least efficient way to consume, since every fetch is a round trip and an idle queue is polled on every loop. No consumer shows up in the management UI either, which is what #30259 reported. This adds a `prefetch_count` option. Any value greater than zero registers a consumer per queue (`basic.consume`) and lets the broker push messages as capacity frees up: ``` amqp://localhost/%2f/messages?prefetch_count=20 ``` 5000 messages, ack each, against a loopback broker, which is the *best* case for `basic.get` since there is barely any round trip time to save: ``` get 5000 msgs in 3.050s = 1639 msg/s consume 5000 msgs in 0.168s = 29803 msg/s 18.2x get 5000 msgs in 2.918s = 1713 msg/s consume 5000 msgs in 0.215s = 23302 msg/s 13.6x ``` The prefetch count is what buys the pipelining, so it has to be worth setting. With the default `--fetch-size=1`: ``` basic.get (today) 1565 msg/s prefetch_count=1 1380 msg/s prefetch_count=5 6832 msg/s prefetch_count=10 11528 msg/s prefetch_count=50 32884 msg/s ``` ### What changes when it is on - **Ordering across queues.** `getFromQueues()` round-robins the queues itself today. One `consume()` call serves every consumer of the connection (the extension routes each message back to its queue by consumer tag), so the broker decides. Publishing `L,L,L` then `H,H,H` with the high priority queue registered first delivers `L0 L1 L2 H0 H1 H2`. - **`read_timeout` becomes bounded.** It defaults to 0, which means block forever, so consuming defaults it to 1 second. That is also what makes an idle queue return, and since the worker computes its sleep from the start of the iteration, the read timeout takes the place of that sleep instead of adding to it. - **Stopping requeues more.** Prefetched but unhandled messages are unacked, so the broker redelivers them. Nothing is lost, but a large `prefetch_count` means a larger redelivery burst per stop. - `keepalive()` used to send `qos(0, 0)` for the traffic, which would drop the prefetch limit. It now resends the current one. ### Batching The consume callback cannot `yield`, so the receiver collects into an array and yields afterwards, which fits `--fetch-size`. It waits for the first message with the connection read timeout, then fills the rest of the batch with a 1 ms timeout: a partial batch returns at once rather than being held for the whole read timeout, the way `basic.get` returns as soon as a queue runs dry. ``` first get(5): [0,1,2,3,4] in 0.007s second get(5): [5,6,7,8,9] in 0.000s (already pushed, no round trip) third get(5): [10,11] in 0.000s (partial batch, not held) fourth get(5): [] in 1.000s (idle, the read timeout) ``` The 1 ms timeout is what the extension gives us today to say "take what is buffered and come back". php-amqp/php-amqp#627 adds an `AMQP_NB_CONSUME` flag that does exactly that without a timeout and without throwing, which would turn that second phase into `AMQP_JUST_CONSUME | AMQP_NB_CONSUME`. It is open since June; I built it and it works, but the bridge cannot depend on an unreleased extension, so this is a follow-up rather than a blocker. ### Also fixes the pull path The same loss reaches `basic.get`: a signal that arrives while the extension waits for the broker is dropped when that wait ends by throwing, which a read timeout on a stalled connection does. Since 8.2 the receiver implements `KeepaliveReceiverInterface` and the keepalive alarm is rescheduled by its own handler, so one lost `SIGALRM` stops the keepalive for the rest of the process's life and the messages in flight are redelivered once their TTL expires. The dance moved to a private helper used by both paths, and the added test fails without it. ### About signals `AMQPQueue::consume()` reports the read timeout by throwing, and until php/php-src#23624 the engine drops every signal that was queued while an internal function was running when that function returns with an exception pending. A worker spends essentially all of its wall time inside that call, so it would miss **every** `SIGTERM`: 653 iterations of a 10 ms read timeout and the signal never arrives, not even through an explicit `pcntl_signal_dispatch()`. Hence the `pcntl_async_signals()` dance around the blocking call, which dispatches them by hand. Measured stop latency with it, on an unpatched PHP: ``` read_timeout=0.20 [email protected]: stopped, latency 0.094s read_timeout=0.20 [email protected]: stopped, latency 0.083s read_timeout=1.00 [email protected]: stopped, latency 0.601s ``` Stop latency is bounded by `read_timeout`, which is the same contract `get()` has today. The workaround stays correct once the engine no longer drops signals, so it does not need a version guard. Kept opt-in for now, since the ordering and redelivery changes are visible. Tested against RabbitMQ 3.12.1 with ext-amqp 2.2.0; the new integration test covers the round trip. Commits ------- 1e34f3e [Messenger][Amqp] Add a prefetch_count option to consume messages instead of fetching them one by one
Add
AMQP_NB_CONSUMEflag for non-blocking, no-throw consumingSolves: (#628)
Summary
Adds an
AMQP_NB_CONSUMEflag that lets you drain an already-established consumer non-blockingly: if there is a frame buffered locally it is delivered to the callback, and if the buffer is emptyconsume()returns control to PHP immediately without throwingConsumer timeout exceed.Motivation
Today the consume loop has only two waiting modes:
read_timeout == 0→ block indefinitely;read_timeout > 0→ wait, then throwAMQPQueueException("Consumer timeout exceed").There is no way to treat "buffer is empty right now" as a normal control-flow signal. Building a self-driven main loop therefore means catching an exception on every empty poll, which is both awkward and wasteful.
AMQP_NB_CONSUMEfills exactly that gap.What changed
php_amqp.h— defineAMQP_NB_CONSUMEand register the constant.amqp_queue.c— when the flag is set, force a zerotimevalfor the internal wait, and in theAMQP_STATUS_TIMEOUTbranch return cleanly instead of throwing. All other library-exception / connection-error paths are untouched.stubs/AMQP.php,stubs/AMQPQueue.php— document the new flag.tests/— new.phptcases (see below).Semantics
AMQP_NB_CONSUMEthe internal wait uses a zerotimeval, so there is no socket blocking.AMQP_STATUS_TIMEOUTis swallowed; genuine connection/library errors are surfaced exactly as before.falsefrom the callback after taking one leaves the rest for the next call.Intended usage, combined with the existing
AMQP_JUST_CONSUME:Backward compatibility
Fully additive. Behavior is identical to before unless
AMQP_NB_CONSUMEis explicitly passed. No existing flag values change.Implementation notes
The non-blocking wait uses librabbitmq's own zero-
timevalpath rather than a rawselect(), so it does not break the library abstraction or introduce portability issues, and partial-frame reassembly is still handled internally by librabbitmq.Testing
Added
.phptcases modeled onamqpqueue_consume_timeout.phpt:AMQP_JUST_CONSUME | AMQP_NB_CONSUMEreturns immediately and throws nothing (with timing assertion to prove it does not block);