Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feature: non-blocking consume added - #627

Open
adapik wants to merge 2 commits into
php-amqp:latestfrom
adapik:dev/non-blocking-consume
Open

feature: non-blocking consume added#627
adapik wants to merge 2 commits into
php-amqp:latestfrom
adapik:dev/non-blocking-consume

Conversation

@adapik

@adapik adapik commented Jun 4, 2026

Copy link
Copy Markdown

Add AMQP_NB_CONSUME flag for non-blocking, no-throw consuming

Solves: (#628)

Summary

Adds an AMQP_NB_CONSUME flag that lets you drain an already-established consumer non-blockingly: if there is a frame buffered locally it is delivered to the callback, and if the buffer is empty consume() returns control to PHP immediately without throwing Consumer timeout exceed.

Motivation

Today the consume loop has only two waiting modes:

  • read_timeout == 0 → block indefinitely;
  • read_timeout > 0 → wait, then throw AMQPQueueException("Consumer timeout exceed").

There is no way to treat "buffer is empty right now" as a normal control-flow signal. Building a self-driven main loop therefore means catching an exception on every empty poll, which is both awkward and wasteful. AMQP_NB_CONSUME fills exactly that gap.

What changed

  • php_amqp.h — define AMQP_NB_CONSUME and register the constant.
  • amqp_queue.c — when the flag is set, force a zero timeval for the internal wait, and in the AMQP_STATUS_TIMEOUT branch return cleanly instead of throwing. All other library-exception / connection-error paths are untouched.
  • stubs/AMQP.php, stubs/AMQPQueue.php — document the new flag.
  • tests/ — new .phpt cases (see below).

Semantics

  • With AMQP_NB_CONSUME the internal wait uses a zero timeval, so there is no socket blocking.
  • A buffered frame is delivered to the callback as usual, with no socket round-trip.
  • An empty buffer returns control to PHP silently — no exception.
  • Only AMQP_STATUS_TIMEOUT is swallowed; genuine connection/library errors are surfaced exactly as before.
  • QoS is respected: if prefetch left several messages buffered, returning false from the callback after taking one leaves the rest for the next call.

Intended usage, combined with the existing AMQP_JUST_CONSUME:

$queue->consume(null); // subscribe once

while ($running) {
    $msg = null;
    $queue->consume(function (AMQPEnvelope $e) use (&$msg) {
        $msg = $e;
        return false;
    }, AMQP_JUST_CONSUME | AMQP_NB_CONSUME);

    if ($msg !== null) {
        // handle + ack
    }
    // other work / signals / limits
}

Backward compatibility

Fully additive. Behavior is identical to before unless AMQP_NB_CONSUME is explicitly passed. No existing flag values change.

Implementation notes

The non-blocking wait uses librabbitmq's own zero-timeval path rather than a raw select(), so it does not break the library abstraction or introduce portability issues, and partial-frame reassembly is still handled internally by librabbitmq.

Testing

Added .phpt cases modeled on amqpqueue_consume_timeout.phpt:

  • empty buffer with AMQP_JUST_CONSUME | AMQP_NB_CONSUME returns immediately and throws nothing (with timing assertion to prove it does not block);
  • a previously published message is delivered to the callback in non-blocking mode.

@nicolas-grekas nicolas-grekas left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I needed exactly this while adding push consumption to Symfony's AMQP transport (symfony/symfony#65924), so I built this branch against PHP 8.4 and put it through the cases I care about. It does what it says:

1. empty buffer, JUST_CONSUME|NB_CONSUME:  returned in 0.0002s, no exception
   same call without the flag:             1.0003s, AMQPQueueException "Consumer timeout exceed"
2. 8 messages already pushed:              all 8 delivered in 0.0004s
3. 100 bodies of 10B to 700KB (many
   frames each), drained through NB:       intact and in order
4. after disconnect():                     AMQPChannelException still raised

Point 4 matters: only AMQP_STATUS_TIMEOUT is swallowed, so a genuine failure is still surfaced. And 32768 is the next free bit after AMQP_REQUEUE, with no overlap with the flags basic.consume itself reads (AMQP_NOLOCAL, AMQP_AUTOACK, AMQP_EXCLUSIVE).

Why this is worth having

Without it, "the buffer is empty" can only be learned by catching an exception, and that turns out to cost more than the ugliness. Until php/php-src#23624, the engine drops every signal queued while an internal function was running when that function returns with an exception pending: ZEND_VM_FCALL_INTERRUPT_CHECK() sits right after an internal call returns, before the exception is handled, so pcntl_signal_dispatch() runs with EG(exception) set, zend_call_function() no-ops, and the queued signals are recycled without a single handler running.

A worker looping on consume() spends essentially all of its wall time inside that call, so it misses every SIGTERM:

read_timeout=0.20s, SIGTERM at 1.70s: 31 rounds, never stopped
read_timeout=0.05s, SIGTERM at 2.13s: 121 rounds, never stopped
read_timeout=0.01s, SIGTERM at 2.57s: 653 rounds, never stopped

653 iterations of a 10 ms read timeout and the signal never arrives, not even through an explicit pcntl_signal_dispatch() afterwards. Not blocked and not ignored: SigBlk, SigIgn and SigCgt read from inside the consume callback confirm the handler is installed, and the same signal during sleep() or stream_select() is delivered normally. What makes consume() different is only that it throws, which is why a no-throw drain is not just cosmetic.

One doc suggestion

AMQP_NB_CONSUME is non-blocking with respect to the next message, not mid-message: once the method frame is read, amqp_read_message() waits for the header and body frames with no timeout. That is the right behaviour, and it is what keeps the 700KB bodies in my test 3 intact, but the stub currently reads as an unqualified "returns immediately", which could surprise someone on a slow link. Something like "returns as soon as no further message is buffered; a message already being received is still read to completion" would set the expectation better.

Worth noting too that with a callback returning true the flag does not mean "drain what is buffered right now and stop": it keeps going while frames keep arriving. All 100 messages in my test 3 came back in a single call. That is fine, just not obvious from the name.

Nothing blocking from me. Happy to see this land, it would let the Symfony bridge drop a setReadTimeout(0.001) workaround.

@adapik
adapik force-pushed the dev/non-blocking-consume branch from 7e1cca7 to f6d1377 Compare September 11, 2026 12:25
@adapik

adapik commented Sep 12, 2026

Copy link
Copy Markdown
Author

Thanks a lot for the thorough review, @nicolas-grekas! Really appreciate you putting the branch through such a rigorous set of tests, including the signal-handling investigation — that's a great catch and explains a real-world pain point.

I looked into you work here. I had the same intention behind this PR: I actively use Symfony's AMQP Messenger transport myself and wanted to improve its performance, and I can see (and value) how much work you've already put into this, both in testing this PR and in tracking down the SIGTERM/exception-swallowing bug.

I've updated the documentation in the stubs based on your comments — it should now be more precise about the "non-blocking with respect to the next message" semantics and the fact that a callback returning true will keep draining as long as frames keep arriving.

@lstrojny, would you mind taking a look when you have a moment? Given @nicolas-grekas's comment above, this seems to have real, concrete value for a very large consumer of this extension (Symfony), so a review would be much appreciated. Ping me if any changes required.

nicolas-grekas added a commit to symfony/symfony that referenced this pull request Sep 13, 2026
…me messages instead of fetching them one by one (nicolas-grekas)

This PR was squashed before being merged into the 8.2 branch.

Discussion
----------

[Messenger][Amqp] Add a prefetch_count option to consume messages instead of fetching them one by one

| Q             | A
| ------------- | ---
| Branch?       | 8.2
| Bug fix?      | no
| New feature?  | yes
| Deprecations? | no
| Issues        | Fix #65920, Fix #30259
| License       | MIT

The AMQP transport asks the broker for one message at a time: `Connection::get()` calls `AMQPQueue::get()`, one `basic.get` per message, including when the worker wants a batch. RabbitMQ documents this as the least efficient way to consume, since every fetch is a round trip and an idle queue is polled on every loop. No consumer shows up in the management UI either, which is what #30259 reported.

This adds a `prefetch_count` option. Any value greater than zero registers a consumer per queue (`basic.consume`) and lets the broker push messages as capacity frees up:

```
amqp://localhost/%2f/messages?prefetch_count=20
```

5000 messages, ack each, against a loopback broker, which is the *best* case for `basic.get` since there is barely any round trip time to save:

```
get       5000 msgs in 3.050s =  1639 msg/s
consume   5000 msgs in 0.168s = 29803 msg/s   18.2x
get       5000 msgs in 2.918s =  1713 msg/s
consume   5000 msgs in 0.215s = 23302 msg/s   13.6x
```

The prefetch count is what buys the pipelining, so it has to be worth setting. With the default `--fetch-size=1`:

```
basic.get (today)         1565 msg/s
prefetch_count=1          1380 msg/s
prefetch_count=5          6832 msg/s
prefetch_count=10        11528 msg/s
prefetch_count=50        32884 msg/s
```

### What changes when it is on

- **Ordering across queues.** `getFromQueues()` round-robins the queues itself today. One `consume()` call serves every consumer of the connection (the extension routes each message back to its queue by consumer tag), so the broker decides. Publishing `L,L,L` then `H,H,H` with the high priority queue registered first delivers `L0 L1 L2 H0 H1 H2`.
- **`read_timeout` becomes bounded.** It defaults to 0, which means block forever, so consuming defaults it to 1 second. That is also what makes an idle queue return, and since the worker computes its sleep from the start of the iteration, the read timeout takes the place of that sleep instead of adding to it.
- **Stopping requeues more.** Prefetched but unhandled messages are unacked, so the broker redelivers them. Nothing is lost, but a large `prefetch_count` means a larger redelivery burst per stop.
- `keepalive()` used to send `qos(0, 0)` for the traffic, which would drop the prefetch limit. It now resends the current one.

### Batching

The consume callback cannot `yield`, so the receiver collects into an array and yields afterwards, which fits `--fetch-size`. It waits for the first message with the connection read timeout, then fills the rest of the batch with a 1 ms timeout: a partial batch returns at once rather than being held for the whole read timeout, the way `basic.get` returns as soon as a queue runs dry.

```
first get(5):  [0,1,2,3,4] in 0.007s
second get(5): [5,6,7,8,9] in 0.000s   (already pushed, no round trip)
third get(5):  [10,11]     in 0.000s   (partial batch, not held)
fourth get(5): []          in 1.000s   (idle, the read timeout)
```

The 1 ms timeout is what the extension gives us today to say "take what is buffered and come back". php-amqp/php-amqp#627 adds an `AMQP_NB_CONSUME` flag that does exactly that without a timeout and without throwing, which would turn that second phase into `AMQP_JUST_CONSUME | AMQP_NB_CONSUME`. It is open since June; I built it and it works, but the bridge cannot depend on an unreleased extension, so this is a follow-up rather than a blocker.

### Also fixes the pull path

The same loss reaches `basic.get`: a signal that arrives while the extension waits for the broker is dropped when that wait ends by throwing, which a read timeout on a stalled connection does. Since 8.2 the receiver implements `KeepaliveReceiverInterface` and the keepalive alarm is rescheduled by its own handler, so one lost `SIGALRM` stops the keepalive for the rest of the process's life and the messages in flight are redelivered once their TTL expires. The dance moved to a private helper used by both paths, and the added test fails without it.

### About signals

`AMQPQueue::consume()` reports the read timeout by throwing, and until php/php-src#23624 the engine drops every signal that was queued while an internal function was running when that function returns with an exception pending. A worker spends essentially all of its wall time inside that call, so it would miss **every** `SIGTERM`: 653 iterations of a 10 ms read timeout and the signal never arrives, not even through an explicit `pcntl_signal_dispatch()`. Hence the `pcntl_async_signals()` dance around the blocking call, which dispatches them by hand. Measured stop latency with it, on an unpatched PHP:

```
read_timeout=0.20 [email protected]: stopped, latency 0.094s
read_timeout=0.20 [email protected]: stopped, latency 0.083s
read_timeout=1.00 [email protected]: stopped, latency 0.601s
```

Stop latency is bounded by `read_timeout`, which is the same contract `get()` has today. The workaround stays correct once the engine no longer drops signals, so it does not need a version guard.

Kept opt-in for now, since the ordering and redelivery changes are visible. Tested against RabbitMQ 3.12.1 with ext-amqp 2.2.0; the new integration test covers the round trip.

Commits
-------

1e34f3e [Messenger][Amqp] Add a prefetch_count option to consume messages instead of fetching them one by one
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants