ext/pcntl: do not drop queued signals when an exception is pending - #23624
ext/pcntl: do not drop queued signals when an exception is pending#23624nicolas-grekas wants to merge 2 commits into
Conversation
23bfbd0 to
36d571c
Compare
|
Thanks I ll have a look some time in the following days, I would say tough after a quick look there are incoming changes needed. |
|
Is #22538 related? |
|
@TimWolla Related in that #22538 rewrites the same function, but it neither causes nor fixes this, and it does not help the case that made me look. I built it (master, 8.6.0-dev) with pecl/amqp 2.2.0 on top (plus php-amqp#638 to get that compiling) and ran the same worker loop: Two reasons. #22538 runs handlers on So whichever lands first, the other needs the same two guards, setting the exception aside while handlers run and keeping what a throwing handler leaves behind. I can port them onto #22538 once it settles. @devnexen Thanks. Meanwhile I found a problem in my own change and reworked it in 17ff74f: |
00048a7 to
5bfec10
Compare
|
Hitting same issue with Symfony Messenger and keepalive. What happens, step by step:
|
5bfec10 to
6fd9608
Compare
6fd9608 to
da5fb61
Compare
|
I'm not a fan of letting signal handlers execute after one of them throws. Throwing should mean "unwind the call stack, execute only catch/finally blocks". But here we are letting adjacent signal handlers execute. This problem may be fixed in the same way as GH-22538: Execute signal handlers on EINTR. In the case of |
…n pending ZEND_DO_FCALL runs its interrupt check right after an internal function returns, before the pending exception is handled, so pcntl_interrupt_function() reaches the dispatcher with EG(exception) set. call_user_function() returns without calling anything in that state, the "if (EG(exception)) break" added by 296fad1 fires on the first entry, and the drain loop then recycles the whole queue without a single handler having run. The signal is destroyed rather than delayed: a later pcntl_signal_dispatch() finds nothing left. Set the exception aside while the handlers run and chain it back afterwards, the way zend_objects_destroy_object() does for destructors called during unwinding. EG(opline_before_exception) is saved along with it, since ZEND_HANDLE_EXCEPTION derives the throwing op, and from it the enclosing try block, out of it. Any long blocking internal call that throws on timeout reaches this. pecl/amqp throws "Consumer timeout exceed" out of AMQPQueue::consume(), which makes a Symfony messenger worker miss every SIGTERM whatever the timeout is. PDO/SQLite throws "database is locked" once busy_timeout expires, which kills a keepalive SIGALRM for the rest of the process's life.
When a handler threw, the signals queued behind it were recycled without ever being delivered. Put them back on the queue instead, and re-arm the interrupt so that the engine dispatches them once the exception has been handled, rather than leaving them to wait for another signal to come in. Not calling further handlers while the exception propagates is unchanged.
da5fb61 to
f249e7c
Compare
|
@arnaud-lb I split the patch in two: the 1st is the minimum fix. It cannot be tested with a .phpt because it needs an internal function that has a signal delivered while it runs and then throws. The 2nd commit is the part you object to. It isn't unprecedented: destructors already behave that way. If one destructor throws during unwinding, the remaining ones still run, immediately, with the exceptions chained:
The 2nd commit is more conservative than that: it stops calling handlers as soon as one throws, and delivers what is left only once the exception has been handled. There has to be a fix in php-src: PDO/SQLite hits the same defect (comment above), so a third extension is already queued up for the same workaround, etc. That's what tells me where the fix belongs. On GH-22538 and my 2nd commit: it strengthens the case rather than replacing it. GH-22538 makes throwing from a handler the documented way to interrupt a syscall, so "a handler threw while other signals were queued" becomes routine, and its dispatcher still drains those ( |
…me messages instead of fetching them one by one (nicolas-grekas) This PR was squashed before being merged into the 8.2 branch. Discussion ---------- [Messenger][Amqp] Add a prefetch_count option to consume messages instead of fetching them one by one | Q | A | ------------- | --- | Branch? | 8.2 | Bug fix? | no | New feature? | yes | Deprecations? | no | Issues | Fix #65920, Fix #30259 | License | MIT The AMQP transport asks the broker for one message at a time: `Connection::get()` calls `AMQPQueue::get()`, one `basic.get` per message, including when the worker wants a batch. RabbitMQ documents this as the least efficient way to consume, since every fetch is a round trip and an idle queue is polled on every loop. No consumer shows up in the management UI either, which is what #30259 reported. This adds a `prefetch_count` option. Any value greater than zero registers a consumer per queue (`basic.consume`) and lets the broker push messages as capacity frees up: ``` amqp://localhost/%2f/messages?prefetch_count=20 ``` 5000 messages, ack each, against a loopback broker, which is the *best* case for `basic.get` since there is barely any round trip time to save: ``` get 5000 msgs in 3.050s = 1639 msg/s consume 5000 msgs in 0.168s = 29803 msg/s 18.2x get 5000 msgs in 2.918s = 1713 msg/s consume 5000 msgs in 0.215s = 23302 msg/s 13.6x ``` The prefetch count is what buys the pipelining, so it has to be worth setting. With the default `--fetch-size=1`: ``` basic.get (today) 1565 msg/s prefetch_count=1 1380 msg/s prefetch_count=5 6832 msg/s prefetch_count=10 11528 msg/s prefetch_count=50 32884 msg/s ``` ### What changes when it is on - **Ordering across queues.** `getFromQueues()` round-robins the queues itself today. One `consume()` call serves every consumer of the connection (the extension routes each message back to its queue by consumer tag), so the broker decides. Publishing `L,L,L` then `H,H,H` with the high priority queue registered first delivers `L0 L1 L2 H0 H1 H2`. - **`read_timeout` becomes bounded.** It defaults to 0, which means block forever, so consuming defaults it to 1 second. That is also what makes an idle queue return, and since the worker computes its sleep from the start of the iteration, the read timeout takes the place of that sleep instead of adding to it. - **Stopping requeues more.** Prefetched but unhandled messages are unacked, so the broker redelivers them. Nothing is lost, but a large `prefetch_count` means a larger redelivery burst per stop. - `keepalive()` used to send `qos(0, 0)` for the traffic, which would drop the prefetch limit. It now resends the current one. ### Batching The consume callback cannot `yield`, so the receiver collects into an array and yields afterwards, which fits `--fetch-size`. It waits for the first message with the connection read timeout, then fills the rest of the batch with a 1 ms timeout: a partial batch returns at once rather than being held for the whole read timeout, the way `basic.get` returns as soon as a queue runs dry. ``` first get(5): [0,1,2,3,4] in 0.007s second get(5): [5,6,7,8,9] in 0.000s (already pushed, no round trip) third get(5): [10,11] in 0.000s (partial batch, not held) fourth get(5): [] in 1.000s (idle, the read timeout) ``` The 1 ms timeout is what the extension gives us today to say "take what is buffered and come back". php-amqp/php-amqp#627 adds an `AMQP_NB_CONSUME` flag that does exactly that without a timeout and without throwing, which would turn that second phase into `AMQP_JUST_CONSUME | AMQP_NB_CONSUME`. It is open since June; I built it and it works, but the bridge cannot depend on an unreleased extension, so this is a follow-up rather than a blocker. ### Also fixes the pull path The same loss reaches `basic.get`: a signal that arrives while the extension waits for the broker is dropped when that wait ends by throwing, which a read timeout on a stalled connection does. Since 8.2 the receiver implements `KeepaliveReceiverInterface` and the keepalive alarm is rescheduled by its own handler, so one lost `SIGALRM` stops the keepalive for the rest of the process's life and the messages in flight are redelivered once their TTL expires. The dance moved to a private helper used by both paths, and the added test fails without it. ### About signals `AMQPQueue::consume()` reports the read timeout by throwing, and until php/php-src#23624 the engine drops every signal that was queued while an internal function was running when that function returns with an exception pending. A worker spends essentially all of its wall time inside that call, so it would miss **every** `SIGTERM`: 653 iterations of a 10 ms read timeout and the signal never arrives, not even through an explicit `pcntl_signal_dispatch()`. Hence the `pcntl_async_signals()` dance around the blocking call, which dispatches them by hand. Measured stop latency with it, on an unpatched PHP: ``` read_timeout=0.20 [email protected]: stopped, latency 0.094s read_timeout=0.20 [email protected]: stopped, latency 0.083s read_timeout=1.00 [email protected]: stopped, latency 0.601s ``` Stop latency is bounded by `read_timeout`, which is the same contract `get()` has today. The workaround stays correct once the engine no longer drops signals, so it does not need a version guard. Kept opt-in for now, since the ordering and redelivery changes are visible. Tested against RabbitMQ 3.12.1 with ext-amqp 2.2.0; the new integration test covers the round trip. Commits ------- 1e34f3e [Messenger][Amqp] Add a prefetch_count option to consume messages instead of fetching them one by one
pcntl_signal_dispatch()detaches the whole queue fromPCNTL_G(head)before it starts calling handlers, and recycles every entry it walks over. Two paths let queued signals disappear.1. Dispatching while an exception is pending
ZEND_VM_FCALL_INTERRUPT_CHECK()runs right after an internal function returns, before the pending exception is handled. So when an internal function throws,pcntl_interrupt_function()reaches the dispatcher withEG(exception)set. There,zend_call_function()returns without calling anything, theif (EG(exception)) break;added by 296fad1 fires on the very first entry, and the/* drain the remaining */loop recycles the entire queue. No handler ever runs, and nothing is left for a laterpcntl_signal_dispatch()either.Any signal that arrives while such a function is running is therefore lost, not delayed. The fix sets the exception aside for the duration of the dispatch, the way
zend_objects_destroy_object()does around destructors called during unwinding.zend_exception_save()is not usable for that: it goes through the singleEG(prev_exception)slot, so a dispatch happening inside an autoloader called with an exception set aside would hand that exception back too early.2. Signals queued behind a throwing handler
If a handler throws, the signals queued behind it were recycled as well. They now go back to the queue, and with asynchronous signals the interrupt is re-armed, so the engine delivers them on its own once the exception is handled rather than waiting for another signal to come in. The existing behaviour of not calling further handlers while the exception propagates is unchanged, and
pcntl_signal_dispatch_exception.phptstill passes as is.How I ran into it
Any long blocking internal call that throws on timeout hits case 1. pecl/amqp is one:
AMQPQueue::consume()throwsAMQPQueueException("Consumer timeout exceed")when the connection read timeout expires. A Symfony Messenger worker built on it (symfony/symfony#65920) misses everySIGTERM, whatever the timeout is, because the worker spends essentially all of its wall time inside that call:653 iterations of a 10 ms call and the signal is never delivered, not even by an explicit
pcntl_signal_dispatch()afterwards.SigBlk,SigIgnandSigCgtread from inside the extension's callback confirm the signal is neither blocked nor ignored and that the handler is installed, and the same signal duringsleep(),stream_select(),stream_socket_accept()orsocket_read()is delivered normally. What makesconsume()different is only that it throws: when a message arrives and it returns normally instead, the signal is delivered.With this patch, same build, same test, no userland workaround:
I verified this against pecl/amqp 2.2.0 built on this branch, with and without the patch. Case 2 is covered by the two added
.phpts, one with an explicit second dispatch and one where the engine has to come back on its own; case 1 needs an internal function that blocks long enough for a signal to arrive and then throws, which I could not build out of core functions alone, since re-entering the VM for any userland callback dispatches the signal first.ext/pcntl,Zend/testsandext/standard/testsare green (8180 passed, 0 failed).