Do not block writing to wake socket #1767
Conversation
|
I've run a number of medium-term (~15min) tests, and haven't had any However, I am only getting ~25% of the performance compared to 1.4.5 (with #1765). (That's an extra reduction on top of the performance drop I see going from 1.4.4 -> 1.4.5) There seem to be frequent pauses, which are suspiciously always (that I've seen) just under 30s. For example: |
|
Done some more tests to try and isolate the slowdown. As noted in the original ticket discussion, I'm using this (very WIP, hacky) project to do the testing. Specifically, the On a particular cluster/topic I'm using for testing, this version pipes messages a rate of ~500/s, down from ~2k/s on 1.4.5, and ~5k/s on 1.4.4. CPU usage spikes to 100%, but drops to essentially 0% for long periods. Presumably this is related to the pauses observed above. Note that this happens even with the produce buffer is large enough to avoid ever explicitly waiting for a produce future. Using the Using the |
|
Ok -- backpressure clearly needed in your setup. |
Can you clarify what you mean here? Perhaps I'm just a bit dense tonight, but I read through the linked PR's and not understanding whether you mean @braedon needs to add backpressure within his test harness or that backpressure needs to be added to |
|
Neither - the blocking wake socket currently provides backpressure. This PR
made it non blocking and so removed the backpressure. So this is a non
starter for now.
…On Sun, Mar 31, 2019, 7:32 PM Jeff Widman ***@***.***> wrote:
backpressure clearly needed in your setup.
Can you clarify what you mean here? Perhaps I'm just a bit dense tonight,
but I read through the linked PR's and not understanding whether you mean
@braedon <https://github.com/braedon> needs to add backpressure within
his test harness or that backpressure needs to be added to kafka-python
somehow...
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<#1767 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAzetMRjEg9ykdqGyX9yPUjQZ9LAP-zbks5vcW-xgaJpZM4cRrN9>
.
|
Fixes #1760 . We do not need to block on writes to the wakeup socket pair. This should avoid the root cause of #1760, which is that the socket buffer gets full and blocks the thread that is responsible for reading / draining it.
This change is