[perf] Use zmq.asyncio.Context to accelerate notify_data_update process#38
Conversation
Signed-off-by: 0oshowero0 <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]>
CLA Signature Pass0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
There was a problem hiding this comment.
Pull request overview
This PR refactors the ZMQ socket management in the storage manager to improve performance of the notify_data_update process. It switches from a persistent synchronous socket approach to a dynamic asynchronous socket pattern using zmq.asyncio.Context.
Changes:
- Replaced persistent
data_status_update_socketwith dynamic per-request socket creation innotify_data_update - Changed from synchronous zmq.Poller to async
asyncio.wait_forfor receiving ACK responses - Separated handshake socket (sync) from data update operations (async) for better resource management
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: 0oshowero0 <[email protected]>
CLA Signature Pass0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Signed-off-by: 0oshowero0 <[email protected]>
CLA Signature Pass0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Signed-off-by: 0oshowero0 <[email protected]>
CLA Signature Pass0oshowero0, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Description
Problem: The
notify_data_updatemethod was declared as async but internally used synchronouszmq.Contextand blockingzmq.Poller.poll() / socket.send_multipart() / socket.recv_multipart()calls, which blocked the asyncio event loop and degraded concurrency performance.Solution:
Changes
transfer_queue/storage/managers/base.py:
data_status_update_socketwith per-call dynamic zmq.asyncio sockets innotify_data_updatezmq.Poller-based polling withasyncio.wait_fortimeout mechanism