[adb::any::sync] Make operation fetching concurrent #1323
Conversation
| (sink, stream) | ||
| } | ||
| Err(e) => { | ||
| error!(error = %e, "failed to establish connection, exiting"); |
There was a problem hiding this comment.
We used to re-try on connection errors but I think it migt actually be better to just fail. We don't want the client to keep trying to reconnect indefinitely, and the client doesn't have any retry or timeout logic to help it handle an interrupted connection to a server. I think for the sake of this example it's alright to just give up if the connection fails.
There was a problem hiding this comment.
Seems fine here as this is example code. But can the client user configure a per-request timeout or similar if someone wanted to create a more robust app?
There was a problem hiding this comment.
Pull Request Overview
This PR implements concurrent operation fetching for the ADB sync protocol to resolve issue #1214. The changes transform the previously sequential sync client into a concurrent one that can handle multiple outstanding requests simultaneously.
- Replaced sequential operation fetching with a concurrent model using
FuturesUnordered - Refactored the
Resolvertrait to support concurrency withSend + Sync + 'staticbounds - Introduced
AnyResolverstruct to wrap database access inArc<RwLock<>>for thread safety
Reviewed Changes
Copilot reviewed 9 out of 11 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| storage/src/adb/any/sync/resolver.rs | Adds concurrency bounds to Resolver trait and introduces AnyResolver wrapper |
| storage/src/adb/any/sync/mod.rs | Removes TODO comment and adds new error types for concurrent operations |
| storage/src/adb/any/sync/metrics.rs | New metrics module for tracking sync performance |
| storage/src/adb/any/sync/client.rs | Major refactor to support concurrent operation fetching with gap-finding algorithm |
| storage/fuzz/fuzz_targets/adb_sync.rs | Updates to use new AnyResolver wrapper |
| examples/sync/src/resolver.rs | Refactors resolver to use concurrent I/O with request correlation |
| examples/sync/src/protocol.rs | Adds RequestId for correlating concurrent requests and responses |
| examples/sync/src/lib.rs | Adds error module export |
| examples/sync/src/error.rs | New error handling module for the sync example |
Comments suppressed due to low confidence (1)
examples/sync/src/resolver.rs:139
- [nitpick] The field name
_phantomis not descriptive. Consider renaming to_runtimeor_context_phantomto better indicate what phantom data this represents.
_phantom: PhantomData<E>,
There was a problem hiding this comment.
I think the async pattern you are now using in the client seems reasonable (I never mind a good swing at making an explicit state machine if the number of states is well-contained).
I'm not sure it'll work for other crates where there are many more states we could be in (like consensus) but if we can simplify it to this here, we should.
examples/sync/src/resolver.rs
Outdated
| } | ||
|
|
||
| /// Send a request and receive a response using the persistent connection. | ||
| async fn send_request(&self, request: Message) -> Result<Message, ResolverError> { |
There was a problem hiding this comment.
By concurrent here, I mean "interleaved" requests (outstanding requests for multiple ranges).
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## main #1323 +/- ##
==========================================
+ Coverage 91.30% 91.37% +0.06%
==========================================
Files 250 251 +1
Lines 63382 63583 +201
==========================================
+ Hits 57872 58096 +224
+ Misses 5510 5487 -23
Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
This PR resolves #1214.