Multiple TCP Connections for Live Migrations - #7669
likebreath merged 22 commits into
Conversation
a39f7b7 to
b8a8caf
Compare
There was a problem hiding this comment.
Awesome! Left a few remarks. I'll refrain from approving this because I'm biased (Sebastian and Julian are my colleagues)
Please, as discussed, add specific bandwidth numbers to the PR description and the commit series, showing how awesome this work is!
0460ed6 to
001ad37
Compare
rbradford
left a comment
There was a problem hiding this comment.
This change is pretty huge! Will need some time to fully review it. I think you will probably want to regenerate your performance stats after fixing the issue I raise.
001ad37 to
63ad1f7
Compare
|
Thank you for the contribution. This is an important feature for making live migration production-ready. I have started reviewing the PR and expect to finish by tomorrow. |
likebreath
left a comment
There was a problem hiding this comment.
I still have the last three commits to review (mostly around thread teardown), but wanted to send my initial feedback. We will need an integration test for it.
| // If an error occurs in one of the memory sending threads, the thread signals | ||
| // this using this flag. Only the main thread checks this variable, the worker | ||
| // threads will be stopped in the destructor. | ||
| cancel: Arc<AtomicBool>, |
There was a problem hiding this comment.
Please rename this. Otherwise, this will later clash with the migration cancellation. For example any_worker_failed
There was a problem hiding this comment.
It is now called worker_error.
There was a problem hiding this comment.
After reviewing this patch series (particularly the later commits that fix issues in the earlier ones), I have some suggestions to make it easier to review and safer to land:
- Separate refactoring from functional changes
Extract commits that have no functional changes into their own commits (ideally at the beginning of the series when applicable). This makes it easier to verify that refactoring is purely mechanical.
- Organize the new code into a dedicated module
Consider moving the multi-connection data structures and APIs (e.g., ReceiveAdditionalConnections, SendAdditionalConnections, etc.) into a separate file (e.g., migration_transport.rs or parallel_migration.rs). This would:
- Clarify the boundary between public and private APIs
- Make the new functionality easier to review in isolation
- Reduce the size of
vmm/src/lib.rs
- Group commits by logical feature area
Consider organizing the commits in this order:
- Refactoring commits (no functional change)
- Receiver-side multi-connection support
- Integration of receiver into existing vm_receive_migration workflow
- Sender-side multi-connection support
- Integration of sender into existing vm_send_migration workflow
- CLI/API changes
This structure would make each group self-contained and independently reviewable.
Thanks again for taking on such important and substantial work!
|
@likebreath Thanks for tackling this.
And these could/should be in a separate PR for reviewability. |
|
What's the status on this? There's not been any activity recently? |
|
@amphi is preparing the requested changes and is actively working on it |
63ad1f7 to
c540247
Compare
|
Sorry for the silence, I was working on your suggestions.
I know I changed a lot, but I think everything should be more readable and easier to review now. |
1e1fd0b to
b5ef2d5
Compare
phip1611
left a comment
There was a problem hiding this comment.
Amazing work! I didn't review everything in detail yet. A few general notes:
- Is there a possibility to move more code to
vm-migration? cratevmmis already the most expensive compilation unit. Moving functionality out tovm-migrationseems very reasonable to me (helper modules etc). - I suspect you'd miss some typings then
- This PR very soon needs to rebase onto #7799
bc130f2 to
35974fc
Compare
|
It would be really great if you could add a (simplified) sequence diagram (e.g. using a mermaid diagram) showing the whole relation between
Maybe you can also create some simple basic ASCII art to embed it into the commit message. Note that adding a mermaid diagram to the code base can also be valid, if it's value-add is significant (as done in aae5d9b). |
|
I created a sequence diagram, I hope this is helpful. sequenceDiagram
title Live migration using multiple TCP connections
autonumber
participant SrcMain
participant DstMain
participant SrcSend
participant DstRecv
participant AcceptThread
participant SenderWorkers@{ "type" : "collections" }
participant ReceiverWorkers@{ "type" : "collections" }
SrcMain ->> DstMain: Start + Config
DstMain ->> DstRecv: new
DstRecv ->> AcceptThread: Spawn accept thread
SrcMain ->> SrcSend: new
SrcSend ->> SenderWorkers: Spawn sender workers
loop Accept connections
SenderWorkers ->> AcceptThread: TCP connect
AcceptThread ->> ReceiverWorkers: spawn worker
end
loop Memory Iterations
SrcMain ->> SrcSend: send_memory
loop For each chunk
SrcSend ->> SenderWorkers: Message::Memory(chunk)
SenderWorkers ->> ReceiverWorkers: Command::Memory(chunk)
ReceiverWorkers ->> SenderWorkers: Response::ok()
end
SrcSend ->> SenderWorkers: Message::Gate
SenderWorkers -->> SrcSend: Notify::Gate
SenderWorkers ->> SenderWorkers: gate.wait()
SrcSend ->> SrcSend: wait until all workers reached gate
SrcSend ->> SenderWorkers: gate.open()
end
SrcMain ->> SrcSend: cleanup()
SrcSend ->> SenderWorkers: Message::Disconnect
SrcSend ->> SrcSend: join all workers
DstMain ->> DstRecv: cleanup()
DstRecv ->> AcceptThread: signal via terminate_fd
DstRecv ->> ReceiverWorkers: signal via terminate_fd
DstRecv ->> DstRecv: join all workers
SrcMain ->> DstMain: State + Complete
|
On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
And rename it for better naming consistency. On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
This is mainly to clean up the lib.rs a bit more. On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
The memory manager is guarded by a mutex, thus parallel accesses to it and its members are not possible. But we have to execute this function in parallel when we introduce multiple TCP connections. Otherwise, the workers who receive the data and write it into guest memory will block on each other, and thus slow down the migration. Also rename the function to receive_memory_ranges for better naming consistency. On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
That way we avoid having to grab a lock when receiving a chunk of memory over the migration socket. This is a necessary prerequisite for having multiple memory receiving threads. On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
This allows accepting multiple connections in the migration receive path. On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
With this, the receiver side of a migration can wait for incoming connections, while also being able to abort the accept when the migration is done. On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
This just removes some unnecessary indirections. On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
Adds the functionality to accept multiple connections on the receiver side of a live migration. A thread listens for incoming connections and creates a worker for each new connection. On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
For sending memory over multiple connections, we need a way to split up the work. With these changes, we can chop a memory table into same-sized chunks for transmit On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
This gate behaves like a barrier, but it can be opened, meaning that threads can be released before all threads arrived at the gate. This lets us release waiting threads in case of an error, which will be important for the sender side of a live migration with multiple TCP connections. On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
Implements the functionality to send VM memory via multiple connections during a live migration. On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
At this point, we are still only using a single connection. On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
And wire everything up. From now on the multiple connections feature can be used. This commit series is heavily based on Julian Stecklina's work, so kudos to him! Co-authored-by: Julian Stecklina <[email protected]> On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
Validates that there are no conflicting options set, and that the destination URL is valid. On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
41a4a29 to
d2736ed
Compare
likebreath
left a comment
There was a problem hiding this comment.
As I am reviewing the new connection-capping logic on the receiver side, I realized that we are not properly handling the error path from ReceiveAdditionalConnections::accept_connections().
When accept_connections breaks out of the loop on error (e.g. hitting MAX_MIGRATION_CONNECTIONS), it joins all worker threads without first signaling terminate_fd. The workers are blocked in wait_for_readable() with an infinite timeout. While they will unblock when the sender closes its sockets, by design we should not rely on that. The fix should be pretty straight-forward: signal terminate_fd before joining when breaking on error, say:
if first_err.is_err() {
let _ = terminate_fd.write(1);
}
info!("Stopped accepting additional connections. Cleaning up threads.");I believe this is the last issue we need to address before landing this one. Mark it as a "Request changes" to avoid merging by accident.
Check that the amount of parallel connections does not exceed 128 and update documentation. On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
d2736ed to
0ea1127
Compare
likebreath
left a comment
There was a problem hiding this comment.
@amphi Awesome. Thanks again for the good work. Now ready to land.
|
Fantastic! Well done everyone |
Summary
This PR implements VM live migrations using multiple TCP connections. Most of this work is taken from @blitz , so kudos to him!
The send-migration HTTP command now accepts a
connectionsparameter (defaults to 1) that specifies how many connections to use for live migration.Benchmarks
We did a quick test on two of our servers which have a 100G connection. We transferred a VM with 50GB RAM, here are our results:
We also ran iperf between the two machines and got a throughput of 11.5 GiB/s, so I'd say the feature works pretty good.
Hint: MiB/s is Mebibyte per Second, GiB is Gibibyte per second
Design
Diagram/Overview: #7669 (comment)
If
connectionsis larger than 1, the sender will attempt to establish additional TCP connections to the same migration destination. The main (initial) connection handles most of the migration protocol. The additional connections handle onlyMemorycommands for transferring chunks of VM memory.For each additional connection, a thread is created that receives chunks of memory from the main thread, and sends those chunks to the receiver.
For each iteration of sending memory, the
MemoryTablethat describes dirty memory will be split into chunks of fixed size (CHUNK_SIZE). These chunks will then be distributed among the available threads using an MPSC channel wrapped in a Mutex. This channel can have a configurable backlog of outstanding chunks to send (BUFFERED_REQUESTS_PER_THREAD). This is currently 64 chunks per thread to keep the memory consumption at a sensible level (Otherwise, for VMs with a huge amount of memory, this may take up a lot of additional memory).We still use the original request-response scheme. Since we don't pipeline requests, but always wait for the other side to acknowledge them, we have a fundamental limit on throughput that we can reach. The original code only expected one ACK for the whole dirty memory table. We now have one ACK per chunk.
@blitz came up with this formula of the upper bound of throughput per connection:
effective_throughput = chunk_size / (chunk_size / throughput_per_connection + round_trip_time)This formula is also in the code. We've played around with this and with large enough chunks, the impact seems negligible, especially since we can scale up connections. Feel free to plug in your favorite numbers.