vmm: migration: add user-configurable downtime and timeout - #7835
Conversation
b0c61b3 to
ee8ecd4
Compare
96b700b to
32c757e
Compare
32c757e to
bc5f383
Compare
2c4dd21 to
fcbca39
Compare
1663324 to
3487c99
Compare
9341d8f to
3f60a47
Compare
likebreath
left a comment
There was a problem hiding this comment.
Beyond the comments below — what are your thoughts on adding integration tests to cover the new parameters? My initial thought is we could extend the existing live migration tests to override the default values for both positive cases (migration completes successfully within the configured downtime/timeout) and negative cases (migration fails or cancels when the timeout is reached).
This would give us confidence that the convergence logic and timeout strategy behave as expected.
| .subcommand_matches("send-migration") | ||
| .unwrap() | ||
| .get_one::<TimeoutStrategy>("timeout-strategy") | ||
| .unwrap(), |
There was a problem hiding this comment.
I understand it has been a legacy issue since --local was introduced, but with three new parameters it's reaching a point where it should be properly addressed.
I'd suggest: implement VmSendMigrationData::parse(cmd: &str) -> Result<Self> so that ch-remote passes a single comma-separated string rather than individual --flags.
The CLI would become:
$ ch-remote --api-socket=/tmp/api send-migration tcp:{dst}:{port},local=false,downtime_ms=300,timeout_s=60,timeout_strategy=cancel
instead of:
$ ch-remote --api-socket=/tmp/api send-migration tcp:{dst}:{port} --downtime-ms 300 --timeout-s 60 --timeout-strategy cancel
This keeps send-migration consistent with how the rest of ch-remote and cloud-hypervisor handle commandline options - using the option_parser comma-separated style. Having send-migration be the only subcommand using individual --flags creates an inconsistency for both users and contributors.
Beyond consistency, this would also:
- Eliminate the duplicated code from ch-remote;
- Use the exact data structure defined from vmm crate say (avoid defining a separate
TimeoutStrategyenum in ch-remote); - Make future parameter additions a one-line change instead of touching multiple call sites.
I understand this is a breaking change for --local, but I think now is the right time — before more parameters accumulate and the breaking change becomes harder to make.
There was a problem hiding this comment.
While I agree that duplicate types are not ideal, I would argue that those types are only duplicates in name, not in function. User APIs and internal representation have different requirements and mixing both in a single type leads usability or type safety conflicts like the Duration one discussed above.
While the VmSendMigrationData::parse(cmd: &str) -> Result<Self> approach does separate these concerns, it skips some of the nice things clap already provides like the error messages.
Compare, for example, these two error messages caused by a trailing comma:
> cargo run --bin ch-remote -- --api-socket=/tmp/chv1.sock add-device path=foo,
[2026-03-17T08:06:12Z ERROR cloud_hypervisor] Fatal error: AddDeviceConfig(ParseDevice(UnknownOption("")))
Error: ch-remote exited with the following chain of errors:
0: Error parsing device syntax
1: Error parsing --device
2: unknown option:
> cargo run --bin ch-remote -- --api-socket=/tmp/chv1.sock send-migration tcp:127.0.0.1:1337 --downtime-ms 200,
error: invalid value '200,' for '--downtime-ms <downtime-ms>': invalid digit found in string
For more information, try '--help'.From what I can see, VmSendMigrationData::parse(cmd: &str) -> Result<Self> also requires significant manual implementation.
I don't know the reasons for why this API style was chosen though, so it's very possible I'm missing important points.
There was a problem hiding this comment.
I'm with @arctic-alpaca that clap's convenience is outstanding. That being said, I'm with you that we should stay consistent. We could refactor this in the future eventually.
There was a problem hiding this comment.
Right. Let's fix the inconsistency issue here and follow-up on broader CLI changes in #7551.
| 1. **local migration**: Migrating a VM from one Cloud Hypervisor instance to another on the same machine; also called | ||
| UNIX socket migration. | ||
| 1. **TCP migration**: migrating a VM between two TCP/IP hosts. |
There was a problem hiding this comment.
I think the original terminology was more precise. Transport mechanism (TCP vs UNIX socket) is orthogonal to migration use cases ("remote" vs "local"), say:
- TCP to
127.0.0.1is a local migration over TCP - A UNIX socket forwarded by a management layer could serve a remote migration.
I'd suggest keeping the original naming.
There was a problem hiding this comment.
thanks - changed that.
|
I think we are good to go - integration test included! |
Great. I will get to it later today (and the multi-TCP next week). |
d595bab to
82efedf
Compare
likebreath
left a comment
There was a problem hiding this comment.
@phip1611 Thank you again for the excellent work. We now have a proper, user-configurable mechanism for managing live migration convergence - a critical feature towards making Live Migration production-ready.
|
Think I solved it in the latest commit |
607b270 to
d236565
Compare
d236565 to
2fc1595
Compare
|
@rbradford Can we get this over the finish line today? |
rbradford
left a comment
There was a problem hiding this comment.
Please go through the documentation and double check it's correct. I'm not 100% sure it is.
| format!("--api-socket={}", &src_api_socket), | ||
| "send-migration".to_string(), | ||
| format! {"unix:{migration_socket}"}, | ||
| format!( | ||
| "destination_url=unix:{migration_socket},local={}", |
There was a problem hiding this comment.
nit: I think this should probably be integrated into the previous commit since it breaks test bisect. I think it's fine to add new test in a later commit but adapting them (especially when the change is so simple) probably should be in the commit that changes it.
| /// Cancel the migration and keep the VM running on the source. | ||
| Cancel, | ||
| /// Force the migration and ignore any downtime requirement. | ||
| Force, |
There was a problem hiding this comment.
nit: I feel like Ignore is the better terminology here? It's ignoring the timeout vs forcing some behaviour. But this would be an annoying change for you to make so I could live with it.
There was a problem hiding this comment.
Na, no problem. Fixed it
| Cloud Hypervisor supports additional parameters to control the | ||
| migration process. Via the API or `ch-remote`, you may specify: | ||
|
|
||
| - `downtime-ms <milliseconds>`: \ |
There was a problem hiding this comment.
There are hyphens here but underscores above?
|
|
||
| ```console | ||
| $ target/release/ch-remote --api-socket=/tmp/api2 receive-migration unix:/tmp/sock | ||
| $ target/release/ch-remote --api-socket=/tmp/api2 receive-migration destination_url=unix:/tmp/sock |
There was a problem hiding this comment.
Does receive-migration actually take a URL like this?
There was a problem hiding this comment.
very good catch, thanks
2fc1595 to
0d3b5d1
Compare
This change prepares upcoming options (following commit) that are added to VmSendMigrationData. VmSendMigrationData is a special case as it is currently the only "rich configuration" type that lives outside `config.rs`, as it is purely API-facing. Therefore, it isn't integrated into the existing OptionParser infrastructure. We therefore introduce a `parse()` method to use that in `ch-remote` in the following. In `ch-remote`, we remove `--local` for `send-migration` and switch to the new option string parsing constructor (breaking change!). This prepares the addition of downtime and timeout options in the following and streamlines the `ch-remote` command line interface with other commands, such as `ch-remote add-net`. Lastly, this commit updates the integration tests. Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
Management software needs fine-grained control over live migration to meet QoS requirements for VM guests. Add `downtime_ms`, `timeout_s`, and `timeout_strategy` fields to `VmSendMigrationData`, exposed via API. This commit contains the API changes only; the VMM does not yet act on these values. This follows in the next commit. For the JSON API, downtime and timeout are represented as plain integers (downtime_ms and timeout_s) to make the units explicit. Using Duration directly would require custom (de)serialization logic, so instead the internal raw integers are exposed as Duration via getters. This introduces minor conversion overhead but keeps the Rust API clear and unambiguous. Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
Wire the new `downtime_ms`, `timeout_s`, and `timeout_strategy` fields from `VmSendMigrationData` into the precopy loop, replacing the previous hard-coded 5-iteration cap. Each iteration now evaluates three convergence criteria in order: - no dirty pages remain; - the estimated final-iteration downtime is within the configured budget - or the overall migration timeout has elapsed. On timeout, `TimeoutStrategy::Cancel` aborts and keeps the VM live on the source, while `TimeoutStrategy::Force` proceeds regardless of the downtime target. The convergence callback is updated to return a Result to propagate the cancel error cleanly up the call stack. With the recent changes [0], it is fairly easy to implement the new checks and operate on actual metrics. These changes are inspired by [1] but differ significantly in details. [0] cloud-hypervisor#7799 [1] cloud-hypervisor#7033 Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
This adds two new integration tests for the new functionality: - VM under load, downtime=1ms, timeout=1s, timeout_strategy=cancel - VM under load, downtime=1ms, timeout=1s, timeout_strategy=force By using a short downtime and timeout plus adding a stress worker in the guest, we can prevent quick migration. Therefore, we can nicely test the timeout_strategy. Testing for a specific downtime is cumbersome to do and highly depends on CPU/host utilization. To prevent flakiness, there is no such test integration test. I did, however, manual testing of that functionality. Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
Mosts tests used 4GB of RAM, although the VM is mostly idling. In CI, we experienced OOM issues on the ARM runners. If we reduce the VM memory of the parallel live migration tests to 1.5GB RAM, we still have enough capacity in the VM so that everything succeeds while reducing resource usage. Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
0d3b5d1 to
e2951da
Compare
Follow-up of #7799.
This patch adds user-controllable convergence parameters to Cloud Hypervisor's
live migration, replacing the previous hard-coded 5-iteration precopy cap with a
principled, metrics-driven approach.
What's new
Three new fields are added to
VmSendMigrationData(API +ch-remote):downtime_ms: maximum acceptable VM downtime (default: 300 ms, matching QEMU)timeout_s: overall migration time limit (default: 3600 s)timeout_strategy: action on timeout:cancel(abort, keep VM live on source)or
force(proceed despite unmet downtime budget)The precopy loop now evaluates convergence in order:
On timeout, the chosen strategy is applied cleanly, with
cancelpropagating aMigratableErrorup the call stack.Breaking change
The
ch-remote send-migrationCLI drops the--localflag in favour of aunified option string, consistent with other
ch-remotesubcommands:Testing
Two new integration tests cover the timeout path under memory pressure
(
stress --vm), verifying thatcancelleaves the source VM responsive andforceterminates it after a successful forced migration. A specific downtimetest is omitted due to host-load sensitivity; manual testing confirmed
correctness.
Docs and OpenAPI spec updated accordingly.
These changes are inspired by [0] but differ significantly in details.
[0] #7033