Thanks to visit codestin.com
Credit goes to github.com

Skip to content

vmm: migration: add user-configurable downtime and timeout - #7835

Merged
rbradford merged 7 commits into
cloud-hypervisor:mainfrom
phip1611:upstream-downtime-configuration
Mar 24, 2026
Merged

rbradford merged 7 commits into
cloud-hypervisor:mainfrom
phip1611:upstream-downtime-configuration

Conversation

@phip1611

@phip1611 phip1611 commented Mar 13, 2026

Copy link
Copy Markdown
Member

Follow-up of #7799.

This patch adds user-controllable convergence parameters to Cloud Hypervisor's
live migration, replacing the previous hard-coded 5-iteration precopy cap with a
principled, metrics-driven approach.

What's new

Three new fields are added to VmSendMigrationData (API + ch-remote):

  • downtime_ms: maximum acceptable VM downtime (default: 300 ms, matching QEMU)
  • timeout_s: overall migration time limit (default: 3600 s)
  • timeout_strategy: action on timeout: cancel (abort, keep VM live on source)
    or force (proceed despite unmet downtime budget)

The precopy loop now evaluates convergence in order:

  • no dirty pages remain
  • estimated downtime is within budget
  • timeout elapsed:

On timeout, the chosen strategy is applied cleanly, with cancel propagating a MigratableError up the call stack.

Breaking change

The ch-remote send-migration CLI drops the --local flag in favour of a
unified option string, consistent with other ch-remote subcommands:

ch-remote --api-socket=/tmp/api send-migration \
  destination_url=tcp:host:port,downtime_ms=200,timeout_s=3600,timeout_strategy=cancel

Testing

Two new integration tests cover the timeout path under memory pressure
(stress --vm), verifying that cancel leaves the source VM responsive and
force terminates it after a successful forced migration. A specific downtime
test is omitted due to host-load sensitivity; manual testing confirmed
correctness.

Docs and OpenAPI spec updated accordingly.


These changes are inspired by [0] but differ significantly in details.

[0] #7033

@phip1611 phip1611 self-assigned this Mar 13, 2026
@phip1611
phip1611 force-pushed the upstream-downtime-configuration branch from b0c61b3 to ee8ecd4 Compare March 13, 2026 09:07
@phip1611 phip1611 changed the title vmm: migration: add user-configurable downtime and timeout [Waiting for #7799] vmm: migration: add user-configurable downtime and timeout Mar 13, 2026
@phip1611
phip1611 force-pushed the upstream-downtime-configuration branch 3 times, most recently from 96b700b to 32c757e Compare March 13, 2026 10:02
@phip1611
phip1611 force-pushed the upstream-downtime-configuration branch from 32c757e to bc5f383 Compare March 13, 2026 21:12
@phip1611 phip1611 changed the title [Waiting for #7799] vmm: migration: add user-configurable downtime and timeout vmm: migration: add user-configurable downtime and timeout Mar 13, 2026
@phip1611
phip1611 marked this pull request as ready for review March 13, 2026 21:13
@phip1611
phip1611 requested a review from a team as a code owner March 13, 2026 21:13
@phip1611
phip1611 force-pushed the upstream-downtime-configuration branch 3 times, most recently from 2c4dd21 to fcbca39 Compare March 13, 2026 22:21
Comment thread cloud-hypervisor/src/bin/ch-remote.rs Outdated
Comment thread vmm/src/lib.rs Outdated
@phip1611
phip1611 force-pushed the upstream-downtime-configuration branch 3 times, most recently from 1663324 to 3487c99 Compare March 14, 2026 09:02
Comment thread cloud-hypervisor/src/bin/ch-remote.rs Outdated
Comment thread vmm/src/api/mod.rs Outdated
Comment thread vmm/src/api/mod.rs Outdated
@phip1611
phip1611 force-pushed the upstream-downtime-configuration branch 4 times, most recently from 9341d8f to 3f60a47 Compare March 16, 2026 19:24
Comment thread cloud-hypervisor/src/bin/ch-remote.rs Outdated

@likebreath likebreath left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beyond the comments below — what are your thoughts on adding integration tests to cover the new parameters? My initial thought is we could extend the existing live migration tests to override the default values for both positive cases (migration completes successfully within the configured downtime/timeout) and negative cases (migration fails or cancels when the timeout is reached).

This would give us confidence that the convergence logic and timeout strategy behave as expected.

Comment thread vmm/src/lib.rs Outdated
Comment thread cloud-hypervisor/src/bin/ch-remote.rs Outdated
.subcommand_matches("send-migration")
.unwrap()
.get_one::<TimeoutStrategy>("timeout-strategy")
.unwrap(),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand it has been a legacy issue since --local was introduced, but with three new parameters it's reaching a point where it should be properly addressed.

I'd suggest: implement VmSendMigrationData::parse(cmd: &str) -> Result<Self> so that ch-remote passes a single comma-separated string rather than individual --flags.
The CLI would become:

$ ch-remote --api-socket=/tmp/api send-migration tcp:{dst}:{port},local=false,downtime_ms=300,timeout_s=60,timeout_strategy=cancel

instead of:

$ ch-remote --api-socket=/tmp/api send-migration tcp:{dst}:{port} --downtime-ms 300 --timeout-s 60 --timeout-strategy cancel

This keeps send-migration consistent with how the rest of ch-remote and cloud-hypervisor handle commandline options - using the option_parser comma-separated style. Having send-migration be the only subcommand using individual --flags creates an inconsistency for both users and contributors.

Beyond consistency, this would also:

  • Eliminate the duplicated code from ch-remote;
  • Use the exact data structure defined from vmm crate say (avoid defining a separate TimeoutStrategy enum in ch-remote);
  • Make future parameter additions a one-line change instead of touching multiple call sites.

I understand this is a breaking change for --local, but I think now is the right time — before more parameters accumulate and the breaking change becomes harder to make.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I agree that duplicate types are not ideal, I would argue that those types are only duplicates in name, not in function. User APIs and internal representation have different requirements and mixing both in a single type leads usability or type safety conflicts like the Duration one discussed above.

While the VmSendMigrationData::parse(cmd: &str) -> Result<Self> approach does separate these concerns, it skips some of the nice things clap already provides like the error messages.
Compare, for example, these two error messages caused by a trailing comma:

> cargo run --bin ch-remote -- --api-socket=/tmp/chv1.sock add-device path=foo,

[2026-03-17T08:06:12Z ERROR cloud_hypervisor] Fatal error: AddDeviceConfig(ParseDevice(UnknownOption("")))
Error: ch-remote exited with the following chain of errors:
  0: Error parsing device syntax
  1: Error parsing --device
  2: unknown option: 
> cargo run --bin ch-remote -- --api-socket=/tmp/chv1.sock send-migration tcp:127.0.0.1:1337 --downtime-ms 200,

error: invalid value '200,' for '--downtime-ms <downtime-ms>': invalid digit found in string

For more information, try '--help'.

From what I can see, VmSendMigrationData::parse(cmd: &str) -> Result<Self> also requires significant manual implementation.

I don't know the reasons for why this API style was chosen though, so it's very possible I'm missing important points.

@phip1611 phip1611 Mar 17, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm with @arctic-alpaca that clap's convenience is outstanding. That being said, I'm with you that we should stay consistent. We could refactor this in the future eventually.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. Let's fix the inconsistency issue here and follow-up on broader CLI changes in #7551.

Comment thread cloud-hypervisor/src/bin/ch-remote.rs Outdated
Comment thread docs/live_migration.md Outdated
Comment on lines +6 to +8
1. **local migration**: Migrating a VM from one Cloud Hypervisor instance to another on the same machine; also called
UNIX socket migration.
1. **TCP migration**: migrating a VM between two TCP/IP hosts.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the original terminology was more precise. Transport mechanism (TCP vs UNIX socket) is orthogonal to migration use cases ("remote" vs "local"), say:

  • TCP to 127.0.0.1 is a local migration over TCP
  • A UNIX socket forwarded by a management layer could serve a remote migration.

I'd suggest keeping the original naming.

@phip1611 phip1611 Mar 17, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks - changed that.

Comment thread docs/live_migration.md Outdated
@phip1611

Copy link
Copy Markdown
Member Author

I think we are good to go - integration test included!

@likebreath

Copy link
Copy Markdown
Member

I think we are good to go - integration test included!

Great. I will get to it later today (and the multi-TCP next week).

@phip1611
phip1611 force-pushed the upstream-downtime-configuration branch from d595bab to 82efedf Compare March 20, 2026 18:20

@likebreath likebreath left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@phip1611 Thank you again for the excellent work. We now have a proper, user-configurable mechanism for managing live migration convergence - a critical feature towards making Live Migration production-ready.

@likebreath
likebreath added this pull request to the merge queue Mar 20, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Mar 21, 2026
@phip1611

phip1611 commented Mar 21, 2026

Copy link
Copy Markdown
Member Author

Seems like we're running out of memory. The two new tests already just need 2gb per VM.. What's the proper solution to address this? Move them to the sequential tests?

Think I solved it in the latest commit

@phip1611
phip1611 force-pushed the upstream-downtime-configuration branch 2 times, most recently from 607b270 to d236565 Compare March 21, 2026 09:26
@phip1611
phip1611 force-pushed the upstream-downtime-configuration branch from d236565 to 2fc1595 Compare March 23, 2026 07:07
@phip1611

Copy link
Copy Markdown
Member Author

@rbradford Can we get this over the finish line today?

@rbradford rbradford left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please go through the documentation and double check it's correct. I'm not 100% sure it is.

format!("--api-socket={}", &src_api_socket),
"send-migration".to_string(),
format! {"unix:{migration_socket}"},
format!(
"destination_url=unix:{migration_socket},local={}",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think this should probably be integrated into the previous commit since it breaks test bisect. I think it's fine to add new test in a later commit but adapting them (especially when the change is so simple) probably should be in the commit that changes it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, adressed!

Comment thread vmm/src/api/mod.rs Outdated
/// Cancel the migration and keep the VM running on the source.
Cancel,
/// Force the migration and ignore any downtime requirement.
Force,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I feel like Ignore is the better terminology here? It's ignoring the timeout vs forcing some behaviour. But this would be an annoying change for you to make so I could live with it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Na, no problem. Fixed it

Comment thread docs/live_migration.md Outdated
Cloud Hypervisor supports additional parameters to control the
migration process. Via the API or `ch-remote`, you may specify:

- `downtime-ms <milliseconds>`: \

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are hyphens here but underscores above?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, fixed

Comment thread docs/live_migration.md Outdated

```console
$ target/release/ch-remote --api-socket=/tmp/api2 receive-migration unix:/tmp/sock
$ target/release/ch-remote --api-socket=/tmp/api2 receive-migration destination_url=unix:/tmp/sock

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does receive-migration actually take a URL like this?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very good catch, thanks

@phip1611
phip1611 force-pushed the upstream-downtime-configuration branch from 2fc1595 to 0d3b5d1 Compare March 24, 2026 12:54
@phip1611

Copy link
Copy Markdown
Member Author

This change prepares upcoming options (following commit) that are added
to VmSendMigrationData.

VmSendMigrationData is a special case as it is currently the only
"rich configuration" type that lives outside `config.rs`, as it is
purely API-facing. Therefore, it isn't integrated into the existing
OptionParser infrastructure. We therefore introduce a `parse()` method
to use that in `ch-remote` in the following.

In `ch-remote`, we remove `--local` for `send-migration` and switch to
the new option string parsing constructor (breaking change!). This
prepares the addition of downtime and timeout options in the following
and streamlines the `ch-remote` command line interface with other
commands, such as `ch-remote add-net`.

Lastly, this commit updates the integration tests.

Signed-off-by: Philipp Schuster <[email protected]>
On-behalf-of: SAP [email protected]
Management software needs fine-grained control over live migration to
meet QoS requirements for VM guests. Add `downtime_ms`, `timeout_s`, and
`timeout_strategy` fields to `VmSendMigrationData`, exposed via API.

This commit contains the API changes only; the VMM does not yet act on
these values. This follows in the next commit.

For the JSON API, downtime and timeout are represented as plain integers
(downtime_ms and timeout_s) to make the units explicit. Using Duration
directly would require custom (de)serialization logic, so instead the
internal raw integers are exposed as Duration via getters. This
introduces minor conversion overhead but keeps the Rust API clear and
unambiguous.

Signed-off-by: Philipp Schuster <[email protected]>
On-behalf-of: SAP [email protected]
Wire the new `downtime_ms`, `timeout_s`, and `timeout_strategy` fields
from `VmSendMigrationData` into the precopy loop, replacing the previous
hard-coded 5-iteration cap.

Each iteration now evaluates three convergence criteria in order:
- no dirty pages remain;
- the estimated final-iteration downtime is within the configured budget
- or the overall migration timeout has elapsed.

On timeout, `TimeoutStrategy::Cancel` aborts and keeps the VM live on
the source, while `TimeoutStrategy::Force` proceeds regardless of the
downtime target. The convergence callback is updated to return a Result
to propagate the cancel error cleanly up the call stack.

With the recent changes [0], it is fairly easy to implement the new
checks and operate on actual metrics.

These changes are inspired by [1] but differ significantly in details.

[0] cloud-hypervisor#7799
[1] cloud-hypervisor#7033

Signed-off-by: Philipp Schuster <[email protected]>
On-behalf-of: SAP [email protected]
Signed-off-by: Philipp Schuster <[email protected]>
On-behalf-of: SAP [email protected]
Signed-off-by: Philipp Schuster <[email protected]>
On-behalf-of: SAP [email protected]
This adds two new integration tests for the new functionality:

- VM under load, downtime=1ms, timeout=1s, timeout_strategy=cancel
- VM under load, downtime=1ms, timeout=1s, timeout_strategy=force

By using a short downtime and timeout plus adding a stress worker in the
guest, we can prevent quick migration. Therefore, we can nicely test the
timeout_strategy.

Testing for a specific downtime is cumbersome to do and highly depends
on CPU/host utilization. To prevent flakiness, there is no such test
integration test. I did, however, manual testing of that functionality.

Signed-off-by: Philipp Schuster <[email protected]>
On-behalf-of: SAP [email protected]
Mosts tests used 4GB of RAM, although the VM is mostly idling. In CI, we
experienced OOM issues on the ARM runners. If we reduce the VM memory of
the parallel live migration tests to 1.5GB RAM, we still have enough
capacity in the VM so that everything succeeds while reducing resource
usage.

Signed-off-by: Philipp Schuster <[email protected]>
On-behalf-of: SAP [email protected]
@phip1611
phip1611 force-pushed the upstream-downtime-configuration branch from 0d3b5d1 to e2951da Compare March 24, 2026 12:56

@rbradford rbradford left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do it 🚀

@rbradford
rbradford added this pull request to the merge queue Mar 24, 2026
Merged via the queue into cloud-hypervisor:main with commit 8149524 Mar 24, 2026
37 checks passed
@rbradford rbradford moved this to ✅ Done in Cloud Hypervisor Roadmap Apr 25, 2026
@phip1611
phip1611 deleted the upstream-downtime-configuration branch September 15, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

4 participants