Codestin Search App

phip1611 · 2026-04-14T16:49:05Z

This is part of our upstreaming plan (#7111 (comment)), specifically the "core mechanism" track.

The main changes are:

API call send-migration now only has dispatch semantics
- receive-migration remains blocking
- on the sender-side, to observe one can
  - check the event monitor for vm.migration-{finished,failed}
  - soon (not this PR): via a dedicated vm.migration-status endpoint
- the call returns now when the connection was either established or failed early (this is expected to happen in the first MS) but won't block the API event loop any longer
The migration happens in a dedicated thread
further API calls during an ongoing migration are now possible (the API loop is no longer blocked (hence the term "asynchronization"))
- Only non-VM changing calls are allowed: AddNet, AddBlk etc are disallowed
clear ownership semantics prevent externally-triggered lifecycle events to the VM as it is migrating
important groundwork for future live migration enhancements
~~seccomp for the migration path (which was missing!)~~
- Will follow in a nother PR

Closes #7039

likebreath

@phip1611 Thanks for the patience. On top of the comments below, two high-level questions:

What's the broader plan for an async HTTP API? This PR makes the handler async on the VMM side, but achieving full async semantics, say a cancel endpoint, or a progress query that can run alongside an in-flight migration, needs the HTTP layer to be async too, since it's currently a single-threaded serial loop.
With send-migration becoming dispatch-only, how is the management layer meant to learn that a migration finished or failed?

phip1611 · 2026-05-20T19:17:44Z

What's the broader plan for an async HTTP API? This PR makes the handler async on the VMM side, but achieving full async semantics, say a cancel endpoint, or a progress query that can run alongside an in-flight migration, needs the HTTP layer to be async too, since it's currently a single-threaded serial loop.

Fair question!

This PR makes the handler async on the VMM side,

Exactly! Thus the name "asynchronization" in the PR. However, with the temporary workaround vmm: api: temporarily make VmSendMigration call blocking again fe7fc5e (this PR), the HTTP API will block and will not allow other calls. You are perfectly right. From a HTTP point of view, the behavior is unchanged in this PR (and I need to fix the dbus-based API).

But I see this as temporary workaround and I would like to entirely get rid of this behavior shortly after this gets merged. send-migration should just have dispatch semantics (in a follow-up PR). With that, I come to your next question.

With send-migration becoming dispatch-only, how is the management layer meant to learn that a migration finished or failed?

By querying the upcoming vm.migration-status endpoint. The idea is that management software polls that (e.g. every 50ms). That is what we've implemented in our fork and use at scale (couple of thousands VMs with migrations). This design has proven to work well and reliable and we do not need any long-living blocking HTTP connections anymore.

With all that knowledge. Would you be fine with giving send-migration dispatch semantics in a follow-up?

By the way: For a couple of weeks we actually tried keeping a blocking send-migration call and made the HTTP server multithreaded - but this design turned out to be much more complicated in management software and in CH.

Does that help and clarify all your questions?

A few pointers into our fork (please apologize the sometimes rather pragmatic implementation, haha):

Vmm::vm_send_migration() with dispatch semantics: https://github.com/cyberus-technology/cloud-hypervisor/blob/9902cf3d8b31a272800f88cce06039d23ca247f1/vmm/src/lib.rs#L3361
Vmm::vm_cancel_migration(): can be called asynchronously any time https://github.com/cyberus-technology/cloud-hypervisor/blob/9902cf3d8b31a272800f88cce06039d23ca247f1/vmm/src/lib.rs#L3497
Vmm::vm_migration_progress(): can be called asynchronously any time https://github.com/cyberus-technology/cloud-hypervisor/blob/9902cf3d8b31a272800f88cce06039d23ca247f1/vmm/src/lib.rs#L3505

rbradford

Do you need to update the live migration documentation?

rbradford · 2026-06-19T09:54:20Z

+    }
+
+    /// Returns an error if the VM is currently owned by a migration worker.
+    fn reject_if_migrating(&self) -> result::Result<(), VmError> {


Would something like ok_or_migrating() be more Rust idiomatic? The "reject" verb feels off - not sure we use that anywhere else. This is frustrating because this feels off but i'm struggling to make a better suggestion. I like the code deduplication that this provides.

I like the new name. Please note that this helper just exists for a single commit to keep the diff smaller and changes less invasive. THe following commit streamlines the use of match self.vm {}

rbradford · 2026-06-19T10:04:36Z

                    EpollDispatch::GuestExit => {
                        info!("VM guest exit event");
                        self.guest_exit_evt.read().map_err(Error::EventFdRead)?;
+                        // TODO: Future follow-up must resolve lifecycle handling while migrating.


I think this feels pretty fundamental - we could easily get exit events or reset events during a migration. How does it work before this change?

I agree with you - the necessary changes are not trivial and I want to keep them out of this already complex PR. In our fork, we catch postponed lifecycle events (reboot, shutdown) in a field and replay them on the destination after successfull migration - but that is a non-trivial (but also not overly complex) change.

Currently (without this), when the guest reboots or shutdowns during a migration, the VMM is in a weird state: the API loop is blocked because of the ongoing long-living send_migration call, eventually the vCPUs are in a weird state (producing panics) and producing scary logs and then the guest reboots on the host and the migration fails.

So IMHO the new code makes existing technical debt visible - doesn't necessarily makes things worse.

That was what was unclear to me - whether these TODOs are just surfacing the existing undefined behaviour or making it worse.

rbradford

This code introduces lots of assertions and panics - I hope you're sure they're ok?

rbradford · 2026-06-19T10:59:16Z

+    fn reject_if_migrating(&self) -> result::Result<(), VmError> {
+        match self {
+            VmOwnership::Migration { .. } => Err(VmError::VmMigrating),
+            _ => Ok(()),
        }
    }


Weird, i'm sure I left a comment about this - I know i'm brown M&Ming here - but this feels slightly off. I like the code deduplication but i'm not sure about the naming. Can we look for a more Rust idiomatic name?

I agree with you, also not 100% happy. I'll brainstorm a bit

phip1611 · 2026-06-19T11:15:08Z

This code introduces lots of assertions and panics - I hope you're sure they're ok?

Let me double-check. The whole PR is crafted with a very special focus on not panicking, to keep the VM running at all costs. I usually just do panics for situations of programming errors, which typically end in non-recoverable issues anyway

This initializes the module and the thread that will handle (control) the migration. This introduces the new types without the necessary wiring. On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

This is a pre-requisite for the following commit which puts the migration into a dedicated thread. It allows the VMM to react to migration events (success/failure). The commit series was inspired by @ljcore [0] but was changed quite significantly. [0] cloud-hypervisor#7038 On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

phip1611 · 2026-06-19T13:24:40Z

This code introduces lots of assertions and panics - I hope you're sure they're ok?

Let me double-check. The whole PR is crafted with a very special focus on not panicking, to keep the VM running at all costs. I usually just do panics for situations of programming errors, which typically end in non-recoverable issues anyway

TBH, I've iterated on this so much that right now I can't see the forest for the trees. Time for weekend :D I'm confident it's solid, since I've also done multiple manual and LLM review rounds. I'm planning follow-ups anyway, where I can put more focus on this.

This will serve as the base for many great future follow-ups, although this is the biggest change and enables many more features.

PS: From a very coarse-grained check, I think most unwraps were there already and are just a result of the changed indentation

This puts the send-migration action into a dedicated thread, laying the groundwork for many follow-ups towards first-class live-migration in CH. This means: 1. The send-migration call will exit sooner (just trigger the migration - dispatch semantics) 2. Other API calls can be triggered while a migration is ongoing but will not be able to alter the VM as the VM's ownership is transferred from the VMM to the migration thread. Example: hotplugging won't work (which is good). 3. This is the basis for migration statistics via a dedicated endpoint (future work). The whole change was done with a special focus on graceful recover and cleanup: even if anything on the migration paths go wrong, the proper cleanups are already executed and the VMM can take back the ownership of the VM. The receive-migration API call remains blocking. To observe any status changes about the migration on the sender side, one can observe the event-monitor output and look for `vm.migration-{failed,finished}`. These changes are inspired by [0] but differ significantly in details. [0] cloud-hypervisor#7038 On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

This streamlines the behavior with the other request handlers so that now almost every request handler uses a match on self.vm. On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

rbradford

Thank you for your patience - this has been open a long time!

rbradford · 2026-06-19T15:07:38Z

I'll leave for @likebreath to do a final review and merge.

This comment was marked as outdated.

Sign in to view

phip1611 force-pushed the upstream-asynchronization branch 2 times, most recently from eddec32 to 201577b Compare May 12, 2026 10:57

This comment was marked as outdated.

Sign in to view

phip1611 force-pushed the upstream-asynchronization branch 5 times, most recently from ebf5a69 to a949d28 Compare May 12, 2026 15:43

phip1611 self-assigned this May 12, 2026

phip1611 marked this pull request as ready for review May 12, 2026 16:01

phip1611 requested a review from a team as a code owner May 12, 2026 16:01

phip1611 mentioned this pull request May 13, 2026

Live Migration Feature Tracking and Collaboration #7111

Open

phip1611 force-pushed the upstream-asynchronization branch from a949d28 to b279f71 Compare May 13, 2026 10:54

phip1611 requested review from arctic-alpaca and likebreath May 13, 2026 14:42

This comment was marked as outdated.

Sign in to view

phip1611 commented May 13, 2026

View reviewed changes

Comment thread vmm/src/lib.rs Outdated

This comment was marked as outdated.

Sign in to view

arctic-alpaca reviewed May 15, 2026

View reviewed changes

phip1611 commented May 15, 2026

View reviewed changes

Comment thread vmm/src/migration/worker.rs Outdated

phip1611 force-pushed the upstream-asynchronization branch 2 times, most recently from ebf1862 to 07c63b8 Compare May 15, 2026 17:49

likebreath reviewed May 20, 2026

View reviewed changes

Comment thread vmm/src/lib.rs Outdated

Comment thread vmm/src/api/http/http_endpoint.rs Outdated

Comment thread vmm/src/lib.rs Outdated

Comment thread vmm/src/seccomp_filters.rs Outdated

Comment thread vmm/src/api/http/http_endpoint.rs Outdated

This comment was marked as outdated.

Sign in to view

phip1611 marked this pull request as ready for review June 17, 2026 10:50

This comment was marked as outdated.

Sign in to view

phip1611 requested a review from rbradford June 17, 2026 10:51

phip1611 force-pushed the upstream-asynchronization branch 2 times, most recently from f44d5c6 to 205e914 Compare June 19, 2026 09:46

rbradford reviewed Jun 19, 2026

View reviewed changes

phip1611 force-pushed the upstream-asynchronization branch from 205e914 to 5ac8592 Compare June 19, 2026 13:19

phip1611 added 2 commits June 19, 2026 15:19

vmm: init migration worker module

bf69bb7

This initializes the module and the thread that will handle (control) the migration. This introduces the new types without the necessary wiring. On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

phip1611 force-pushed the upstream-asynchronization branch from 5ac8592 to 9b12af5 Compare June 19, 2026 13:19

phip1611 requested a review from rbradford June 19, 2026 13:24

phip1611 added 4 commits June 19, 2026 15:26

vmm: streamline request handlers to use match{} on vm

5433840

This streamlines the behavior with the other request handlers so that now almost every request handler uses a match on self.vm. On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

vmm: move migration modules into folder

ae73e68

On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

tests: adjust to new dispatch semantics of ch-remote send-migration

ec2dc9b

On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

phip1611 force-pushed the upstream-asynchronization branch from 9b12af5 to ec2dc9b Compare June 19, 2026 13:27

rbradford approved these changes Jun 19, 2026

View reviewed changes

likebreath approved these changes Jun 22, 2026

View reviewed changes

likebreath added this pull request to the merge queue Jun 22, 2026

Merged via the queue into cloud-hypervisor:main with commit 6a16b65 Jun 22, 2026
38 checks passed

phip1611 deleted the upstream-asynchronization branch June 22, 2026 20:53

This was referenced Jun 23, 2026

vmm: migration: add seccomp rules for migration threads #8453

Merged

vmm: keep virtio activation alive in migration #8462

Merged

rbradford added this to Cloud Hypervisor Roadmap Jul 8, 2026

likebreath mentioned this pull request Jul 10, 2026

build: Release v53.0 #8576

Merged

Uh oh!

Conversation

phip1611 commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

likebreath left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phip1611 commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rbradford Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

phip1611 Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

rbradford Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

phip1611 Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

rbradford Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

rbradford Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

phip1611 Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

phip1611 commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

phip1611 commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

rbradford commented Jun 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

phip1611 commented Apr 14, 2026 •

edited

Loading

phip1611 commented May 20, 2026 •

edited

Loading

phip1611 commented Jun 19, 2026 •

edited

Loading

phip1611 commented Jun 19, 2026 •

edited

Loading