Codestin Search App

phip1611 · 2026-04-08T15:04:28Z

This PR touches a few areas described in #7111:

it further prepares live migration statistics (queryable via a dedicated API endpoint)
- here, we just gather more data, that we eventually might export
log the actual downtime of the VM (on the source side)
log the expensive tasks during the downtime (break down downtime in its components)
- this helps in the analysis of the downtime in further refactorings and developments

Please review this commit-by-commit.

New Log Messages

I performed a local TCP migration of a small VM with a workload.

New log messages on sender:

cloud-hypervisor:   7.703402s: <vmm> INFO:vmm/src/lib.rs:1494 -- Migration completed after 2.2s with a downtime of 298ms (goal was 300ms)
cloud-hypervisor:   7.703453s: <vmm> DEBUG:vmm/src/lib.rs:1500 -- Downtime breakdown: 298ms (final_iter:269ms state:7ms send_state:19ms complete:1ms)

New log messages on destination:

cloud-hypervisor:   7.283424s: <vmm> DEBUG:vmm/src/lib.rs:948 -- Migration (incoming): recv_snapshot:3ms restore:10ms
cloud-hypervisor:   7.284824s: <vmm> DEBUG:vmm/src/lib.rs:967 -- Migration (incoming): resume:1ms
cloud-hypervisor:   7.284842s: <vmm> DEBUG:vmm/src/lib.rs:977 -- Migration (incoming): Receiving final state and resuming the VM took 15ms

Learnings

For very small downtimes, we might have to optimize the snapshot path. Future work.

phip1611 · 2026-04-08T15:14:21Z

@@ -238,169 +238,174 @@ impl Display for MemoryMigrationContext {

 #[cfg(test)]
 mod unit_tests {
-    use std::time::{Duration, Instant};


diff doesn't look that nice 🤔 I just moved everything into sub module memory_migration_ctx_tests

It will get much cleaner if you enable the "Hide whitespace" option if you are using the GitHub Web UI.

likebreath

Looking good. Mostly cosmetic comments.

Great to see more observability for live migration.

likebreath · 2026-04-09T23:20:32Z

@@ -238,169 +238,174 @@ impl Display for MemoryMigrationContext {

 #[cfg(test)]
 mod unit_tests {
-    use std::time::{Duration, Instant};


It will get much cleaner if you enable the "Hide whitespace" option if you are using the GitHub Web UI.

likebreath · 2026-04-10T00:21:31Z

+    pub snapshotting_duration: Duration,
+    /// The time needed to send the snapshot including deserializing it on the
+    /// destination
+    pub sending_snapshot_duration: Duration,
+    /// The time of the completion request. This includes resuming the VM (if it
+    /// was running before the migration).
+    pub completing_duration: Duration,


Nit: I understand the caller side uses snapshot, I think it is better to stay consistent with the protocol naming, e.g. consider state_duration, send_state_duration, and complete_duration.

yup, thanks!

removed the inconsistent state/snapshot naming in favor of snapshot

I'd think state would be better, as that's what the migration protocol uses:

cloud-hypervisor/vm-migration/src/protocol.rs

Lines 111 to 121 in 474106a

pub enum Command {

#[default]

Invalid,

Start,

Config,

State,

Memory,

Complete,

Abandon,

MemoryFd,

}

Got it! Will update next week.

I removed the commit, updated a few identifiers (snapshot -> state) and added this section to the commit message

# Terminology At first glance, the use of "state" and "[VM] snapshot" may seem confusing. As discussed in [0], we use "state" consistently in the migration code. On the VM side, "snapshotting" is merely the mechanism used to obtain the VM state. [0] https://github.com/cloud-hypervisor/cloud-hypervisor/pull/7979#discussion_r3061359899

phip1611 · 2026-04-10T08:39:46Z

Since your review:

I refatored the state machine to return a Result and let the caller panic
slightly enhanced the reporting/metrics on the destination
removed the inconsistent state/snapshot naming in favor of snapshot

phip1611 · 2026-04-10T08:53:33Z

+            ctx.downtime_ctx.effective_downtime.as_millis(),
+            send_data_migration.downtime().as_millis()
+        );
+        debug!("Downtime breakdown: {}", ctx.downtime_ctx);


The new timings are now very very helpful (see PR description) to keep an eye on the downtime as Cloud Hypervisor progresses! After playing around a little, I think I've found the sweet spot for granularity here. Further breaking down the costs should be done in targeted analysis and development; not needed to have that in upstream code

phip1611 · 2026-04-10T08:54:15Z

+                    let (recv_snapshot_dur, restore_vm_dur) =
+                        self.vm_receive_snapshot(req, socket, config_data.memory_manager)?;
+                    debug!(
+                        "Migration (incoming): recv_snapshot:{}ms restore:{}ms",


The new timings are now very very helpful (see PR description) to keep an eye on the downtime as Cloud Hypervisor progresses! After playing around a little, I think I've found the sweet spot for granularity here. Further breaking down the costs should be done in targeted analysis and development; not needed to have that in upstream code

This helps to better separate the unit tests from the new ones in the following commit. On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

Expose the finalized per-iteration timing fields needed by higher-level migration metrics and factor the iteration-overhead calculation into a small helper. This keeps the existing MemoryMigrationContext behavior intact while making the timing data easier to consume from migration-level context in the following commits. On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

Add migration-level context types that extend the existing memory-only metrics with overall migration duration and downtime breakdown. OngoingMigrationContext models the sender-side migration progress until all inputs needed for final downtime accounting are available. CompletedMigrationContext then stores the finalized migration metrics, including the final memory iteration, snapshotting, snapshot transfer, and completion phase. This provides the data needed to log effective downtime in the VMM and lays the groundwork for future migration statistics reporting. # Terminology At first glance, the use of "state" and "[VM] snapshot" may seem confusing. As discussed in [0], we use "state" consistently in the migration code. On the VM side, "snapshotting" is merely the mechanism used to obtain the VM state. [0] cloud-hypervisor#7979 (comment) On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

This is helpful in the following to properly aggregate statistics for local migrations. On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

Add a small helper that returns both the successful result of an operation and the time it took to complete. Subsequent migration instrumentation uses this to keep timing code compact and consistent. On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

Use OngoingMigrationContext to measure and log the effective VM downtime (pause to remote resume) and the cost of each non-trivial step in the downtime window: snapshotting, sending the snapshot, and awaiting completion. This makes it straightforward to identify and reduce downtime as live migration matures. Example: ``` cloud-hypervisor: 7.703402s: <vmm> INFO:vmm/src/lib.rs:1494 -- Migration completed after 2.2s with a downtime of 298ms (goal was 300ms) cloud-hypervisor: 7.703453s: <vmm> DEBUG:vmm/src/lib.rs:1500 -- Downtime breakdown: 298ms (final_iter:269ms state:7ms send_state:19ms complete:1ms) ``` Note: downtime is measured on the source only; cross-host clock skew may cause unreliable results. # Terminology At first glance, the use of "state" and "[VM] snapshot" may seem confusing. As discussed in [0], we use "state" consistently in the migration code. On the VM side, "snapshotting" is merely the mechanism used to obtain the VM state. [0] cloud-hypervisor#7979 (comment) On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

Instrument the two main downtime-phase operations on the destination side - receiving state and resuming the VM - so their costs are visible in logs and can be iterated on. The new log messages may look like this: ```text cloud-hypervisor: 7.283424s: <vmm> DEBUG:vmm/src/lib.rs:948 -- Migration (incoming): recv_snapshot:3ms restore:10ms cloud-hypervisor: 7.284824s: <vmm> DEBUG:vmm/src/lib.rs:967 -- Migration (incoming): resume:1ms cloud-hypervisor: 7.284842s: <vmm> DEBUG:vmm/src/lib.rs:977 -- Migration (incoming): Receiving final state and resuming the VM took 15ms ``` On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

rbradford

Nice! Easy to review PR. Thanks.

Add migration-level context types that extend the existing memory-only metrics with overall migration duration and downtime breakdown. OngoingMigrationContext models the sender-side migration progress until all inputs needed for final downtime accounting are available. CompletedMigrationContext then stores the finalized migration metrics, including the final memory iteration, snapshotting, snapshot transfer, and completion phase. This provides the data needed to log effective downtime in the VMM and lays the groundwork for future migration statistics reporting. # Terminology At first glance, the use of "state" and "[VM] snapshot" may seem confusing. As discussed in [0], we use "state" consistently in the migration code. On the VM side, "snapshotting" is merely the mechanism used to obtain the VM state. [0] #7979 (comment) On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

Use OngoingMigrationContext to measure and log the effective VM downtime (pause to remote resume) and the cost of each non-trivial step in the downtime window: snapshotting, sending the snapshot, and awaiting completion. This makes it straightforward to identify and reduce downtime as live migration matures. Example: ``` cloud-hypervisor: 7.703402s: <vmm> INFO:vmm/src/lib.rs:1494 -- Migration completed after 2.2s with a downtime of 298ms (goal was 300ms) cloud-hypervisor: 7.703453s: <vmm> DEBUG:vmm/src/lib.rs:1500 -- Downtime breakdown: 298ms (final_iter:269ms state:7ms send_state:19ms complete:1ms) ``` Note: downtime is measured on the source only; cross-host clock skew may cause unreliable results. # Terminology At first glance, the use of "state" and "[VM] snapshot" may seem confusing. As discussed in [0], we use "state" consistently in the migration code. On the VM side, "snapshotting" is merely the mechanism used to obtain the VM state. [0] #7979 (comment) On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

phip1611 requested a review from a team as a code owner April 8, 2026 15:04

phip1611 self-assigned this Apr 8, 2026

phip1611 force-pushed the upstream-migration-effective-downtime branch from b78bd51 to f8ebc2c Compare April 8, 2026 15:11

phip1611 requested a review from likebreath April 8, 2026 15:13

phip1611 commented Apr 8, 2026

View reviewed changes

phip1611 force-pushed the upstream-migration-effective-downtime branch 2 times, most recently from 2f0ff71 to 6e4e25b Compare April 8, 2026 15:18

phip1611 commented Apr 8, 2026

View reviewed changes

Comment thread vmm/src/lib.rs Outdated

phip1611 commented Apr 8, 2026

View reviewed changes

Comment thread vm-migration/src/context.rs

phip1611 commented Apr 8, 2026

View reviewed changes

Comment thread vm-migration/src/context.rs Outdated

phip1611 force-pushed the upstream-migration-effective-downtime branch 2 times, most recently from c015419 to cbea647 Compare April 9, 2026 08:47

likebreath reviewed Apr 10, 2026

View reviewed changes

phip1611 force-pushed the upstream-migration-effective-downtime branch 3 times, most recently from 8c220df to bd985ff Compare April 10, 2026 08:38

phip1611 force-pushed the upstream-migration-effective-downtime branch from bd985ff to a078a5c Compare April 10, 2026 08:52

phip1611 commented Apr 10, 2026

View reviewed changes

Coffeeri reviewed Apr 10, 2026

View reviewed changes

Comment thread vmm/src/lib.rs Outdated

phip1611 force-pushed the upstream-migration-effective-downtime branch from a078a5c to 1249b0c Compare April 10, 2026 11:24

phip1611 added 2 commits April 13, 2026 08:37

vm-migration: context: move unit tests into sub module

d4f5b56

This helps to better separate the unit tests from the new ones in the following commit. On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

phip1611 force-pushed the upstream-migration-effective-downtime branch from 1249b0c to cd427ef Compare April 13, 2026 06:37

phip1611 added 4 commits April 13, 2026 08:39

vm-migration: add MemoryMigrationContext::empty_finalized() helper

c19e44b

This is helpful in the following to properly aggregate statistics for local migrations. On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

phip1611 force-pushed the upstream-migration-effective-downtime branch from cd427ef to 452ad3f Compare April 13, 2026 06:39

rbradford approved these changes Apr 13, 2026

View reviewed changes

rbradford added this pull request to the merge queue Apr 13, 2026

Merged via the queue into cloud-hypervisor:main with commit 2515b06 Apr 13, 2026
37 of 38 checks passed

phip1611 deleted the upstream-migration-effective-downtime branch April 13, 2026 12:01

rbradford added this to Cloud Hypervisor Roadmap Apr 25, 2026

rbradford moved this to ✅ Done in Cloud Hypervisor Roadmap Apr 25, 2026

	pub enum Command {
	#[default]
	Invalid,
	Start,
	Config,
	State,
	Memory,
	Complete,
	Abandon,
	MemoryFd,
	}

Uh oh!

Conversation

phip1611 commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New Log Messages

Learnings

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

likebreath left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phip1611 Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phip1611 commented Apr 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

phip1611 commented Apr 8, 2026 •

edited

Loading

phip1611 Apr 13, 2026 •

edited

Loading