Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Post-migration network announcements - #8263

Merged
rbradford merged 2 commits into
cloud-hypervisor:mainfrom
amphi:upstream-virtio-net-announcements
Jul 2, 2026
Merged

rbradford merged 2 commits into
cloud-hypervisor:mainfrom
amphi:upstream-virtio-net-announcements

Conversation

@amphi

@amphi amphi commented May 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Add post-migration network announcements for virtio-net devices so migrated guests
refresh their L2 state on the new host more quickly.

In our tests, connectivity after live migration can take up to 20 seconds
to recover before the VM becomes pingable again. With the post-migration
announcements, the network path is refreshed within a few milliseconds.

The new flow does two things after incoming migration:

  • sends a host-side RARP on supported virtio-net devices
  • triggers guest-visible re-announcement when VIRTIO_NET_F_GUEST_ANNOUNCE was negotiated

This is implemented for both virtio-net and vhost-user-net, with VMM hooks to
run the announcement sequence after the destination VM resumes.

Details

  • Add a post_migration_announce hook to virtio devices and wire it through the VMM
  • Implement post-migration announcement handling for virtio-net and vhost-user-net
  • Retry announcements for a short window after migration
  • Invalidate stale announcers when the device is reset, paused, or starts a new migration
  • Fix the virtio-net control queue parser to accept VIRTIO_NET_CTRL_ANNOUNCE_ACK
    without a data descriptor
  • Add unit tests for the ACK parser case and announcer invalidation
  • Update live migration documentation

Open Questions

  • The post-migration announcers only fire if the VM was not paused prior to the migration, and there is currently no way to trigger the announcers using the API. Is this fine, or do we either

@amphi
amphi requested a review from a team as a code owner May 21, 2026 08:19

@phip1611 phip1611 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! 🚀 Left a few comments.

PS: If I'm not mistaken, this introduces VMM-driven RARP-packages and guest-driven GARP packages, yes? I think the latter is not entirely clear from the description IMHO

Comment thread virtio-devices/src/net.rs Outdated
avail_features: u64,
acked_features: u64,
config: VirtioNetConfig,
queue_sizes: Vec<u16>,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

while on it: I think you could refactor this easily to Box<[u16]>. This makes clear that this data structure does not resize once constructed. I just tried it locally and it is pretty forward. This would nicely align with changes such as #8231

@amphi amphi May 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VirtioCommon uses Vec<u16> for queue_sizes too, I don't think we gain a lot by changing it here but not in VirtioCommon. I think this should be done in one PR.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the clarification

Comment thread virtio-devices/src/net.rs Outdated
Comment thread virtio-devices/src/vhost_user/net.rs Outdated
Comment thread virtio-devices/src/vhost_user/net.rs
Comment thread virtio-devices/src/net.rs
Comment thread net_util/src/ctrl_queue.rs
Comment thread virtio-devices/src/device.rs Outdated
Comment thread vmm/src/device_manager.rs Outdated
Comment thread vmm/src/device_manager.rs Outdated
Comment thread vmm/src/lib.rs Outdated
@likebreath
likebreath self-requested a review May 22, 2026 00:52

@rbradford rbradford left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this in CH?

Comment thread vmm/src/device_manager.rs Outdated
Comment on lines +5344 to +5359
let _ = thread::Builder::new()
.name("post-migration-announcers".to_string())
.spawn(move || {
for round in 0..rounds {
info!("Post migration announce (async): {}/{}", round + 1, rounds);

// The first announcement already was done synchronously, thus
// we sleep at the start of the loop.

let delay = (initial_delay + step_delay.saturating_mul(round)).min(max_delay);
debug!("Sleeping {}ms", delay.as_millis());
thread::sleep(delay);

announcers.iter_mut().for_each(|a| a.announce());
}
});

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can't just spin up threads and not track them - they need to exit on exit_evt - need join, etc. Is all this even necessary - I thought if you used libvirt it handles the TAP creation so why are you doing this in CH?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added life cycle management, please take another look.

Comment thread virtio-devices/src/net.rs Outdated
Comment on lines +1115 to +1120
match unsafe {
libc::write(
tap.as_raw_fd(),
buf.as_ptr() as *const libc::c_void,
buf.len(),
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tap implements Write so you don't need this unsafe libc

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I am using the taps Write now.

@amphi

amphi commented May 26, 2026

Copy link
Copy Markdown
Contributor Author

Why do we need this in CH?

The motivation is to reduce the post-migration connectivity gap. In our tests, after live migration the guest sometimes only became reachable again after several seconds, in the worst case around 20s. With post-migration announcements, the network path was refreshed within a few milliseconds. So the goal is to make network connectivity recover promptly after migration.

I think Cloud Hypervisor is the right layer to trigger this for two reasons:

The VMM knows the precise point at which the destination VM has resumed and the migrated virtio-net device is active again. Management software can observe that migration completed, but it does not necessarily know the exact device-level point where an announcement is both useful and safe.
The required mechanisms are device-internal. For the guest-side path, Cloud Hypervisor has to set VIRTIO_NET_S_ANNOUNCE, inject the virtio config interrupt, and handle VIRTIO_NET_CTRL_ANNOUNCE_ACK. Management software normally does not have direct access to those virtio-net internals.

There is also a practical split of responsibility here: management software can orchestrate migration, but the hypervisor owns the emulated device state and the post-resume device semantics. Implementing this in every management stack would duplicate hypervisor/device-specific knowledge and would still require a VMM-specific API for operations such as triggering the virtio config interrupt.

The host-side RARP and the guest-side VIRTIO_NET_S_ANNOUNCE path also complement each other. RARP can refresh switch MAC learning for the visible guest MAC, while the guest-side announcement lets the guest emit the protocol-specific gratuitous packets it knows how to produce, for example for VLANs or other guest network configuration that the VMM should not try to infer.

This also matches existing hypervisor behavior. QEMU handles virtio-net guest announcements from the VMM/device side after migration, rather than relying purely on management software.

So my view is that Cloud Hypervisor should provide the automatic post-migration announcement behavior as part of virtio-net migration handling. A management API to manually trigger announcements could still be useful as an additional control surface, but it should not be the only mechanism required for the normal live-migration path.

@rbradford

Copy link
Copy Markdown
Member

I think PR still has planned changes?

@amphi
amphi force-pushed the upstream-virtio-net-announcements branch 2 times, most recently from 1cc9ec7 to 10fcd29 Compare May 29, 2026 11:57
@amphi

amphi commented May 29, 2026

Copy link
Copy Markdown
Contributor Author

I think PR still has planned changes?

Yes, I just pushed some changes. It took me a while to get the life cycle management of the new thread right.

Comment thread virtio-devices/src/net.rs Outdated
Comment on lines +403 to +404
/// Tracks whether the guest still needs to acknowledge a post-migration
/// announce request through the control queue.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this comment is not needed - this member has a really nice clear name.

Comment thread virtio-devices/src/net.rs Outdated
avail_features: u64,
acked_features: u64,
config: VirtioNetConfig,
announce_pending: bool,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you need to add #[serde(default)]

Comment thread net_util/src/ctrl_queue.rs Outdated
Comment on lines +84 to +85
/// Tracks whether the guest still needs to acknowledge a post-migration
/// announce request through the control queue.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto - this member as a clear name.

Comment thread virtio-devices/src/net.rs Outdated
Comment on lines +418 to +419

/// Constructor-time copy of the fields needed to initialize the live device

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is "Constructor-time" hyphenated and what does that even mean? I hate this refactoring. It makes it hard to read the code and we don't use this pattern anywhere else in the project.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason for this refactoring was reviewability. In our downstream branch, adding the announce-related state directly to the existing constructor caused a lot of surrounding code movement, which made the functional change harder to review.

That said, I understand the concern about introducing a pattern that is not used elsewhere in the project. I do not feel strongly about keeping this refactoring in the upstream PR, so I will remove it and fold the required changes back into the existing structure.

Comment thread virtio-devices/src/net.rs Outdated
Comment on lines +451 to +453
offload_tso: bool,
offload_ufo: bool,
offload_csum: bool,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A good refactoring would be to create a NetOffloadControl struct holding these.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The refactoring here was only meant to reduce review noise for this PR by keeping the functional changes more localized.

I do not intend to include additional unrelated refactorings.

Comment thread virtio-devices/src/net.rs Outdated
Comment on lines +466 to +483
if offload_csum {
avail_features |= (1 << VIRTIO_NET_F_CSUM)
| (1 << VIRTIO_NET_F_GUEST_CSUM)
| (1 << VIRTIO_NET_F_CTRL_GUEST_OFFLOADS);

if offload_tso {
avail_features |= (1 << VIRTIO_NET_F_HOST_ECN)
| (1 << VIRTIO_NET_F_HOST_TSO4)
| (1 << VIRTIO_NET_F_HOST_TSO6)
| (1 << VIRTIO_NET_F_GUEST_ECN)
| (1 << VIRTIO_NET_F_GUEST_TSO4)
| (1 << VIRTIO_NET_F_GUEST_TSO6);
}

if offload_ufo {
avail_features |= (1 << VIRTIO_NET_F_HOST_UFO) | (1 << VIRTIO_NET_F_GUEST_UFO);
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could then turn that struct into features via a method on it.

Comment thread net_util/src/ctrl_queue.rs Outdated
pub struct CtrlQueue {
pub taps: Vec<Tap>,
/// Tracks whether the guest still needs to acknowledge a post-migration
/// announce request through the control queue.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

Comment thread vmm/src/post_migration_announcer.rs Outdated
exit_evt: EventFd,
}

impl PostMigrationAnnouncerTask {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is all way too complicated and overengineered. Can't you just handle it in the virtio-net thread in the destination VM. You know if you've been migrated or restored as it will come through the construction with a State struct.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not work.

At that point the device object is being created, but the destination VM has not necessarily resumed yet and the datapath is not necessarily at the point where an announcement can refresh the external network state. The announcement needs to happen after incoming migration, once the migrated device is active on the destination side.

We also observed that a single announcement is not always sufficient. In our downstream setup we repeat the announcements for a short window after migration, and we even added a REST API endpoint to trigger them manually a few seconds after migration completed. That was useful in practice because the timing depends not only on the device construction, but also on when the VM, guest networking, tap/bridge setup, and surrounding network have actually settled.

So I would prefer to keep the trigger tied to the post-migration/resume path rather than the constructor. I can change the implementation if you have any suggestions, but I do not think moving the announcement request into construction would preserve the behavior we need.

@phip1611 phip1611 Jun 1, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The announcement needs to happen after incoming migration, once the migrated device is active on the destination side.

A little more specific:

The announcement needs to happen after a successfully resumed VM (coming from an incoming migration). Otherwise the network's routing information could be in a weird state if the migration failed and keeps running on the source but the "network announcements" happened already on the destination.

@rbradford rbradford Jun 1, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The virtio-net epoll thread is only running once the VM is resumed. Put a timerfd into the epoll to handle the periodic notifications.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try that.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rbradford I moved the retry logic into the epoll thread.

@amphi
amphi force-pushed the upstream-virtio-net-announcements branch 2 times, most recently from b0eff3a to 94023cb Compare June 3, 2026 11:06
Comment thread vmm/src/lib.rs Outdated
);

// Advertise new VM location to network switches.
vm.trigger_post_migration_announcements();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain why you can't just do this on resume? This announcement functionality make sense on snapshot restore (which is a kind of offline migration)

You can avoid doing it on a resume without restore or migration by only enabling it if the Net was constructed with a State.

@phip1611 phip1611 Jun 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see where this idea comes from and it would be great to also have that functionality for snapshot/restore (cold migration). I do have one concern: I'm afraid that doing this in Vm::resume() might cause network weirdness when we just did pause/resume() on the same host - which is valid behavior.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a.) I don't think the RARP requests should do that
b.) That's why I said it can be easily gated on if the Net was instatiated with a State struct or not - it won't be if it's just a pause-resume on the same host but it will be for restore or live migration.

I would rather use the signals we already have rather than add code.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah okay! I agree with you, makes sense. This is a cleaner design!

We might need vm.trigger_post_migration_announcements() in a dedicated endpoint as well but this is an orthogonal discussion and a dedicated PR. I'm not the expert there but @amphi is.

@amphi you plan to do this in a follow-up I guess, yeah?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please can we fix this in this PR.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we only know whether it was instantiated with a State struct or not in the constructor, right? We don't store this information for later. But at this point we cannot run the announcers (maybe we can run the RARP announcer, but for the guest announcement we need the interrupt_cb, which I think is installed during activation).

We could set announce_pending in the constructor if the device was instantiated from a State, and then during activation check whether we have to do announcements.

Is this what you have envisioned?

Adding the endpoint later would still be easy enough.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right - I was leaving that detail to you:

If you put these: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/8263/changes#diff-871d3317222951a9828d52adbf479d80ad1d43c3f192a186d65ea445502a833bR477-R483 into a struct - you can then put it into an Option<..> and set it only when it's been restored or however you see fit.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made the changes necessary to always do announcements if the device was constructed from a State.

@amphi
amphi force-pushed the upstream-virtio-net-announcements branch 2 times, most recently from 8e5c605 to aacd77c Compare June 8, 2026 09:32
@amphi
amphi requested a review from rbradford June 8, 2026 09:41
@amphi
amphi force-pushed the upstream-virtio-net-announcements branch from aacd77c to fd3d8e4 Compare June 8, 2026 10:51
Comment thread virtio-devices/src/vhost_user/net.rs Outdated
Comment on lines +274 to +296
/// Return the guest-visible virtio-net config, recomputing `status` from the
/// current state of the device.
fn config_with_status(&self) -> VirtioNetConfig {
let mut config = self.config;

// We want to recompute the guest-visible status field from the current state of
// the device. We clear this field first to avoid showing stale data.
config.status = 0;

if self
.vu_common
.virtio_common
.feature_acked(VIRTIO_NET_F_STATUS.into())
{
config.status |= VIRTIO_NET_S_LINK_UP as u16;

if self.announce_pending.load(Ordering::Acquire) {
config.status |= VIRTIO_NET_S_ANNOUNCE as u16;
}
}

config
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would't it be cleaner to ensure that self.config was correct at all points in time. Rather than just updating a copy of it here when capturing the state?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we only really use it here, I see no value in adding the code to every path that may or may not change it to be honest.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then do it as a update method that modifies it in place or construct it on demand. Don't take what's there, modify it and then derive from it. That's just confusing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed this to construct the returned config more directly.

@amphi
amphi force-pushed the upstream-virtio-net-announcements branch from fd3d8e4 to c0942ec Compare June 8, 2026 12:39

@rbradford rbradford left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to refactor so that more code is shared between virtio-net and vhost-user-net?

@amphi

amphi commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

Is it possible to refactor so that more code is shared between virtio-net and vhost-user-net?

I don't think there is a lot of code that can be shared, partly due to the subtle differences between virtio-net and vhost-user-net (e.g. self.common.feature_acked vs self.vu_common.virtio_common.feature_acked). There are really small parts that could be shared, but this would require introducing an extra layer of abstractions, which makes the code more confusing to read.

Thus I don't think there are any sensible refactors to share more code between virtio-net and vhost-user-net.

@amphi
amphi force-pushed the upstream-virtio-net-announcements branch from c0942ec to ab9bf51 Compare June 9, 2026 07:38
Comment thread virtio-devices/src/net.rs Outdated
Done,
}

/// [`NetCtrlEpollHandler`] is shared between virtio-net and vhost-user-net, but

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this comment doesn't really hit the right tone. This sounds like the why part which is very unusual as the first sentence for rustdoc of a type.

@rbradford rbradford left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is suffering a little bit from too many comments hiding what is important. Can you take a critical eye over this to make sure it's as a minimal as possible

Comment thread net_util/src/lib.rs Outdated

fn vnet_hdr_len() -> usize {
/// Returns the virtio-net header size configured on TAP file descriptors.
///

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think comment is excessive

Comment thread virtio-devices/src/net.rs
Comment on lines +527 to 575
let (avail_features, acked_features, config, queue_sizes, paused, announce_pending) =
if let Some(state) = state {
info!("Restoring virtio-net {id}");
// Always set [`Self::announce_pending`] to true if the device was restored to
// make sure the device announces itself.
(
state.avail_features,
state.acked_features,
state.config,
state.queue_size,
true,
true,
)
} else {
let mut avail_features = (1 << VIRTIO_RING_F_EVENT_IDX) | (1 << VIRTIO_F_VERSION_1);

if access_platform_enabled {
avail_features |= 1u64 << VIRTIO_F_ACCESS_PLATFORM;
}
if mtu.is_some() {
avail_features |= 1 << VIRTIO_NET_F_MTU;
}

// Configure TSO/UFO features when hardware checksum offload is enabled.
if offload_csum {
avail_features |= (1 << VIRTIO_NET_F_CSUM)
| (1 << VIRTIO_NET_F_GUEST_CSUM)
| (1 << VIRTIO_NET_F_CTRL_GUEST_OFFLOADS);

if offload_tso {
avail_features |= (1 << VIRTIO_NET_F_HOST_ECN)
| (1 << VIRTIO_NET_F_HOST_TSO4)
| (1 << VIRTIO_NET_F_HOST_TSO6)
| (1 << VIRTIO_NET_F_GUEST_ECN)
| (1 << VIRTIO_NET_F_GUEST_TSO4)
| (1 << VIRTIO_NET_F_GUEST_TSO6);
if access_platform_enabled {
avail_features |= 1u64 << VIRTIO_F_ACCESS_PLATFORM;
}

if offload_ufo {
avail_features |= (1 << VIRTIO_NET_F_HOST_UFO) | (1 << VIRTIO_NET_F_GUEST_UFO);
// Configure TSO/UFO features when hardware checksum offload is enabled.
if offload_csum {
avail_features |= (1 << VIRTIO_NET_F_CSUM)
| (1 << VIRTIO_NET_F_GUEST_CSUM)
| (1 << VIRTIO_NET_F_CTRL_GUEST_OFFLOADS);

if offload_tso {
avail_features |= (1 << VIRTIO_NET_F_HOST_ECN)
| (1 << VIRTIO_NET_F_HOST_TSO4)
| (1 << VIRTIO_NET_F_HOST_TSO6)
| (1 << VIRTIO_NET_F_GUEST_ECN)
| (1 << VIRTIO_NET_F_GUEST_TSO4)
| (1 << VIRTIO_NET_F_GUEST_TSO6);
}

if offload_ufo {
avail_features |=
(1 << VIRTIO_NET_F_HOST_UFO) | (1 << VIRTIO_NET_F_GUEST_UFO);
}
}
}

avail_features |= 1 << VIRTIO_NET_F_CTRL_VQ;
let queue_num = num_queues + 1;
avail_features |= 1 << VIRTIO_NET_F_CTRL_VQ;
avail_features |= 1 << VIRTIO_NET_F_STATUS;
let queue_num = num_queues + 1;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are there so many lines changed here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added announce_pending in line 527, that made the line break, and that made many lines shift to the right. In our fork, we added a refactor first, so it was easier to see what really changed. This is was the refactor that you didn't like and that I reverted.

@amphi
amphi force-pushed the upstream-virtio-net-announcements branch 3 times, most recently from df28237 to 707e908 Compare June 10, 2026 11:21
Comment thread virtio-devices/src/seccomp_filters.rs Outdated
Comment on lines +185 to +188
(libc::SYS_socket, vec![]),
(libc::SYS_getsockname, vec![]),
(libc::SYS_timerfd_settime, vec![]),
(libc::SYS_ioctl, create_virtio_net_ctl_ioctl_seccomp_rule()),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should be in alphabetical order like the rest of the file.

@amphi
amphi force-pushed the upstream-virtio-net-announcements branch from 707e908 to 6ba7725 Compare June 10, 2026 16:37
@rbradford
rbradford marked this pull request as draft June 12, 2026 11:31
@amphi
amphi force-pushed the upstream-virtio-net-announcements branch 4 times, most recently from 6e8c44c to 04134c2 Compare June 19, 2026 19:48

@rbradford rbradford left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is looking a lot cleaner now - thanks for the iterations!

Comment thread virtio-devices/src/seccomp_filters.rs Outdated
(libc::SYS_ioctl, create_virtio_net_ctl_ioctl_seccomp_rule()),
(libc::SYS_socket, vec![]),
(libc::SYS_timerfd_settime, vec![]),
(libc::SYS_write, vec![]),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SYS_write Already included in common

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that was a debugging leftover, I removed it.

Comment thread virtio-devices/src/seccomp_filters.rs Outdated
fn virtio_net_ctl_thread_rules() -> Vec<(i64, Vec<SeccompRule>)> {
vec![(libc::SYS_ioctl, create_virtio_net_ctl_ioctl_seccomp_rule())]
vec![
(libc::SYS_getsockname, vec![]),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this actually needed?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debugging leftover, I removed it.

Comment thread virtio-devices/src/seccomp_filters.rs Outdated
vec![
(libc::SYS_getsockname, vec![]),
(libc::SYS_ioctl, create_virtio_net_ctl_ioctl_seccomp_rule()),
(libc::SYS_socket, vec![]),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debugging leftover, I removed it.

@@ -186,7 +186,13 @@ fn create_virtio_net_ctl_ioctl_seccomp_rule() -> Vec<SeccompRule> {
}

fn virtio_net_ctl_thread_rules() -> Vec<(i64, Vec<SeccompRule>)> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The syscall changes should be folded into the commit that needs them to make it bisectable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment thread virtio-devices/src/vhost_user/net.rs Outdated
Comment on lines +56 to +58
/// When signaled, the epoll thread will do the post-migration
/// announcements.
announce_evt: EventFd,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not just about migration - it triggers on restore from a snapshot too. Please cleanup your comments/commit messages etc to cover the broader restore case.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, in case I was not, you should make sure this functionality works correctly for restore from a snapshot too (offline migration).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the commit messages and comments so it now states that these are post-migration and post-restore announcers.

Comment thread virtio-devices/src/net.rs Outdated
if let Some(state) = state {
info!("Restoring virtio-net {id}");
// Always set [`Self::announce_pending`] to true if the device was restored to
// make sure the device announces itself.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you're always going to set this to true - why bother storing it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not always true. It is set to true only for the restore-from-state path, where we need a pending post-migration announcement.

For the normal fresh-device path it remains false.

So the field tracks transient pending work: whether this device still needs to announce itself after restore. I can make that clearer by deriving the initial value from state.is_some() before consuming the restored NetState.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What i'm saying is why store it in the NetState if you are always going to set it.

Comment thread virtio-devices/src/net.rs Outdated
Comment on lines +1105 to +1130
fn send_guest_announce(&mut self) -> AnnounceOutcome {
if !self.guest_announce_negotiated {
// [`Net::announce_pending`] does double duty: we use it to signal that the
// driver in the guest has to send announcements, but we also use it to signal
// that the device has been constructed from a `State` (e.g. after a
// live-migration or during snapshot restore). Thus, it may happen that
// VIRTIO_NET_F_GUEST_ANNOUNCE was not negotiated, but [`Net::announce_pending`]
// is set. In this case, we just clear it here.
self.announce_pending.store(false, Ordering::Release);
return AnnounceOutcome::Done;
}

// If the guest hasn't ack'ed the announce, we trigger the interrupt.
if self.announce_pending.load(Ordering::Acquire) {
self.interrupt_cb
.trigger(VirtioInterruptType::Config)
.inspect_err(|e| {
warn!("Unable to send interrupt for virtio-net device: {e}");
})
.ok();

// We have to check again whether the driver ack'ed the announcement.
return AnnounceOutcome::Retry;
}
AnnounceOutcome::Done
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QEMU limits it to 3 notifications (in case the guest never ACKs it) we should do something similar otherwise we're going to keep interrupting.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By default Qemu does it five times (see https://github.com/qemu/qemu/blob/master/migration/options.c#L81), and I am doing the same now.

Comment thread virtio-devices/src/net.rs Outdated
Comment on lines +494 to +495
#[serde(default)]
pub announce_pending: bool,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yeah right, that is obsolete now.

Comment thread virtio-devices/src/net.rs Outdated
if let Some(state) = state {
info!("Restoring virtio-net {id}");
// Always set [`Self::announce_pending`] to true if the device was restored to
// make sure the device announces itself.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What i'm saying is why store it in the NetState if you are always going to set it.

@amphi
amphi force-pushed the upstream-virtio-net-announcements branch 3 times, most recently from 4cd5c94 to 6e2b233 Compare June 22, 2026 12:32
@amphi
amphi marked this pull request as ready for review June 22, 2026 13:49
@amphi
amphi requested a review from rbradford June 24, 2026 09:11
@rbradford

Copy link
Copy Markdown
Member

@amphi Needs a rebase

Expose `VIRTIO_NET_S_LINK_UP` through the virtio-net config status field
when `VIRTIO_NET_F_STATUS` was negotiated.

This makes the guest-visible status bits reflect the device runtime
state and prepares the config status path used by later post-migration
announce handling.

On-behalf-of: SAP [email protected]
Signed-off-by: Sebastian Eydam <[email protected]>
@amphi
amphi force-pushed the upstream-virtio-net-announcements branch from 6e2b233 to 18e7816 Compare July 1, 2026 08:16
@amphi

amphi commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

@rbradford I just rebased.

@rbradford rbradford left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you try and reduce the duplication?

Comment thread virtio-devices/src/vhost_user/net.rs Outdated

impl AnnounceOps for VhostUserNetAnnounceOps {
fn initialize(&mut self) {
self.generation = self.announce_generation.load(Ordering::Acquire);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the virtio-net version you set announcements_done to 0 here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This struct is gone now.

Comment thread virtio-devices/src/vhost_user/net.rs Outdated
Comment on lines +524 to +528
pub interrupt_cb: Arc<dyn VirtioInterrupt>,
pub guest_announce_negotiated: bool,
pub announce_pending: Arc<AtomicBool>,
pub announce_generation: Arc<AtomicU64>,
pub generation: u64,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these need to be pub?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This struct is gone now.

Comment thread virtio-devices/src/net.rs
state.config,
state.queue_size,
true,
true,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In vhost-user-net announce_pending is gated on VIRTIO_NET_F_GUEST_ANNOUNCE being acked.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but for virtio-net we can also just do the host-side RARP announcements. So we unconditionally set this to true and check later which announcements can be done.

Comment thread docs/snapshot_restore.md Outdated
Comment on lines +105 to +112
When the restored VM resumes, Cloud Hypervisor asks supported network devices
to re-announce the VM on the network. For `virtio-net`, the current
implementation sets `VIRTIO_NET_S_ANNOUNCE`, raises a config interrupt,
retries that request a few times in the background, and also sends host-side
RARP announcements on the TAP interfaces. A guest re-announcement therefore
only happens when the guest negotiated `VIRTIO_NET_F_GUEST_ANNOUNCE`. For
`vhost-user-net`, the current implementation only uses the guest announcement
path.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph is duplicated everywhere - just add a section about announcement to live migration and reference it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@rbradford

Copy link
Copy Markdown
Member

@amphi I asked Claude to give consolidating it a go: main...rbradford:cloud-hypervisor:202607/virtio-net-consolidated net 300 fewer lines.

Advertise `VIRTIO_NET_F_GUEST_ANNOUNCE` on virtio-net devices, surface
`VIRTIO_NET_S_ANNOUNCE` through config status, and handle
`VIRTIO_NET_CTRL_ANNOUNCE_ACK` on the control queue.

This adds the guest-visible state needed for post-migration or
post-restore announce requests; the VMM side triggering is added in
follow-up commits.

The motivation is to reduce post-migration and post-restore
connectivity gap. After a live migration or after restoring, it can
take the guest several seconds to be reachable again over the network.
With these announcements, the network path should be refreshed within a
few milliseconds.

On-behalf-of: SAP [email protected]
Signed-off-by: Sebastian Eydam <[email protected]>
@amphi
amphi force-pushed the upstream-virtio-net-announcements branch from 18e7816 to 629155a Compare July 2, 2026 11:45
@amphi

amphi commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

@amphi I asked Claude to give consolidating it a go: main...rbradford:cloud-hypervisor:202607/virtio-net-consolidated net 300 fewer lines.

I took inspiration from that branch and applied the changes I deemed sensible. E.g. moving status_bits into GuestAnnounce doesn't seem like a very clean abstraction to me.

There are a few less LOC now.

@rbradford rbradford left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm - thanks for the patience on the interations!

@rbradford
rbradford added this pull request to the merge queue Jul 2, 2026
Merged via the queue into cloud-hypervisor:main with commit 53a00c0 Jul 2, 2026
39 checks passed
@phip1611

phip1611 commented Jul 2, 2026

Copy link
Copy Markdown
Member

Awesome to have that! Thanks everyone for patience, review, testing and engineering!

Our Cyberus Technology patchset is getting shorter and shorter 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new-feature New feature to include in release notes

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

5 participants