Post-migration network announcements - #8263
Conversation
phip1611
left a comment
There was a problem hiding this comment.
Great work! 🚀 Left a few comments.
PS: If I'm not mistaken, this introduces VMM-driven RARP-packages and guest-driven GARP packages, yes? I think the latter is not entirely clear from the description IMHO
| avail_features: u64, | ||
| acked_features: u64, | ||
| config: VirtioNetConfig, | ||
| queue_sizes: Vec<u16>, |
There was a problem hiding this comment.
while on it: I think you could refactor this easily to Box<[u16]>. This makes clear that this data structure does not resize once constructed. I just tried it locally and it is pretty forward. This would nicely align with changes such as #8231
There was a problem hiding this comment.
VirtioCommon uses Vec<u16> for queue_sizes too, I don't think we gain a lot by changing it here but not in VirtioCommon. I think this should be done in one PR.
| let _ = thread::Builder::new() | ||
| .name("post-migration-announcers".to_string()) | ||
| .spawn(move || { | ||
| for round in 0..rounds { | ||
| info!("Post migration announce (async): {}/{}", round + 1, rounds); | ||
|
|
||
| // The first announcement already was done synchronously, thus | ||
| // we sleep at the start of the loop. | ||
|
|
||
| let delay = (initial_delay + step_delay.saturating_mul(round)).min(max_delay); | ||
| debug!("Sleeping {}ms", delay.as_millis()); | ||
| thread::sleep(delay); | ||
|
|
||
| announcers.iter_mut().for_each(|a| a.announce()); | ||
| } | ||
| }); |
There was a problem hiding this comment.
You can't just spin up threads and not track them - they need to exit on exit_evt - need join, etc. Is all this even necessary - I thought if you used libvirt it handles the TAP creation so why are you doing this in CH?
There was a problem hiding this comment.
I added life cycle management, please take another look.
| match unsafe { | ||
| libc::write( | ||
| tap.as_raw_fd(), | ||
| buf.as_ptr() as *const libc::c_void, | ||
| buf.len(), | ||
| ) |
There was a problem hiding this comment.
The tap implements Write so you don't need this unsafe libc
There was a problem hiding this comment.
Good catch, I am using the taps Write now.
The motivation is to reduce the post-migration connectivity gap. In our tests, after live migration the guest sometimes only became reachable again after several seconds, in the worst case around 20s. With post-migration announcements, the network path was refreshed within a few milliseconds. So the goal is to make network connectivity recover promptly after migration. I think Cloud Hypervisor is the right layer to trigger this for two reasons: The VMM knows the precise point at which the destination VM has resumed and the migrated virtio-net device is active again. Management software can observe that migration completed, but it does not necessarily know the exact device-level point where an announcement is both useful and safe. There is also a practical split of responsibility here: management software can orchestrate migration, but the hypervisor owns the emulated device state and the post-resume device semantics. Implementing this in every management stack would duplicate hypervisor/device-specific knowledge and would still require a VMM-specific API for operations such as triggering the virtio config interrupt. The host-side RARP and the guest-side This also matches existing hypervisor behavior. QEMU handles virtio-net guest announcements from the VMM/device side after migration, rather than relying purely on management software. So my view is that Cloud Hypervisor should provide the automatic post-migration announcement behavior as part of virtio-net migration handling. A management API to manually trigger announcements could still be useful as an additional control surface, but it should not be the only mechanism required for the normal live-migration path. |
|
I think PR still has planned changes? |
1cc9ec7 to
10fcd29
Compare
Yes, I just pushed some changes. It took me a while to get the life cycle management of the new thread right. |
| /// Tracks whether the guest still needs to acknowledge a post-migration | ||
| /// announce request through the control queue. |
There was a problem hiding this comment.
I think this comment is not needed - this member has a really nice clear name.
| avail_features: u64, | ||
| acked_features: u64, | ||
| config: VirtioNetConfig, | ||
| announce_pending: bool, |
There was a problem hiding this comment.
I think you need to add #[serde(default)]
| /// Tracks whether the guest still needs to acknowledge a post-migration | ||
| /// announce request through the control queue. |
There was a problem hiding this comment.
Ditto - this member as a clear name.
|
|
||
| /// Constructor-time copy of the fields needed to initialize the live device |
There was a problem hiding this comment.
Why is "Constructor-time" hyphenated and what does that even mean? I hate this refactoring. It makes it hard to read the code and we don't use this pattern anywhere else in the project.
There was a problem hiding this comment.
The reason for this refactoring was reviewability. In our downstream branch, adding the announce-related state directly to the existing constructor caused a lot of surrounding code movement, which made the functional change harder to review.
That said, I understand the concern about introducing a pattern that is not used elsewhere in the project. I do not feel strongly about keeping this refactoring in the upstream PR, so I will remove it and fold the required changes back into the existing structure.
| offload_tso: bool, | ||
| offload_ufo: bool, | ||
| offload_csum: bool, |
There was a problem hiding this comment.
A good refactoring would be to create a NetOffloadControl struct holding these.
There was a problem hiding this comment.
The refactoring here was only meant to reduce review noise for this PR by keeping the functional changes more localized.
I do not intend to include additional unrelated refactorings.
| if offload_csum { | ||
| avail_features |= (1 << VIRTIO_NET_F_CSUM) | ||
| | (1 << VIRTIO_NET_F_GUEST_CSUM) | ||
| | (1 << VIRTIO_NET_F_CTRL_GUEST_OFFLOADS); | ||
|
|
||
| if offload_tso { | ||
| avail_features |= (1 << VIRTIO_NET_F_HOST_ECN) | ||
| | (1 << VIRTIO_NET_F_HOST_TSO4) | ||
| | (1 << VIRTIO_NET_F_HOST_TSO6) | ||
| | (1 << VIRTIO_NET_F_GUEST_ECN) | ||
| | (1 << VIRTIO_NET_F_GUEST_TSO4) | ||
| | (1 << VIRTIO_NET_F_GUEST_TSO6); | ||
| } | ||
|
|
||
| if offload_ufo { | ||
| avail_features |= (1 << VIRTIO_NET_F_HOST_UFO) | (1 << VIRTIO_NET_F_GUEST_UFO); | ||
| } | ||
| } |
There was a problem hiding this comment.
You could then turn that struct into features via a method on it.
| pub struct CtrlQueue { | ||
| pub taps: Vec<Tap>, | ||
| /// Tracks whether the guest still needs to acknowledge a post-migration | ||
| /// announce request through the control queue. |
| exit_evt: EventFd, | ||
| } | ||
|
|
||
| impl PostMigrationAnnouncerTask { |
There was a problem hiding this comment.
This is all way too complicated and overengineered. Can't you just handle it in the virtio-net thread in the destination VM. You know if you've been migrated or restored as it will come through the construction with a State struct.
There was a problem hiding this comment.
This does not work.
At that point the device object is being created, but the destination VM has not necessarily resumed yet and the datapath is not necessarily at the point where an announcement can refresh the external network state. The announcement needs to happen after incoming migration, once the migrated device is active on the destination side.
We also observed that a single announcement is not always sufficient. In our downstream setup we repeat the announcements for a short window after migration, and we even added a REST API endpoint to trigger them manually a few seconds after migration completed. That was useful in practice because the timing depends not only on the device construction, but also on when the VM, guest networking, tap/bridge setup, and surrounding network have actually settled.
So I would prefer to keep the trigger tied to the post-migration/resume path rather than the constructor. I can change the implementation if you have any suggestions, but I do not think moving the announcement request into construction would preserve the behavior we need.
There was a problem hiding this comment.
The announcement needs to happen after incoming migration, once the migrated device is active on the destination side.
A little more specific:
The announcement needs to happen after a successfully resumed VM (coming from an incoming migration). Otherwise the network's routing information could be in a weird state if the migration failed and keeps running on the source but the "network announcements" happened already on the destination.
There was a problem hiding this comment.
The virtio-net epoll thread is only running once the VM is resumed. Put a timerfd into the epoll to handle the periodic notifications.
There was a problem hiding this comment.
@rbradford I moved the retry logic into the epoll thread.
b0eff3a to
94023cb
Compare
| ); | ||
|
|
||
| // Advertise new VM location to network switches. | ||
| vm.trigger_post_migration_announcements(); |
There was a problem hiding this comment.
Can you explain why you can't just do this on resume? This announcement functionality make sense on snapshot restore (which is a kind of offline migration)
You can avoid doing it on a resume without restore or migration by only enabling it if the Net was constructed with a State.
There was a problem hiding this comment.
I see where this idea comes from and it would be great to also have that functionality for snapshot/restore (cold migration). I do have one concern: I'm afraid that doing this in Vm::resume() might cause network weirdness when we just did pause/resume() on the same host - which is valid behavior.
There was a problem hiding this comment.
a.) I don't think the RARP requests should do that
b.) That's why I said it can be easily gated on if the Net was instatiated with a State struct or not - it won't be if it's just a pause-resume on the same host but it will be for restore or live migration.
I would rather use the signals we already have rather than add code.
There was a problem hiding this comment.
Ah okay! I agree with you, makes sense. This is a cleaner design!
We might need vm.trigger_post_migration_announcements() in a dedicated endpoint as well but this is an orthogonal discussion and a dedicated PR. I'm not the expert there but @amphi is.
@amphi you plan to do this in a follow-up I guess, yeah?
There was a problem hiding this comment.
Please can we fix this in this PR.
There was a problem hiding this comment.
But we only know whether it was instantiated with a State struct or not in the constructor, right? We don't store this information for later. But at this point we cannot run the announcers (maybe we can run the RARP announcer, but for the guest announcement we need the interrupt_cb, which I think is installed during activation).
We could set announce_pending in the constructor if the device was instantiated from a State, and then during activation check whether we have to do announcements.
Is this what you have envisioned?
Adding the endpoint later would still be easy enough.
There was a problem hiding this comment.
Right - I was leaving that detail to you:
If you put these: https://github.com/cloud-hypervisor/cloud-hypervisor/pull/8263/changes#diff-871d3317222951a9828d52adbf479d80ad1d43c3f192a186d65ea445502a833bR477-R483 into a struct - you can then put it into an Option<..> and set it only when it's been restored or however you see fit.
There was a problem hiding this comment.
I made the changes necessary to always do announcements if the device was constructed from a State.
8e5c605 to
aacd77c
Compare
aacd77c to
fd3d8e4
Compare
| /// Return the guest-visible virtio-net config, recomputing `status` from the | ||
| /// current state of the device. | ||
| fn config_with_status(&self) -> VirtioNetConfig { | ||
| let mut config = self.config; | ||
|
|
||
| // We want to recompute the guest-visible status field from the current state of | ||
| // the device. We clear this field first to avoid showing stale data. | ||
| config.status = 0; | ||
|
|
||
| if self | ||
| .vu_common | ||
| .virtio_common | ||
| .feature_acked(VIRTIO_NET_F_STATUS.into()) | ||
| { | ||
| config.status |= VIRTIO_NET_S_LINK_UP as u16; | ||
|
|
||
| if self.announce_pending.load(Ordering::Acquire) { | ||
| config.status |= VIRTIO_NET_S_ANNOUNCE as u16; | ||
| } | ||
| } | ||
|
|
||
| config | ||
| } |
There was a problem hiding this comment.
Would't it be cleaner to ensure that self.config was correct at all points in time. Rather than just updating a copy of it here when capturing the state?
There was a problem hiding this comment.
Since we only really use it here, I see no value in adding the code to every path that may or may not change it to be honest.
There was a problem hiding this comment.
Then do it as a update method that modifies it in place or construct it on demand. Don't take what's there, modify it and then derive from it. That's just confusing.
There was a problem hiding this comment.
I changed this to construct the returned config more directly.
fd3d8e4 to
c0942ec
Compare
rbradford
left a comment
There was a problem hiding this comment.
Is it possible to refactor so that more code is shared between virtio-net and vhost-user-net?
I don't think there is a lot of code that can be shared, partly due to the subtle differences between virtio-net and vhost-user-net (e.g. Thus I don't think there are any sensible refactors to share more code between virtio-net and vhost-user-net. |
c0942ec to
ab9bf51
Compare
| Done, | ||
| } | ||
|
|
||
| /// [`NetCtrlEpollHandler`] is shared between virtio-net and vhost-user-net, but |
There was a problem hiding this comment.
I think this comment doesn't really hit the right tone. This sounds like the why part which is very unusual as the first sentence for rustdoc of a type.
rbradford
left a comment
There was a problem hiding this comment.
I think this is suffering a little bit from too many comments hiding what is important. Can you take a critical eye over this to make sure it's as a minimal as possible
|
|
||
| fn vnet_hdr_len() -> usize { | ||
| /// Returns the virtio-net header size configured on TAP file descriptors. | ||
| /// |
| let (avail_features, acked_features, config, queue_sizes, paused, announce_pending) = | ||
| if let Some(state) = state { | ||
| info!("Restoring virtio-net {id}"); | ||
| // Always set [`Self::announce_pending`] to true if the device was restored to | ||
| // make sure the device announces itself. | ||
| ( | ||
| state.avail_features, | ||
| state.acked_features, | ||
| state.config, | ||
| state.queue_size, | ||
| true, | ||
| true, | ||
| ) | ||
| } else { | ||
| let mut avail_features = (1 << VIRTIO_RING_F_EVENT_IDX) | (1 << VIRTIO_F_VERSION_1); | ||
|
|
||
| if access_platform_enabled { | ||
| avail_features |= 1u64 << VIRTIO_F_ACCESS_PLATFORM; | ||
| } | ||
| if mtu.is_some() { | ||
| avail_features |= 1 << VIRTIO_NET_F_MTU; | ||
| } | ||
|
|
||
| // Configure TSO/UFO features when hardware checksum offload is enabled. | ||
| if offload_csum { | ||
| avail_features |= (1 << VIRTIO_NET_F_CSUM) | ||
| | (1 << VIRTIO_NET_F_GUEST_CSUM) | ||
| | (1 << VIRTIO_NET_F_CTRL_GUEST_OFFLOADS); | ||
|
|
||
| if offload_tso { | ||
| avail_features |= (1 << VIRTIO_NET_F_HOST_ECN) | ||
| | (1 << VIRTIO_NET_F_HOST_TSO4) | ||
| | (1 << VIRTIO_NET_F_HOST_TSO6) | ||
| | (1 << VIRTIO_NET_F_GUEST_ECN) | ||
| | (1 << VIRTIO_NET_F_GUEST_TSO4) | ||
| | (1 << VIRTIO_NET_F_GUEST_TSO6); | ||
| if access_platform_enabled { | ||
| avail_features |= 1u64 << VIRTIO_F_ACCESS_PLATFORM; | ||
| } | ||
|
|
||
| if offload_ufo { | ||
| avail_features |= (1 << VIRTIO_NET_F_HOST_UFO) | (1 << VIRTIO_NET_F_GUEST_UFO); | ||
| // Configure TSO/UFO features when hardware checksum offload is enabled. | ||
| if offload_csum { | ||
| avail_features |= (1 << VIRTIO_NET_F_CSUM) | ||
| | (1 << VIRTIO_NET_F_GUEST_CSUM) | ||
| | (1 << VIRTIO_NET_F_CTRL_GUEST_OFFLOADS); | ||
|
|
||
| if offload_tso { | ||
| avail_features |= (1 << VIRTIO_NET_F_HOST_ECN) | ||
| | (1 << VIRTIO_NET_F_HOST_TSO4) | ||
| | (1 << VIRTIO_NET_F_HOST_TSO6) | ||
| | (1 << VIRTIO_NET_F_GUEST_ECN) | ||
| | (1 << VIRTIO_NET_F_GUEST_TSO4) | ||
| | (1 << VIRTIO_NET_F_GUEST_TSO6); | ||
| } | ||
|
|
||
| if offload_ufo { | ||
| avail_features |= | ||
| (1 << VIRTIO_NET_F_HOST_UFO) | (1 << VIRTIO_NET_F_GUEST_UFO); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| avail_features |= 1 << VIRTIO_NET_F_CTRL_VQ; | ||
| let queue_num = num_queues + 1; | ||
| avail_features |= 1 << VIRTIO_NET_F_CTRL_VQ; | ||
| avail_features |= 1 << VIRTIO_NET_F_STATUS; | ||
| let queue_num = num_queues + 1; | ||
|
|
There was a problem hiding this comment.
Why are there so many lines changed here?
There was a problem hiding this comment.
I added announce_pending in line 527, that made the line break, and that made many lines shift to the right. In our fork, we added a refactor first, so it was easier to see what really changed. This is was the refactor that you didn't like and that I reverted.
df28237 to
707e908
Compare
| (libc::SYS_socket, vec![]), | ||
| (libc::SYS_getsockname, vec![]), | ||
| (libc::SYS_timerfd_settime, vec![]), | ||
| (libc::SYS_ioctl, create_virtio_net_ctl_ioctl_seccomp_rule()), |
There was a problem hiding this comment.
These should be in alphabetical order like the rest of the file.
707e908 to
6ba7725
Compare
6e8c44c to
04134c2
Compare
rbradford
left a comment
There was a problem hiding this comment.
I think this is looking a lot cleaner now - thanks for the iterations!
| (libc::SYS_ioctl, create_virtio_net_ctl_ioctl_seccomp_rule()), | ||
| (libc::SYS_socket, vec![]), | ||
| (libc::SYS_timerfd_settime, vec![]), | ||
| (libc::SYS_write, vec![]), |
There was a problem hiding this comment.
SYS_write Already included in common
There was a problem hiding this comment.
Yeah that was a debugging leftover, I removed it.
| fn virtio_net_ctl_thread_rules() -> Vec<(i64, Vec<SeccompRule>)> { | ||
| vec![(libc::SYS_ioctl, create_virtio_net_ctl_ioctl_seccomp_rule())] | ||
| vec![ | ||
| (libc::SYS_getsockname, vec![]), |
There was a problem hiding this comment.
Debugging leftover, I removed it.
| vec![ | ||
| (libc::SYS_getsockname, vec![]), | ||
| (libc::SYS_ioctl, create_virtio_net_ctl_ioctl_seccomp_rule()), | ||
| (libc::SYS_socket, vec![]), |
There was a problem hiding this comment.
Debugging leftover, I removed it.
| @@ -186,7 +186,13 @@ fn create_virtio_net_ctl_ioctl_seccomp_rule() -> Vec<SeccompRule> { | |||
| } | |||
|
|
|||
| fn virtio_net_ctl_thread_rules() -> Vec<(i64, Vec<SeccompRule>)> { | |||
There was a problem hiding this comment.
The syscall changes should be folded into the commit that needs them to make it bisectable.
| /// When signaled, the epoll thread will do the post-migration | ||
| /// announcements. | ||
| announce_evt: EventFd, |
There was a problem hiding this comment.
It's not just about migration - it triggers on restore from a snapshot too. Please cleanup your comments/commit messages etc to cover the broader restore case.
There was a problem hiding this comment.
To be clear, in case I was not, you should make sure this functionality works correctly for restore from a snapshot too (offline migration).
There was a problem hiding this comment.
I changed the commit messages and comments so it now states that these are post-migration and post-restore announcers.
| if let Some(state) = state { | ||
| info!("Restoring virtio-net {id}"); | ||
| // Always set [`Self::announce_pending`] to true if the device was restored to | ||
| // make sure the device announces itself. |
There was a problem hiding this comment.
If you're always going to set this to true - why bother storing it?
There was a problem hiding this comment.
This is not always true. It is set to true only for the restore-from-state path, where we need a pending post-migration announcement.
For the normal fresh-device path it remains false.
So the field tracks transient pending work: whether this device still needs to announce itself after restore. I can make that clearer by deriving the initial value from state.is_some() before consuming the restored NetState.
There was a problem hiding this comment.
What i'm saying is why store it in the NetState if you are always going to set it.
| fn send_guest_announce(&mut self) -> AnnounceOutcome { | ||
| if !self.guest_announce_negotiated { | ||
| // [`Net::announce_pending`] does double duty: we use it to signal that the | ||
| // driver in the guest has to send announcements, but we also use it to signal | ||
| // that the device has been constructed from a `State` (e.g. after a | ||
| // live-migration or during snapshot restore). Thus, it may happen that | ||
| // VIRTIO_NET_F_GUEST_ANNOUNCE was not negotiated, but [`Net::announce_pending`] | ||
| // is set. In this case, we just clear it here. | ||
| self.announce_pending.store(false, Ordering::Release); | ||
| return AnnounceOutcome::Done; | ||
| } | ||
|
|
||
| // If the guest hasn't ack'ed the announce, we trigger the interrupt. | ||
| if self.announce_pending.load(Ordering::Acquire) { | ||
| self.interrupt_cb | ||
| .trigger(VirtioInterruptType::Config) | ||
| .inspect_err(|e| { | ||
| warn!("Unable to send interrupt for virtio-net device: {e}"); | ||
| }) | ||
| .ok(); | ||
|
|
||
| // We have to check again whether the driver ack'ed the announcement. | ||
| return AnnounceOutcome::Retry; | ||
| } | ||
| AnnounceOutcome::Done | ||
| } |
There was a problem hiding this comment.
QEMU limits it to 3 notifications (in case the guest never ACKs it) we should do something similar otherwise we're going to keep interrupting.
There was a problem hiding this comment.
By default Qemu does it five times (see https://github.com/qemu/qemu/blob/master/migration/options.c#L81), and I am doing the same now.
| #[serde(default)] | ||
| pub announce_pending: bool, |
There was a problem hiding this comment.
Ah yeah right, that is obsolete now.
| if let Some(state) = state { | ||
| info!("Restoring virtio-net {id}"); | ||
| // Always set [`Self::announce_pending`] to true if the device was restored to | ||
| // make sure the device announces itself. |
There was a problem hiding this comment.
What i'm saying is why store it in the NetState if you are always going to set it.
4cd5c94 to
6e2b233
Compare
|
@amphi Needs a rebase |
Expose `VIRTIO_NET_S_LINK_UP` through the virtio-net config status field when `VIRTIO_NET_F_STATUS` was negotiated. This makes the guest-visible status bits reflect the device runtime state and prepares the config status path used by later post-migration announce handling. On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
6e2b233 to
18e7816
Compare
|
@rbradford I just rebased. |
rbradford
left a comment
There was a problem hiding this comment.
Could you try and reduce the duplication?
|
|
||
| impl AnnounceOps for VhostUserNetAnnounceOps { | ||
| fn initialize(&mut self) { | ||
| self.generation = self.announce_generation.load(Ordering::Acquire); |
There was a problem hiding this comment.
In the virtio-net version you set announcements_done to 0 here.
There was a problem hiding this comment.
This struct is gone now.
| pub interrupt_cb: Arc<dyn VirtioInterrupt>, | ||
| pub guest_announce_negotiated: bool, | ||
| pub announce_pending: Arc<AtomicBool>, | ||
| pub announce_generation: Arc<AtomicU64>, | ||
| pub generation: u64, |
There was a problem hiding this comment.
This struct is gone now.
| state.config, | ||
| state.queue_size, | ||
| true, | ||
| true, |
There was a problem hiding this comment.
In vhost-user-net announce_pending is gated on VIRTIO_NET_F_GUEST_ANNOUNCE being acked.
There was a problem hiding this comment.
Yes, but for virtio-net we can also just do the host-side RARP announcements. So we unconditionally set this to true and check later which announcements can be done.
| When the restored VM resumes, Cloud Hypervisor asks supported network devices | ||
| to re-announce the VM on the network. For `virtio-net`, the current | ||
| implementation sets `VIRTIO_NET_S_ANNOUNCE`, raises a config interrupt, | ||
| retries that request a few times in the background, and also sends host-side | ||
| RARP announcements on the TAP interfaces. A guest re-announcement therefore | ||
| only happens when the guest negotiated `VIRTIO_NET_F_GUEST_ANNOUNCE`. For | ||
| `vhost-user-net`, the current implementation only uses the guest announcement | ||
| path. |
There was a problem hiding this comment.
This paragraph is duplicated everywhere - just add a section about announcement to live migration and reference it.
|
@amphi I asked Claude to give consolidating it a go: main...rbradford:cloud-hypervisor:202607/virtio-net-consolidated net 300 fewer lines. |
Advertise `VIRTIO_NET_F_GUEST_ANNOUNCE` on virtio-net devices, surface `VIRTIO_NET_S_ANNOUNCE` through config status, and handle `VIRTIO_NET_CTRL_ANNOUNCE_ACK` on the control queue. This adds the guest-visible state needed for post-migration or post-restore announce requests; the VMM side triggering is added in follow-up commits. The motivation is to reduce post-migration and post-restore connectivity gap. After a live migration or after restoring, it can take the guest several seconds to be reachable again over the network. With these announcements, the network path should be refreshed within a few milliseconds. On-behalf-of: SAP [email protected] Signed-off-by: Sebastian Eydam <[email protected]>
18e7816 to
629155a
Compare
I took inspiration from that branch and applied the changes I deemed sensible. E.g. moving There are a few less LOC now. |
rbradford
left a comment
There was a problem hiding this comment.
lgtm - thanks for the patience on the interations!
|
Awesome to have that! Thanks everyone for patience, review, testing and engineering! Our Cyberus Technology patchset is getting shorter and shorter 🚀 |
Summary
Add post-migration network announcements for virtio-net devices so migrated guests
refresh their L2 state on the new host more quickly.
In our tests, connectivity after live migration can take up to 20 seconds
to recover before the VM becomes pingable again. With the post-migration
announcements, the network path is refreshed within a few milliseconds.
The new flow does two things after incoming migration:
virtio-netdevicesVIRTIO_NET_F_GUEST_ANNOUNCEwas negotiatedThis is implemented for both
virtio-netandvhost-user-net, with VMM hooks torun the announcement sequence after the destination VM resumes.
Details
post_migration_announcehook to virtio devices and wire it through the VMMvirtio-netandvhost-user-netVIRTIO_NET_CTRL_ANNOUNCE_ACKwithout a data descriptor
Open Questions