virtio-devices: Signal NEEDS_RESET - #8295
Conversation
| return Err(e); | ||
| } | ||
|
|
||
| self.common.epoll_threads = Some(epoll_threads); |
There was a problem hiding this comment.
This abort_partial_activate() feels a bit messy. The thing it does is assign the threads to the common and then call .reset()
If you checked the activate error here then you could just straight up call .reset() as the partial set of threads would have been assigned to the common?
Alternatively maybe we should do some refactoring and move spawn_virtio_thread to be a method on VirtioCommon and have it clean itself up (also we could remove this assignment from every device.
Could you look at a refactoring like that?
There was a problem hiding this comment.
I'm looking at that refactoring - and it looks like it would be a nice cleanup - watch this space - 👀
There was a problem hiding this comment.
@rbradford rebased and ready for the next round.
d4b2f14 to
c01b4bc
Compare
rbradford
left a comment
There was a problem hiding this comment.
I notice this doesn't need touch all the devices - that's a nice cleanup!
| format_args!("{}: virtio device activation failed: {e:?}", self.id), | ||
| ); | ||
| } else { | ||
| self.device_activated.store(true, Ordering::SeqCst); |
There was a problem hiding this comment.
nit: I know you are just reordering existing code but I think this should be Release and not SeqCst
https://doc.rust-lang.org/std/sync/atomic/enum.Ordering.html#variant.Release
No need to change it here, but I wanted to mention that I used to think of SeqCst as a safe fallback as well. After many fruitful discussions about this, I now think that treating it that way is actually an antipattern. @olivereanderson
| for activator in self.pending_activations.lock().unwrap().drain(..) { | ||
| activator | ||
| .activate() | ||
| .map_err(DeviceManagerError::VirtioActivate)?; |
There was a problem hiding this comment.
Does this make VirtioActivate an unused error type?
There was a problem hiding this comment.
Indeed, good catch. Folded the removal into the first commit.
Thanks
When the guest writes DRIVER_OK and the device fails to activate, the VMM previously bubbled the error up via VirtioActivate and never released the activation barrier, leaving the vCPU that wrote DRIVER_OK blocked on the barrier and effectively deadlocking the guest. Per virtio 1.3 section 2.1.2, a device that has experienced an error it cannot recover from should set DEVICE_NEEDS_RESET in its status and notify the driver via a configuration change interrupt. Do that on activation failure through the existing mark_device_needs_reset helper, then release the activation barrier so the vCPU can resume. DeviceManager::activate_virtio_devices now logs and continues instead of aborting the whole pending list, so one failing device does not take down the VMM or block pause and migration. The activator has already reported the failure with the device id. Signed-off-by: Anatol Belski <[email protected]>
Add TestVirtioDevice with a controllable ActivateResult, TestVirtioInterrupt that records delivered interrupt types, and a make_activator helper that builds a complete VirtioPciDeviceActivator with observable status, activated flag, interrupt log, and barrier. Assisted-by: Claude:Opus-4.7 Signed-off-by: Anatol Belski <[email protected]>
Feed BadActivate into VirtioPciDeviceActivator and assert that the error propagates, device_activated stays false, DEVICE_NEEDS_RESET is set in status, a single Config interrupt is delivered, and a thread waiting on the activation barrier unblocks. The barrier release is the deadlock fixed by the NEEDS_RESET on activation failure change. Assisted-by: Claude:Opus-4.7 Signed-off-by: Anatol Belski <[email protected]>
Feed Ok into VirtioPciDeviceActivator and assert that activate returns Ok, device_activated becomes true, DEVICE_NEEDS_RESET is not set, status is otherwise unchanged, no Config interrupt is delivered, and the barrier waiter unblocks normally. Assisted-by: Claude:Opus-4.7 Signed-off-by: Anatol Belski <[email protected]>
c01b4bc to
928ac67
Compare
|
This CI failure in the VFIO tests looks unrelated. The host log shows the single The failure is the guest NVIDIA driver bailing with which seems a BAR mapping flake. Thanks |
|
It wasn't kicked out of the MQ because VFIO - I don't really understand why it got kicked out at all. |
When a virtio device fails to activate after the guest writes DRIVER_OK, the VMM bubbles the error up and never releases the activation barrier. The vCPU stays blocked and the guest deadlocks.
The fix is to signal
DEVICE_NEEDS_RESET, deliver a config change interrupt, and release the activation barrier so the guest can recover.