Codestin Search App

phip1611 · 2026-04-24T09:18:43Z

Fix of #7990 - 7990 introduced another race condition causing a deadlock. I hope the situation is now resolved once and for all.

TL;DR

In #7990 we replaced the old deadlock with another deadlock. This commit finally resolves (hopefully) all dead locks on that code path by not holding CpuManager::vcpu_states while waiting for vCPU pause acknowledgements. A vCPU can receive the pause kick while servicing the ACPI CPU hotplug MMIO device, and that MMIO path also needs vcpu_states. Holding the mutex across the wait phase deadlocks pause against that MMIO access.

Problem

signal_vcpus() used to lock CpuManager::vcpu_states for the whole function, signal every vCPU, and then wait for each vCPU to acknowledge the kick.

That lock scope is too wide. A vCPU is allowed to observe the kick in userspace rather than returning directly from KVM_RUN. During boot, vcpu0 can be in an MMIO access on the ACPI CPU hotplug device when pause arrives. AcpiCpuHotplugController::read() and write() both lock vcpu_states to inspect or update the selected vCPU state.

The deadlock looks like this:

    VMM thread                           vCPU thread
    ----------                           ----------
    lock(vcpu_states)
    signal_vcpus()
    wait for ack  ---------------------> receives pause kick
                                          enters ACPI CPU hotplug MMIO
                                          lock(vcpu_states)  [blocks]
    wait for ack  <--------------------- cannot set vcpu_run_interrupted

The VMM thread waits for vcpu_run_interrupted to flip, but the vCPU cannot reach the pause acknowledgement path because it is sleeping on the same mutex.

The debug logs matched that cycle exactly: signal delivery kept working, vcpu0 stayed in one unmatched run() invocation, the stuck thread sampled in futex_do_wait, and the backtrace pointed at AcpiCpuHotplugController::read().

Reproducer

This was reproducible by continuously issuing pause() / resume() from while a Linux guest was still booting. That boot-tim window reliably exercises the ACPI CPU hotplug MMIO access that participates in the deadlock. Once the guest had finished booting, the problem became much harder to trigger (as there is no MMIO operation without explicit CPU plugging).

Solution

Keep the existing two-phase behavior so all vCPUs are still signalled before the wait phase, but narrow the lifetime of the vcpu_states mutex. Reacquire it only long enough to access one VcpuState at a time in each phase.

That preserves the original pause semantics and the fast signal-all / wait-all structure, while removing the lock inversion with the ACPI CPU hotplug MMIO path.

This also remains safe if a vCPU is hot-removed while pause is in progress. Hot-remove does not shrink vcpu_states; it stops the thread and clears the VcpuState handle in place. signal_vcpus() can therefore snapshot the vector length up front, and if a vCPU disappears between the signal and wait phases,
wait_until_signal_acknowledged() will observe handle.is_none() and return successfully.

The interruption handshake itself lives in atomics inside each VcpuState. The outer mutex is only needed to reach the state objects, not to keep the acknowledgement protocol correct. Dropping the mutex between iterations therefore does not weaken the pause protocol, but it does allow MMIO handlers and other vcpu_states users to make forward progress while the VMM waits for the kick to be observed.

# TL;DR In cloud-hypervisor#7990 we replaced the old deadlock with another deadlock. This commit finally resolves (hopefully) all dead locks on that code path by not holding `CpuManager::vcpu_states` while waiting for vCPU pause acknowledgements. A vCPU can receive the pause kick while servicing the ACPI CPU hotplug MMIO device, and that MMIO path also needs `vcpu_states`. Holding the mutex across the wait phase deadlocks pause against that MMIO access. # Problem `signal_vcpus()` used to lock `CpuManager::vcpu_states` for the whole function, signal every vCPU, and then wait for each vCPU to acknowledge the kick. That lock scope is too wide. A vCPU is allowed to observe the kick in userspace rather than returning directly from `KVM_RUN`. During boot, `vcpu0` can be in an MMIO access on the ACPI CPU hotplug device when pause arrives. `AcpiCpuHotplugController::read()` and `write()` both lock `vcpu_states` to inspect or update the selected vCPU state. The deadlock looks like this: VMM thread vCPU thread ---------- ---------- lock(vcpu_states) signal_vcpus() wait for ack ---------------------> receives pause kick enters ACPI CPU hotplug MMIO lock(vcpu_states) [blocks] wait for ack <--------------------- cannot set vcpu_run_interrupted The VMM thread waits for `vcpu_run_interrupted` to flip, but the vCPU cannot reach the pause acknowledgement path because it is sleeping on the same mutex. The debug logs matched that cycle exactly: signal delivery kept working, `vcpu0` stayed in one unmatched `run()` invocation, the stuck thread sampled in `futex_do_wait`, and the backtrace pointed at `AcpiCpuHotplugController::read()`. # Reproducer This was reproducible by continuously issuing `pause()` / `resume()` from while a Linux guest was still booting. That boot-tim window reliably exercises the ACPI CPU hotplug MMIO access that participates in the deadlock. Once the guest had finished booting, the problem became much harder to trigger (as there is no MMIO operation without explicit CPU plugging). # Solution Keep the existing two-phase behavior so all vCPUs are still signalled before the wait phase, but narrow the lifetime of the `vcpu_states` mutex. Reacquire it only long enough to access one `VcpuState` at a time in each phase. That preserves the original pause semantics and the fast signal-all / wait-all structure, while removing the lock inversion with the ACPI CPU hotplug MMIO path. This also remains safe if a vCPU is hot-removed while pause is in progress. Hot-remove does not shrink `vcpu_states`; it stops the thread and clears the `VcpuState` handle in place. `signal_vcpus()` can therefore snapshot the vector length up front, and if a vCPU disappears between the signal and wait phases, `wait_until_signal_acknowledged()` will observe `handle.is_none()` and return successfully. The interruption handshake itself lives in atomics inside each `VcpuState`. The outer mutex is only needed to reach the state objects, not to keep the acknowledgement protocol correct. Dropping the mutex between iterations therefore does not weaken the pause protocol, but it does allow MMIO handlers and other `vcpu_states` users to make forward progress while the VMM waits for the kick to be observed. On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

rbradford

🚀

This once and for all finally resolves the deadlock that was only partially addressed in cloud-hypervisor#7990 (dfe78a3 and following) and cloud-hypervisor#8092 (2482f56). The first one laid the important basic plumbing but actually just nararrowd down the deadlock. The second PR tightened the new race window further - but never resolved it. This commit removes the Mutex around `Vec<VcpuState>`, which was recently introduced in cloud-hypervisor#7990 (dfe78a3). The deadlock was: - CpuManager::pause() -> VcpuState::wait_until_signal_acknowleded() - Mutex is held - vCPU could not do MMIO on AcpiCpuHotplugController and therefore never acknowledge the signal. The Mutex around `Vec<VcpuState>` was introduced because: - CpuManager sometimes needs access to all vCPU states (the whole vec) or individual ones - AcpiPciHotplugController needs access to individual vCPU states The mutating operations caused the Mutex in the first place where: - setting `handle: Option<std::thread::Handle>` - from `None` -> `Some` in `CpuManager::start_vcpu()` - from `Some` -> `None`: - in `AcpiCpuHotplugController::remove_vcpu()` - in `CpuManager::shutdown()` - the two bools `inserting` and `removing`, which were guest-visible and used to check if guest ACKed CPU hotplug events I resolved that by moving the Mutex around the Option to the thread handle and switching the two raw bools to `AtomicBool`. After comprehensive analysis, I am confident this is a proper and clean solution that doesn't introduce any new (critical) races. On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

TL;DR: fix deadlock CpuManager::pause() signals all vCPUs and then waits until each run loop acknowledges the signal through a shared bool. That can deadlock against ACPI CPU hotplug MMIO: VMM thread vCPU thread ---------- ---------- pause() signal_vcpus() read-lock selected vCPU state wait for acknowledgement -------> exits KVM_RUN on signal enters ACPI CPU hotplug MMIO write-locks same state [blocks] keeps waiting <------ cannot finish MMIO or ack signal This was the case before v51 but also the partial fixes introduced in v51 [0] [1]. Since the previous commit introduces per-vcpu-locking, we now can refactor wait_until_signal_acknowledged() in a way that it locks and releases the lock again on every loop iteration, allowing forward progress on the MMIO path. This change is needed as AcpiCpuHotplugManager::remove_vcpu() needs mutable access to the underlying structure. Otherwise, the existing read lock would be sufficient. [0] cloud-hypervisor#7990 [1] cloud-hypervisor#8092 On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>

phip1611 requested a review from a team as a code owner April 24, 2026 09:18

phip1611 mentioned this pull request Apr 24, 2026

vmm: avoid pause deadlock on CPU hotplug MMIO cyberus-technology/cloud-hypervisor#150

Closed

1 task

phip1611 force-pushed the upstream-fix-cpu-deadlock branch from 05f5caa to 2482f56 Compare April 24, 2026 09:22

rbradford approved these changes Apr 24, 2026

View reviewed changes

rbradford enabled auto-merge April 24, 2026 09:38

rbradford added this pull request to the merge queue Apr 24, 2026

Merged via the queue into cloud-hypervisor:main with commit 6d01695 Apr 24, 2026
38 checks passed

phip1611 deleted the upstream-fix-cpu-deadlock branch April 24, 2026 14:24

likebreath added the bug-fix Bug fix to include in release notes label Apr 24, 2026

rbradford added this to Cloud Hypervisor Roadmap Apr 25, 2026

rbradford moved this to ✅ Done in Cloud Hypervisor Roadmap Apr 25, 2026

phip1611 mentioned this pull request Apr 28, 2026

vmm: cpu: remove Mutex around Vec<VcpuState> to prevent deadlock cyberus-technology/cloud-hypervisor#152

Closed

phip1611 mentioned this pull request May 13, 2026

Fix deadlock regarding VcpuState handling (finally) #8210

Closed

1 task

Navigation Menu

Uh oh!

vmm: avoid pause deadlock on CPU hotplug MMIO - #8092

vmm: avoid pause deadlock on CPU hotplug MMIO#8092
rbradford merged 1 commit into
cloud-hypervisor:mainfrom
phip1611:upstream-fix-cpu-deadlock

phip1611 commented Apr 24, 2026

Uh oh!

rbradford left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

phip1611 commented Apr 24, 2026

TL;DR

Problem

Reproducer

Solution

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants