Thanks to visit codestin.com
Credit goes to github.com

Skip to content

vmm: defer PCI device visibility to fix hotplug race condition - #8369

Merged
rbradford merged 2 commits into
cloud-hypervisor:mainfrom
elainewu1222:avoid-bar-reprogramming-race
Jun 12, 2026
Merged

vmm: defer PCI device visibility to fix hotplug race condition#8369
rbradford merged 2 commits into
cloud-hypervisor:mainfrom
elainewu1222:avoid-bar-reprogramming-race

Conversation

@elainewu1222

@elainewu1222 elainewu1222 commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Fix a race condition during PCI device hotplug where the guest can discover a partially-initialized device, causing BAR reprogramming failures.

Problem

When PCI devices are hotplugged, the Linux guest kernel scans all PCI slots upon each hotplug notification — not just the slot that triggered it. A device that has been added to the PCI bus but whose device_tree entry and ioeventfds have not yet been registered can be discovered and probed by the guest prematurely.

The previous add_pci_device() made the device visible on the PCI bus (via pci_bus.add_device()) before completing device-specific setup. The following timeline illustrates the race:

  // disk2 hotplug — normal
  
  Jun 04 04:38:35 h1027 containerd[2140]: {"msg":""cloud-hypervisor: 10.382729s:  INFO:vmm/src/api/mod.rs:475 -- API request event: AddDisk DiskConfig { path: Some(\"/dev/rbd3\"), ... }"", ...}
  Jun 04 04:38:35 h1027 containerd[2140]: {"msg":""cloud-hypervisor: 10.383702s:  INFO:vmm/src/device_manager.rs:2379 -- Using asynchronous RAW disk file (io_uring)"", ...}
  Jun 04 04:38:35 h1027 containerd[2140]: {"msg":""cloud-hypervisor: 10.387161s:  INFO:pci/src/configuration.rs:977 -- Detected BAR reprogramming: (BAR 4) 0xe7f00000->0xc0000000"", ...}
  Jun 04 04:38:35 h1027 containerd[2140]: {"msg":""cloud-hypervisor: 10.391038s:  INFO:virtio-devices/src/transport/pci_device.rs:1222 -- _virtio-pci-_disk2: Needs activation; writing to activate event fd"", ...}

  // disk3 hotplug — while vcpu3 device scan is still in progress

  Jun 04 04:38:35 h1027 containerd[2140]: {"msg":""cloud-hypervisor: 10.399767s:  INFO:vmm/src/api/mod.rs:475 -- API request event: AddDisk DiskConfig { path: Some(\"/dev/rbd6\"), ... }"", ...}
  Jun 04 04:38:35 h1027 containerd[2140]: {"msg":""cloud-hypervisor: 10.400550s:  INFO:vmm/src/device_manager.rs:2379 -- Using asynchronous RAW disk file (io_uring)"", ...}

  // vcpu3's scan reaches disk3's slot — on bus but device_tree not yet registered

  Jun 04 04:38:35 h1027 containerd[2140]: {"msg":""cloud-hypervisor: 10.403578s:  INFO:pci/src/configuration.rs:977 -- Detected BAR reprogramming: (BAR 4) 0xe7f00000->0xc0080000"", ...}
  Jun 04 04:38:35 h1027 containerd[2140]: {"msg":""cloud-hypervisor: 10.403598s:  ERROR:pci/src/bus.rs:270 -- Failed moving device BAR: Couldn't find device _virtio-pci-_disk3 from device tree: 0xe7f00000->0xc0080000(0x80000)"", ...}

// subsequent fallout

  Jun 04 04:38:35 h1027 containerd[2140]: {"msg":""cloud-hypervisor: 10.417011s:  ERROR:virtio-devices/src/transport/pci_device.rs:1196 -- Unexpected write to notification BAR: offset = 0x6000"", ...}

The root cause is that pci_bus.add_device() made disk3 discoverable on the bus before its device_tree entry was registered, so when vcpu3 scanned all slots and attempted BAR reprogramming on disk3, the VMM could not find the device in the device tree to complete the BAR move.

Solution

Commit 1 refactors PciBus::register_mapping() into DeviceManager::register_bar_mapping(), since it operates on mmio_bus/io_bus which are external to PciBus. The corresponding error variants are moved from PciRootError to DeviceManagerError.

Commit 2 splits add_pci_device() into two phases:

  • allocate_pci_bars() — reserves BAR address space only, without making the device visible to the guest.
  • commit_pci_device() — pushes the device into bus_devices, registers BAR mappings on the VMM buses, and finally adds the device to the PCI bus. At this point, the device is discoverable by guest.

All six callers now follow the pattern:

allocate_pci_bars() → device-specific setup → commit_pci_device()

This ensures ioeventfds, device_tree entries, and MMIO mappings are fully in place before the device becomes discoverable by the guest, closing the race window.

@elainewu1222
elainewu1222 requested a review from a team as a code owner June 11, 2026 08:54
@phip1611

phip1611 commented Jun 11, 2026

Copy link
Copy Markdown
Member

I think this partially goes into the direction we discussed in #8255 - nice

@elainewu1222

elainewu1222 commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

I think this partially goes into the direction we discussed in #8255 - nice

@phip1611 I noticed the lack of proper rollback on error paths as well. However, my current commits only focus on fixing the race condition rather than tackling the broader rollback problem.

Happy to hear any suggestions:)

rbradford
rbradford approved these changes Jun 11, 2026
View reviewed changes

@rbradford rbradford left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really happy. Thanks! Although it needs a rebase.

phip1611
phip1611 approved these changes Jun 11, 2026
View reviewed changes

@phip1611 phip1611 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@elainewu1222
vmm: move BAR mapping registration from PciBus to DeviceManager
31aed9d 
`PciBus::register_mapping()` operates on `mmio_bus` and `io_bus`
which are passed in as external parameters and have nothing to do
with PciBus internal state. Move this logic into
`DeviceManager::register_bar_mapping()` where it belongs, and move
the `PioInsert`/`MmioInsert` error variants from `PciRootError` to
`DeviceManagerError` accordingly.

Signed-off-by: wuxinyue <[email protected]>
Assisted-by: Claude:Opus-4.6
@elainewu1222
vmm: defer PCI device visibility to fix hotplug race condition
159090e 
Split `add_pci_device()` into two phases: `allocate_pci_bars()` which
only allocates BAR address space, and `commit_pci_device()` which
makes the device visible to the guest on the PCI bus.

All callers now follow the pattern: allocate BARs → perform device-
specific setup (ioeventfd, device_tree, mmio mapping) → commit device.
This eliminates a race window where the guest could discover a
partially-initialized device via `acpiphp_check_bridge()` during rapid
sequential hotplug, causing BAR reprogramming to fail because
ioeventfds and device_tree entries were not yet in place.

Signed-off-by: wuxinyue <[email protected]>
Assisted-by: Claude:Opus-4.6
@elainewu1222
elainewu1222 force-pushed the avoid-bar-reprogramming-race branch from b3c08f6 to 159090e Compare June 12, 2026 02:23
@elainewu1222

elainewu1222 commented Jun 12, 2026

Copy link
Copy Markdown
Contributor Author

@rbradford rebase done 🙌

@rbradford
rbradford added this pull request to the merge queue Jun 12, 2026
Merged via the queue into cloud-hypervisor:main with commit 75b0fe5 Jun 12, 2026
41 checks passed
@rbradford rbradford added the bug-fix Bug fix to include in release notes label Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rbradford rbradford rbradford approved these changes
@phip1611 phip1611 phip1611 approved these changes
@sboeuf sboeuf sboeuf approved these changes

Assignees

No one assigned

Labels

bug-fix Bug fix to include in release notes

Projects

Cloud Hypervisor Roadmap
Status: ✅ Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@elainewu1222 @phip1611 @rbradford @sboeuf