Thanks to visit codestin.com
Credit goes to github.com

Skip to content

vmm: Add support for explicit PCI device IDs - #7965

Merged
rbradford merged 18 commits into
cloud-hypervisor:mainfrom
rbradford:202604/custom-pci-device-id
Apr 17, 2026
Merged

rbradford merged 18 commits into
cloud-hypervisor:mainfrom
rbradford:202604/custom-pci-device-id

Conversation

@rbradford

@rbradford rbradford commented Apr 4, 2026

Copy link
Copy Markdown
Member

Based on the work in #7631 but adjusted to correctly handle clashes between
device types (not just within virtio devices). Building on the refactoring from
#7962

  • pci: Refactor bus.rs to better fit a PCI bus's semantics
  • vmm: Allow for device ID allocation on a segment
  • vmm: Fix segment log message formatting
  • vmm: Add tests for allocate_device_id in PciSegment
  • pci: Add support for reserving but not allocating slots
  • vmm: Use the PciBus::allocate_device_id on restore path
  • vmm: config: Add pci_device_id to PciDeviceCommonConfig
  • vmm: openapi: Add pci_device_id to the required device entries
  • vmm: config: Add pci_device_id to SYNTAX for supported devices
  • docs: Update the relevant documentation
  • vmm: Validate PCI device ID
  • vmm: Propagate PCI device ID from the config
  • vmm: device_manager: Reserve explicitly used PCI device IDs
  • tests: Return stderr when executing commands
  • tests: Add an integration test to verify PCI device allocations
  • tests: Add an integration test to check duplicate PCI device IDs
  • tests: Add an integration test for PCI device ID allocation errors

@phip1611

phip1611 commented Apr 8, 2026

Copy link
Copy Markdown
Member

@scholzp please run our internal test suite (just the BDF test cases without live migration) against this. Does it work?

Comment thread vmm/src/vm_config.rs
@rbradford
rbradford force-pushed the 202604/custom-pci-device-id branch from cc0a512 to 4d713f9 Compare April 14, 2026 22:50
@rbradford
rbradford marked this pull request as ready for review April 14, 2026 22:51
@rbradford
rbradford requested a review from a team as a code owner April 14, 2026 22:51
@rbradford

Copy link
Copy Markdown
Member Author

@phip1611 @scholzp I've rebased

@phip1611 phip1611 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lovely! Just a few nits.

PS: @scholzp will finish his final review also today before ~2pm - looking good!

To summarize for my own understanding:

Phase 1reserve_explicit_device_ids() is called once before any device
creation. It iterates virtio_devices, config.devices, and
config.user_devices, and for every entry with pci_device_id: Some(id) calls
PciBus::reserve_device_id(id): Free → Reserved. This blocks anonymous
allocation from stealing explicitly requested slots without yet committing them.

Phase 2 — per-device, when actually added to the bus, pci_resources() calls
PciSegment::allocate_device_id(pci_device_id):

  • Some(id): Reserved → Allocated (also accepts Free, rejects Allocated)
  • None: scans for first Free slot, skipping Reserved and Allocated

Both virtio-net and vhost-user reach this via
add_virtio_pci_device(..., pci_device_id)pci_resources(..., pci_device_id);
VFIO/vfio-user call pci_resources directly.

State machine:

    Free ──reserve──► Reserved ──allocate(Some)──► Allocated
    Free ─────────────────────────allocate(None)──► Allocated
    Allocated/Reserved ──────put──────────────────► Free

Duplicate explicit IDs fail at phase 1; out-of-range and reserved IDs (e.g. the root device at slot 0) fail at config validation before either phase runs.

Comment thread pci/src/bus.rs
Comment thread vmm/src/vm_config.rs
Comment thread vmm/src/config.rs Outdated
Comment thread vmm/src/config.rs Outdated

@scholzp scholzp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found a few typos, one of which I have authored. Otherwise I think all looks good.

Thanks again for investing your time! :)

Comment thread cloud-hypervisor/tests/integration.rs
Comment thread cloud-hypervisor/tests/integration.rs
Comment thread pci/src/bus.rs Outdated
@rbradford
rbradford force-pushed the 202604/custom-pci-device-id branch 4 times, most recently from c847aaa to 0394a55 Compare April 16, 2026 20:15
scholzp and others added 16 commits April 16, 2026 22:21
This commit refactors the PCI bus struct. It has two major focuses.
First, we change the type of `device_ids` in `PciBus` to an array. A
fixed-size array better reflects real PCI bus constraints, especially
its limited number of PCI devices. Moreover, it can't be grown
accidentally.

The second focus is changing the type of the key of `devices` in
`PciBus` to `u8`, since device IDs are not allowed to exceed 31. We
furthermore replace magic numbers with constants and make them publicly
available so we can use them in a follow-up change when parsing user
input.

Signed-off-by: Pascal Scholz <[email protected]>
On-behalf-of: SAP [email protected]
Allocating a device ID is crucial for assigning a specific ID to a
device. We need this to implement configurable PCI device ID.

Signed-off-by: Pascal Scholz <[email protected]>
On-behalf-of: SAP [email protected]
Signed-off-by: Rob Bradford <[email protected]>
Next to tests for `allocate_device_id`, we introduce a new constructor
`new_without_address_manager`, only available in the test build. As
there is no way to instantiate an `AddressManager` in the tests, we use
this constructor to work around this.

Signed-off-by: Pascal Scholz <[email protected]>
On-behalf-of: SAP [email protected]
Signed-off-by: Rob Bradford <[email protected]>
This can be used in a two pass approach where all configs that can hold
PCI devices are evaluated to reserve any specific PCI device IDs they
may need. Those device IDs will later be allocated when the devices are
added to the bus. The tri-state Free, Reserved, Allocated also catches
the problem of hotplugging a device with a specific, already used,
device ID.

Signed-off-by: Rob Bradford <[email protected]>
This adds it to all device types that use the common PCI device
configuration.

Signed-off-by: Rob Bradford <[email protected]>
Also add pci_segment that was missing from vfio-user devices.

Signed-off-by: Rob Bradford <[email protected]>
For those devices types that have the the ability to support specifying
the PCI device ID add it to their help syntax.

Signed-off-by: Rob Bradford <[email protected]>
Validate the PCI device ID are within range and not using the reserved
value. We need this option to ensure that invalid device IDs received
via an API call result in an error as soon as possible. In this case,
this would be after deserialization. On this code path, validation via
`parse` is skipped and must be invoked by calling `validate`.

Signed-off-by: Pascal Scholz <[email protected]>
On-behalf-of: SAP [email protected]
Signed-off-by: Rob Bradford <[email protected]>
We pass the device ID from the config to the allocation routine, where
it is then used as the preferred device ID alongside the existing PCI
segment ID.

Signed-off-by: Pascal Scholz <[email protected]>
On-behalf-of: SAP [email protected]
Signed-off-by: Rob Bradford <[email protected]>
Use two passes to first reserve PCI device IDs and then allocate them
when adding the devices to the bus. This prevents a situation where an
anonymous PCI device allocation clashes with an explicitly allocated PCI
device ID.

Signed-off-by: Rob Bradford <[email protected]>
Some of the documentation references PCI segment ID. For those documents
add a mention of the new PCI device ID.

Signed-off-by: Rob Bradford <[email protected]>
If we want to test for error cases, it can be useful to inspect the
`stderr` of a `Command` to analyze the errors. For example, this allows
us to ensure that a `Command` returns an `IoError` by parsing the
error trace, if an `IoError` is expected.

This commit prepares the implementation of negative integration tests
for the configurable BDFs.

Signed-off-by: Pascal Scholz <[email protected]>
On-behalf-of: SAP [email protected]
This commit adds an integration test to verify that the guest sees the
correct BDF. Moreover, we check that we can allocate a random free BDF
and that freeing BDFs works.

Signed-off-by: Pascal Scholz <[email protected]>
On-behalf-of: SAP [email protected]
Signed-off-by: Rob Bradford <[email protected]>
This integration test verifies that the same device ID cannot be
allocated twice. Moreover, we check that the returned error matches our
expectations.

Signed-off-by: Pascal Scholz <[email protected]>
On-behalf-of: SAP [email protected]
Signed-off-by: Rob Bradford <[email protected]>
Adds a test that checks the correct error is returned on allocation of
an invalid device ID (one that is not in the range 0-31) and when trying
to allocate a reserved ID (such as that of the root bridge).

Signed-off-by: Pascal Scholz <[email protected]>
On-behalf-of: SAP [email protected]
Signed-off-by: Rob Bradford <[email protected]>
Our bus slots are now Reserved/Allocated/Free so change the method to
free it to free_device_id() and update error.

Also update to take u8 to match the other methods.

Signed-off-by: Rob Bradford <[email protected]>
@rbradford
rbradford force-pushed the 202604/custom-pci-device-id branch from 0394a55 to cdca8a7 Compare April 16, 2026 21:28
This test was taking > 300s due to SSH backoffs.

Signed-off-by: Rob Bradford <[email protected]>
@rbradford
rbradford added this pull request to the merge queue Apr 17, 2026
Merged via the queue into cloud-hypervisor:main with commit 67cf328 Apr 17, 2026
38 checks passed
@rbradford rbradford moved this to ✅ Done in Cloud Hypervisor Roadmap Apr 25, 2026
phip1611 added a commit to phip1611/cloud-hypervisor that referenced this pull request May 7, 2026
This was missing in [0] but is required for proper explicit PCI BDF
management, e.g., when a VM is created via libvirt and each device has
an explicit BDF.

[0] cloud-hypervisor#7965

On-behalf-of: SAP [email protected]
Signed-off-by: Philipp Schuster <[email protected]>
phip1611 added a commit to phip1611/cloud-hypervisor that referenced this pull request May 7, 2026
This was missing in [0] but is required for proper explicit PCI BDF
management, e.g., when a VM is created via libvirt and each device has
an explicit BDF.

[0] cloud-hypervisor#7965

On-behalf-of: SAP [email protected]
Signed-off-by: Philipp Schuster <[email protected]>
phip1611 added a commit to phip1611/cloud-hypervisor that referenced this pull request May 7, 2026
This was missing in [0] but is required for proper explicit PCI BDF
management, e.g., when a VM is created via libvirt and each device has
an explicit BDF.

[0] cloud-hypervisor#7965

On-behalf-of: SAP [email protected]
Signed-off-by: Philipp Schuster <[email protected]>
phip1611 added a commit to phip1611/cloud-hypervisor that referenced this pull request May 8, 2026
This was missing in [0] but is required for proper explicit PCI BDF
management, e.g., when a VM is created via libvirt and each device has
an explicit BDF.

[0] cloud-hypervisor#7965

On-behalf-of: SAP [email protected]
Signed-off-by: Philipp Schuster <[email protected]>
phip1611 added a commit to phip1611/cloud-hypervisor that referenced this pull request May 8, 2026
This was missing in [0] but is required for proper explicit PCI BDF
management, e.g., when a VM is created via libvirt and each device has
an explicit BDF.

[0] cloud-hypervisor#7965

On-behalf-of: SAP [email protected]
Signed-off-by: Philipp Schuster <[email protected]>
phip1611 added a commit to phip1611/cloud-hypervisor that referenced this pull request May 8, 2026
This was missing in [0] but is required for proper explicit PCI BDF
management, e.g., when a VM is created via libvirt and each device has
an explicit BDF.

[0] cloud-hypervisor#7965

On-behalf-of: SAP [email protected]
Signed-off-by: Philipp Schuster <[email protected]>
phip1611 added a commit to phip1611/cloud-hypervisor that referenced this pull request May 8, 2026
This was missing in [0] but is required for proper explicit PCI BDF
management, e.g., when a VM is created via libvirt and each device has
an explicit BDF.

[0] cloud-hypervisor#7965

On-behalf-of: SAP [email protected]
Signed-off-by: Philipp Schuster <[email protected]>
russell-islam pushed a commit to russell-islam/cloud-hypervisor that referenced this pull request May 9, 2026
This was missing in [0] but is required for proper explicit PCI BDF
management, e.g., when a VM is created via libvirt and each device has
an explicit BDF.

[0] cloud-hypervisor#7965

On-behalf-of: SAP [email protected]
Signed-off-by: Philipp Schuster <[email protected]>
@rbradford
rbradford deleted the 202604/custom-pci-device-id branch June 12, 2026 10:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

4 participants