vmm: Add support for explicit PCI device IDs - #7965
Conversation
|
@scholzp please run our internal test suite (just the BDF test cases without live migration) against this. Does it work? |
cc0a512 to
4d713f9
Compare
phip1611
left a comment
There was a problem hiding this comment.
Lovely! Just a few nits.
PS: @scholzp will finish his final review also today before ~2pm - looking good!
To summarize for my own understanding:
Phase 1 — reserve_explicit_device_ids() is called once before any device
creation. It iterates virtio_devices, config.devices, and
config.user_devices, and for every entry with pci_device_id: Some(id) calls
PciBus::reserve_device_id(id): Free → Reserved. This blocks anonymous
allocation from stealing explicitly requested slots without yet committing them.
Phase 2 — per-device, when actually added to the bus, pci_resources() calls
PciSegment::allocate_device_id(pci_device_id):
Some(id):Reserved → Allocated(also acceptsFree, rejectsAllocated)None: scans for firstFreeslot, skippingReservedandAllocated
Both virtio-net and vhost-user reach this via
add_virtio_pci_device(..., pci_device_id) → pci_resources(..., pci_device_id);
VFIO/vfio-user call pci_resources directly.
State machine:
Free ──reserve──► Reserved ──allocate(Some)──► Allocated
Free ─────────────────────────allocate(None)──► Allocated
Allocated/Reserved ──────put──────────────────► Free
Duplicate explicit IDs fail at phase 1; out-of-range and reserved IDs (e.g. the root device at slot 0) fail at config validation before either phase runs.
c847aaa to
0394a55
Compare
This commit refactors the PCI bus struct. It has two major focuses. First, we change the type of `device_ids` in `PciBus` to an array. A fixed-size array better reflects real PCI bus constraints, especially its limited number of PCI devices. Moreover, it can't be grown accidentally. The second focus is changing the type of the key of `devices` in `PciBus` to `u8`, since device IDs are not allowed to exceed 31. We furthermore replace magic numbers with constants and make them publicly available so we can use them in a follow-up change when parsing user input. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
Allocating a device ID is crucial for assigning a specific ID to a device. We need this to implement configurable PCI device ID. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected] Signed-off-by: Rob Bradford <[email protected]>
Next to tests for `allocate_device_id`, we introduce a new constructor `new_without_address_manager`, only available in the test build. As there is no way to instantiate an `AddressManager` in the tests, we use this constructor to work around this. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected] Signed-off-by: Rob Bradford <[email protected]>
Signed-off-by: Rob Bradford <[email protected]>
This can be used in a two pass approach where all configs that can hold PCI devices are evaluated to reserve any specific PCI device IDs they may need. Those device IDs will later be allocated when the devices are added to the bus. The tri-state Free, Reserved, Allocated also catches the problem of hotplugging a device with a specific, already used, device ID. Signed-off-by: Rob Bradford <[email protected]>
This adds it to all device types that use the common PCI device configuration. Signed-off-by: Rob Bradford <[email protected]>
Also add pci_segment that was missing from vfio-user devices. Signed-off-by: Rob Bradford <[email protected]>
For those devices types that have the the ability to support specifying the PCI device ID add it to their help syntax. Signed-off-by: Rob Bradford <[email protected]>
Validate the PCI device ID are within range and not using the reserved value. We need this option to ensure that invalid device IDs received via an API call result in an error as soon as possible. In this case, this would be after deserialization. On this code path, validation via `parse` is skipped and must be invoked by calling `validate`. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected] Signed-off-by: Rob Bradford <[email protected]>
We pass the device ID from the config to the allocation routine, where it is then used as the preferred device ID alongside the existing PCI segment ID. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected] Signed-off-by: Rob Bradford <[email protected]>
Use two passes to first reserve PCI device IDs and then allocate them when adding the devices to the bus. This prevents a situation where an anonymous PCI device allocation clashes with an explicitly allocated PCI device ID. Signed-off-by: Rob Bradford <[email protected]>
Some of the documentation references PCI segment ID. For those documents add a mention of the new PCI device ID. Signed-off-by: Rob Bradford <[email protected]>
If we want to test for error cases, it can be useful to inspect the `stderr` of a `Command` to analyze the errors. For example, this allows us to ensure that a `Command` returns an `IoError` by parsing the error trace, if an `IoError` is expected. This commit prepares the implementation of negative integration tests for the configurable BDFs. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
This commit adds an integration test to verify that the guest sees the correct BDF. Moreover, we check that we can allocate a random free BDF and that freeing BDFs works. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected] Signed-off-by: Rob Bradford <[email protected]>
This integration test verifies that the same device ID cannot be allocated twice. Moreover, we check that the returned error matches our expectations. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected] Signed-off-by: Rob Bradford <[email protected]>
Adds a test that checks the correct error is returned on allocation of an invalid device ID (one that is not in the range 0-31) and when trying to allocate a reserved ID (such as that of the root bridge). Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected] Signed-off-by: Rob Bradford <[email protected]>
Our bus slots are now Reserved/Allocated/Free so change the method to free it to free_device_id() and update error. Also update to take u8 to match the other methods. Signed-off-by: Rob Bradford <[email protected]>
0394a55 to
cdca8a7
Compare
This test was taking > 300s due to SSH backoffs. Signed-off-by: Rob Bradford <[email protected]>
This was missing in [0] but is required for proper explicit PCI BDF management, e.g., when a VM is created via libvirt and each device has an explicit BDF. [0] cloud-hypervisor#7965 On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>
This was missing in [0] but is required for proper explicit PCI BDF management, e.g., when a VM is created via libvirt and each device has an explicit BDF. [0] cloud-hypervisor#7965 On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>
This was missing in [0] but is required for proper explicit PCI BDF management, e.g., when a VM is created via libvirt and each device has an explicit BDF. [0] cloud-hypervisor#7965 On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>
This was missing in [0] but is required for proper explicit PCI BDF management, e.g., when a VM is created via libvirt and each device has an explicit BDF. [0] cloud-hypervisor#7965 On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>
This was missing in [0] but is required for proper explicit PCI BDF management, e.g., when a VM is created via libvirt and each device has an explicit BDF. [0] cloud-hypervisor#7965 On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>
This was missing in [0] but is required for proper explicit PCI BDF management, e.g., when a VM is created via libvirt and each device has an explicit BDF. [0] cloud-hypervisor#7965 On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>
This was missing in [0] but is required for proper explicit PCI BDF management, e.g., when a VM is created via libvirt and each device has an explicit BDF. [0] cloud-hypervisor#7965 On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>
This was missing in [0] but is required for proper explicit PCI BDF management, e.g., when a VM is created via libvirt and each device has an explicit BDF. [0] cloud-hypervisor#7965 On-behalf-of: SAP [email protected] Signed-off-by: Philipp Schuster <[email protected]>
Based on the work in #7631 but adjusted to correctly handle clashes between
device types (not just within virtio devices). Building on the refactoring from
#7962
allocate_device_idinPciSegment