Conversation
|
Please fix clippy. I can recommend to use the following command:
|
likebreath
left a comment
There was a problem hiding this comment.
Thank you for the contribution. Some quick comments:
- Please update documentation for the new UI
- Please update the OpenAPI spec file accordingly: https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/vmm/src/api/openapi/cloud-hypervisor.yaml
I am converting the PR to a draft before clippy issues are addressed. @saravan2 I know your work on Generic Initiator has some overlaps with this work. Can you please help review the PR when it is ready? Thank you.
01cd731 to
56ac625
Compare
56ac625 to
a5dab6e
Compare
a5dab6e to
3c32aff
Compare
|
I think I'm now ready for review @likebreath and @saravan2 . Thanks in advance! |
phip1611
left a comment
There was a problem hiding this comment.
Awesome work!
Please address the remarks. Further, you could mention that we are already deployed this to a few dozens of nodes with OpenStack and libvirt, and it is working like a charm :)
I'm refraining from an approval for now as I'm biased (Pascal is my colleague)
There was a problem hiding this comment.
Reviewed and requested changes.
I recommend developing an integration test to validate this feature.
The integration test infrastructure already has everything needed to support a minimal test to boot a guest VM with a (disk) device arg :addr=0A.0. The test runner is capable of running lscpi command inside the guest to confirm the device appears on intended address 0A.0
|
Regarding the open question in the PR description: Both Cloud-Hypervisor and QEMU adhere to the standard PCIe addressing format (Segment:Bus:Device.Function). However, their architectural implementations of the PCI hierarchy differ significantly The MCFG (Memory Mapped Configuration Space Base Address Description Table) acts as the guest OS's source of truth for PCI enumeration. cloud-hypervisor/vmm/src/acpi.rs Lines 274 to 275 in caa362c Since the guest OS relies entirely on the MCFG table to discover the bus range, it will never scan or identify a device on Bus 1, regardless of our internal VMM data structures. |
saravan2
left a comment
There was a problem hiding this comment.
cloud-hypervisor.yaml changes should be part of a dedicated commit.
|
I hope to have addressed all remarks accordingly. I triple checked all comments/commit messages for typos or errors and all should be fine now. |
|
@rbradford can we merge this? I think Pascal addressed all concerns |
0d7bb9f to
8d7935d
Compare
This commit refactors the PCI bus struct. It has two major focuses. First, we change the type of `device_ids` in `PciBus` to an array. A fixed-size array better reflects real PCI bus constraints, especially its limited number of PCI devices. Moreover, it can't be grown accidentally. The second focus is changing the type of the key of `devices` in `PciBus` to `u8`, since device IDs are not allowed to exceed 31. We furthermore replace magic numbers with constants and make them publicly available so we can use them in a follow-up change when parsing user input. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
Allocating a device ID is crucial for assigning a specific ID to a device. We need this to implement configurable PCI BDF. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
Next to tests for `allocate_device_bdf`, we introduce a new constructor `new_without_address_manager`, only available in the test build. As there is no way to instantiate an `AddressManager` in the tests, we use this constructor to work around this. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
To pass a PCI device’s device ID when requesting a PCI BDF, we add a new member to each device config that is added as a PCI device. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
The `addr` option is used by Qemu to specify the device and function ID part of a BDF for a PCI device. We use the same format, but implement only single-function devices for now. This means we set the function part to zero and allow specifying only the device part of the BDF with the `addr` option. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
Config structs that allow configuring the devices' PCI bus device ID via `device_bdf` should also allow validating that ID. We need this option to ensure that invalid device IDs received via an API call result in an error as soon as possible. In this case, this would be after deserialization. On this code path, validation via `parse` is skipped and must be invoked by calling `validate`. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
We pass the device ID from the BDF to the allocation routine, where it is used. This enables future support for specifying the segment and bus in the SBDF. We can then extend the design to provide a complete BDF structure and pass its parts at the appropriate levels. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
We add documentation for the `bdf_device` member to all documented structs. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
Update the API doc for all PCI devices that use the `device_bdf` member added previously. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
Note that the HTTP API doesn't support hex literals, as this would require a custom de-/serializer for the `bdf_device` field or using an additional helper crate. We refrain from both as the JSON grammar[0] does not support hex numbers by default. [0] https://www.crockford.com/mckeeman.html Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
We use a VecDeque to implicitly sort devices. Devices with a fixed BDF in their config are added to the front. Devices without a given BDF are added to the back. Processing the `VecDeque` from first to last ensures that no clashes occur when assigning BDFs. Otherwise, we could assign a required BDF to a device that doesn't need it. `add_virtio_device` is an associated function because we need to call it in the same place where we cannot create a mutable borrow of the respective `DeviceManager` instance. This is the case, for example, in `make_virtio_balloon_devices`. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
If we want to test for error cases, it can be useful to inspect the `stderr` of a `Command` to analyze the errors. For example, this allows us to ensure that a `Command` returns an `IoError` by parsing the error trace, if an `IoError` is expected. This commit prepares the implementation of negative integration tests for the configurable BDFs. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
This commit adds an integration test to verify that the guest sees the correct BDF. Moreover, we check that we can allocate a random free BDF and that freeing BDFs works. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
This integration test verifies that a BDF cannot be allocated twice. Moreover, we check that the returned error matches our expectations. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
This integration test verifies two conditions: first, that a requested BDF device ID is valid (in the range of 0-31); second, that requesting a BDF with a function ID returns an error. For both, we confirm that the correct error log appears. Signed-off-by: Pascal Scholz <[email protected]> On-behalf-of: SAP [email protected]
8d7935d to
5fcd293
Compare
| serial=<serial_number>,backing_files=on|off,sparse=on|off,\ | ||
| image_type=<raw,qcow2,vhd,vhdx>,lock_granularity=byte-range|full"; | ||
| image_type=<raw,qcow2,vhd,vhdx>,lock_granularity=byte-range|full,\ | ||
| addr=<DD.F>"; |
There was a problem hiding this comment.
I missed this before - we always keep CLI and JSON API/struct names the same. For consistency with the pci_segment this should be called pci_device_id in the CLI and struct. And it should just be an int. Why specify the function?
There was a problem hiding this comment.
Why specify the function?
for consistency with QEMU, I think. But on the other side, CHV will not support multifunction devices in the foreseeable future I think.
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod unit_tests { |
There was a problem hiding this comment.
I don't see a test for if the user specifies the same pci_device_id for two devices. Do you need to add valodation logic for that?
|
I did a rebase on the config deduplication (and cleaned up some commit messages along the way) https://github.com/rbradford/cloud-hypervisor/commits/202604/custom-pci-device-id/ As you can see if you do some simple refactoring first the desired change becomes much simpler. However whilst doing this I realised that the current implementation has a major flaw. Although it uses a clever trick with a double ended vector for virtio devices to make sure that those that have specific PCI device ID get picked first it this doesn't interleave with the non-virtio devices. e.g. if you have a VFIO device marked with a reserved device ID of 3 and a bunch of virtio devices then the allocation will fail. I'm thinking about other solutions. |
|
First and foremost: Thanks @rbradford for taking the time to refactor and fixing the possible clash. I think that your solution in #7965 is the way forward and we should build on this PR as you already applied all relevant feature changes from my PR (#7631) on top of your refactoring from #7962 there. I also discussed this with @phip1611 and we agreed to close this PR in favor of your work. |
|
See #7631 (comment) |
Configurable PCI BDFs
Motivation
Currently, the PCI address handout in Cloud Hypervisor depends on the order in which devices are added to a VM. This is problematic and can cause VMs to break in two scenarios. First, if we decide to alter the order in which we process devices in the initialization routine, we may end up with two versions that prioritize different device classes. Different prioritization in this case results in different PCI addresses across two Cloud Hypervisor versions, as per the current implementation. Second, management software like libvirt might rely on the fact that it can decide what PCI address a device uses. For example, we observed cases where we hotplugged a disk after VM creation. Disks are prioritized higher than network interfaces in the PCI address handout. After a live migration, for example, this disk was added before the network interface, as network interfaces require a dedicated API call to be added to a VM (configuration). To prevent such a situation, we need a way to tell Cloud Hypervisor which PCI addresses to assign to a specific device. How we can solve this problem is described in the following sections.
Background
Configurable PCI BDF (Bus, Device, Function) allows for choosing the address that a guest can see a PCI device. Qemu, for example, supports adding devices to a VM through the
devicecommand. With the help of theaddroption for thedevicecommand, one can define the device and function part of a device's BDF in the formaddr=DD.F, whereDdenotes a device ID in the range of [0..32] andFa function ID in a range of [0..7]. Qemu denotes the bus number with the optionbus, where a valid expression isbus=pci.2to denote the second PCI bus, for example. [0]With this pull requests, we try to mimic the command-line interface of Qemu and introduce a new command line parameter to CHV, similarly achieve configurable BDFs.
An example of a BDF for Bus=1, Device=3, and Function=5 is
1:03.5. [1]Current status
CHV allows defining multiple PCI domains via the
pci_segmentsoption. For all PCI segments, CHV creates a dedicated host bridge. Via the option argumentpci_segment, one can define to which domain a device is added.Other than that, CHV currently has no support to influence what exact BDF is assigned to a device, other than the order in which devices are specified in the VM configuration. This is done by calling the
next_device_bdffunction of thestruct
PciSegmentfor each PCI device to create.next_device_bdfreturns the a BDF with the following free device ID and a fixed function ID of 0.PciSegmentqueries the underlyingPciBusto obtain the following device ID, which we also must modify.Currently, there is no support for multi-function devices in CHV. Because of this, we ignore the
Functionpart of BDFs for now.Proposed changes
We split the configurable BDF feature into separate PRs with the goals defined in The following subsections.
The BAR address allocation is able to handle the changes listed below. So I don't expect the need for changes there.
1. Definition of Device IDs
To define a device ID, we first must implement some notion for specifying them via the API or command line. For this, we introduce the
addroption argument, similar to Qemu. The expected format isaddr=DDwithDDbeing in the rangeof [0..32]. This argument is stored in a device's configuration and must be propagated to
next_device_bdf. This function will be replaced byspecific_device_bdfto allow for specifying a concrete device identifier. The steps for implementing configurable device IDs are as follows:allocate_device_idto structPciBusto reserve a device ID, depending on the argument given. If no device ID is specified, use the current mechanism to obtain an ID. This replacesnext_device_id.allocate_device_bdfto structPciSegmentto allocate a BDF depending on function parameters. If no device parameter is specified, fall back to the algorithm currently used bynext_device_bdf. Replacesnext_device_bdf.addrparameter to all forms of VirtIo device configurationsand adjust parsers
allocate_device_idinPciBusallocate_device_bdfinPciSegmentOpen Questions
For our work it was enough to also ignore the
Buspart, as we mostly have only one bus on x86. As Qemu also uses an additionalbusoption for this part of a BDF, we should be fine. We did some brief testing with multiple PCI segments though and found no issue in BDF handout, as long as only one bus exists in the segment. Currently, the bus ID is fixed to 0 (see patches).References
[0] https://qemu-project.gitlab.io/qemu/system/device-emulation.html#device-buses
[1] https://wiki.xenproject.org/wiki/Bus:Device.Function_(BDF)_Notation
I'm looking forward for your input!