pci: synchronize VfioMsix::cap and VfioMsix::bar - #7993
Conversation
Currently, when snapshoting a running VFIO device with MSI-X enabled,
we get a snapshot where `msix_config.state.enabled` is not consistent
with `msix_state.cap.msg_ctl`,
```jsonc
{
"snapshots": {
"vfio_common": {
"snapshots": {
"msix_config": {
"snapshots": {},
"state": {
"enabled": true
// ...
}
},
// ..
},
"state": {
"msix_state": {
"cap": {
"msg_ctl": 3,
"table": 1,
"pba": 2049
},
// ...
}
// ...
}
}
},
// ...
}
```
The root cause is, after a `MsixCap` is parsed from the device PCI
config space and propagated to a corresponding `MsixConfig`,
`MsixCap::msg_ctl` is never get updated at runtime, only
`MsixConfig::msg_ctl` is updated.
This commit makes `VfioMsix::update` update both `VfioMsix::bar` (of
type `MsixConfig`) and `VfioMsix::cap` (of type `MsixCap`).
Signed-off-by: Changyuan Lyu <[email protected]>
likebreath
left a comment
There was a problem hiding this comment.
@Lencerf Thanks for the contribution. The fix looks good.
Likely you know, the snapshot/restore for VFIO devices isn't fully functional, since it currently only covers VMM-side state. Restoring the physical device state still requires additional work, specifically implementing parts of the VFIO live migration V2 protocol. Is this something you're planning to work on?
I mention this because @saravan2 will be starting work on VFIO live migration support soon, so there may be a good opportunity to collaborate.
|
Unrelated flaky CI failure: https://github.com/cloud-hypervisor/cloud-hypervisor/actions/runs/24269673912/job/70871907846
|
Hi @likebreath , instead of restoring the physical device state, we (@posk-io and I) are more interested in keeping the device running during snapshot and restore. To give an example, say we have a physical GPU assigned to a VM and is running training workloads, and we want to update cloud-hypervisor or KVM module due to some emergent security bug.
We will gradually upstreaming the live update support described above to cloud-hypervisor. |
|
@Lencerf I am working on adding VFIO migration v2 protocol support. Feel free to reach out if you think collaborating can speed up development. I am currently working on a PR to |
Currently, when snapshoting a running VFIO device with MSI-X enabled, we get a snapshot where
msix_config.state.enabledis not consistent withmsix_state.cap.msg_ctl,{ "snapshots": { "vfio_common": { "snapshots": { "msix_config": { "snapshots": {}, "state": { "enabled": true // ... } }, // .. }, "state": { "msix_state": { "cap": { "msg_ctl": 3, "table": 1, "pba": 2049 }, // ... } // ... } } }, // ... }The root cause is, after a
MsixCapis parsed from the device PCI config space and propagated to a correspondingMsixConfig,MsixCap::msg_ctlis never get updated at runtime, onlyMsixConfig::msg_ctlis updated.This commit makes
VfioMsix::updateupdate bothVfioMsix::bar(of typeMsixConfig) andVfioMsix::cap(of typeMsixCap).