pci: vfio: Implement migration v2 for snapshot and restore - #8303

Wire a migratable VFIO device's migration state to the VM lifecycle so
the device's internal state survives snapshot and restore. A device such
as a ConnectX VF bound to mlx5_vfio_pci would otherwise come back blank,
because a plain snapshot saves only the PCI configuration Cloud
Hypervisor owns, not the device's own state.

On save, pause moves the device to STOP and snapshot() drives it through
STOP_COPY to extract the opaque state blob, attached to the device
snapshot as a base64 encoded child. resume() returns it to RUNNING.

All new behavior is gated on migration_flags.is_some(), so devices
without migration support (including vfio-user) retain their previous
snapshot behavior.

If the data read fails after STOP_COPY was entered, the device is
returned to STOP before the error is bubbled, since the STOP_COPY
to STOP arc stays valid. A failed transition into STOP_COPY returns
immediately because a STOP from the resulting ERROR state cannot
help. Full recovery including device reset is deferred.

Since the non BAR write path goes directly to the VFIO device and not
the shadow, the PciConfiguration shadow can get stale. Mirror every
non BAR, non MSI config write into the shadow via write_byte /
write_word / write_reg so snapshot() can capture PCI_COMMAND. Without
this the shadow keeps the values set at device init and snapshot()
encodes PCI_COMMAND as zero.

Use the raw write_byte, write_word, and write_reg helpers rather than
PciConfiguration::write_config_register, which would otherwise drain
pending_bar_reprogram, consumed by the BAR block below, and rerun
MSI-X set_msg_ctl, already done by update_msix_capabilities.

Signed-off-by: Saravanan D <[email protected]>

When a snapshot is loaded, walk the migration v2 state machine
from VfioCommon::set_state() after interrupt state has been
restored. If the device supports migration and a blob is
present, drive RUNNING to RESUMING in a single transition and
write the blob to the data_fd. The kernel handles the
intermediate STOP arc internally. An explicit STOP dwell was
observed to make mlx5_vfio_pci re initialize SQ, CQ, and EQ
indices on top of the just loaded blob, wedging queue state on
resume. The device is left in RESUMING and resume() drives it
to RUNNING during VM resume.

set_state() also pushes PCI_COMMAND to the device via
write_config() after the blob load. Rebuilding the in memory
MSI or MSI-X structs does not touch the kernel's view of
PCI_COMMAND, so without this the VF sits at post reset defaults
with no bus master and mlx5_core ACCESS_REG times out. It
rearms VFIO_DEVICE_SET_IRQS via enable_msi or enable_msix for
the same reason, since replaying the in memory interrupt state
does not reissue the ioctl and the kernel has no eventfds for
this device until it does. Both match QEMU
vfio_pci_load_config().

In allocate_bars, skip add_pci_bar and add_pci_rom_bar on
restore. PciConfiguration::new(Some(state)) already populated
the BAR registers with used=true, so the extra call trips
BarInUse. The bars vec and mmio_regions pushes still need to
happen so the caller can wire bus mappings.

set_state() retrieves the migration blob from the snapshot
unconditionally and rejects a snapshot that carries migration
state when the device does not support migration, rather than
silently dropping the saved state. A device without migration
support and no blob, including vfio-user, still skips the load.

On any transition or write failure during restore, STOP is
attempted as best effort before bubbling the error.

Signed-off-by: Saravanan D <[email protected]>

The VFIO save and load paths drive the kernel migration state machine
through a strict transition sequence and must recover correctly when a
transition or the data transfer fails. Validating that on real hardware
needs a migratable device, which CI does not have, so the behavior would
otherwise go unverified until it breaks in the field.

Introduce a mock Vfio wrapper that records state transitions and keeps
the migration blob in memory, then test VfioCommon save and load against
it. This pins the transition ordering, the data round trip, and the
failure recovery in CI on any host, independent of hardware. The trait
defaults and the VfioMigrationState conversions are covered too, so a
non migratable device and an unknown state value stay well defined. A
restore that carries migration state onto a device without migration
support is checked to fail instead of dropping the state.

A shadow sync test guards the related save path change, confirming a non
BAR config write reaches the PciConfiguration shadow so a snapshot
captures the live value instead of the post init zero.

Signed-off-by: Saravanan D <[email protected]>

Add a Snapshot and Restore section to docs/vfio.md covering the
migration v2 requirements (Linux 5.18 kernel, variant VFIO
driver such as mlx5_vfio_pci) and the restore sequence for
devices that advertise migration v2.

The behavior description covers the full restore sequence. It
documents the RUNNING to RESUMING single transition (the kernel
walks the intermediate STOP arc), the post load PCI_COMMAND push
to the device, and the MSI or MSI-X eventfd rearm that the
kernel state does not carry. Behavior matches QEMU
vfio_pci_load_config().

It notes one limitation, the snapshot format stores the opaque
device blob as base64 inside the snapshot JSON, which may benefit
from a binary transport path for very large state.

docs/snapshot_restore.md replaces its VFIO out of scope
limitation with a short section that points to docs/vfio.md for
the requirements and behavior.

Signed-off-by: Saravanan D <[email protected]>