fix(vfio): fix VFIO device hot-remove fd leak preventing re-add - #7676
likebreath merged 2 commits into
Conversation
rbradford
left a comment
There was a problem hiding this comment.
Thanks for your submission. Please check the CONTRIBUTING.md guidelines.
9d3c053 to
73b0bf2
Compare
|
@rbradford I updated the PR to match the contributing guidelines and also wrote a disclaimer in the description. |
phip1611
left a comment
There was a problem hiding this comment.
LGTM, thanks for your contribution!
Left 2 remarks
likebreath
left a comment
There was a problem hiding this comment.
Thank you for the reporting and fixing the bug. Very nice write-up and PR. One suggestion below.
026dd2c to
a857220
Compare
|
@likebreath Would like your review once more. Feel free to merge if everything checks out on your end. Tested it on my side and it works! |
Good to confirm. Thank you. What your thought on the following API, which should be simpler and easier to use (single call that compares two instances of impl MmioRegion {
// Returns true if this region has the exact same memory slots as the other region.
pub fn has_matching_slots(&self, other: &MmioRegion) -> bool
}Please also cleanup the commit history so that it only contains changes for the final solution. |
1ceaf04 to
1e3d1aa
Compare
|
Ok. Cleaned up history, changed the API and tested. Works. |
likebreath
left a comment
There was a problem hiding this comment.
Overall looking good.
Last nit: Can you please also cleanup the commit message in the same way - so that it only explain the final change (without context of previous iterations). Thanks a lot!
Change has_matching_slots() to compare two MmioRegion instances directly rather than requiring callers to construct an intermediate HashSet of slot numbers. Remove the now-unused user_memory_region_slots() method and HashSet import. Signed-off-by: Damian Barabonkov <[email protected]>
When a VFIO device with multiple MMIO regions is hot-unplugged, each region must be individually matched and removed from the DeviceManager's mmio_regions list. Compare per-region rather than building an aggregate across all regions, which would never match any individual entry. Also remove the now-unused HashSet import. Signed-off-by: Damian Barabonkov <[email protected]>
1e3d1aa to
cadcf81
Compare
|
How about now? |
likebreath
left a comment
There was a problem hiding this comment.
I appreciate your persistence on this. Thanks again for the contribution—this is an important bug fix.
|
When might it be released? |
Summary
Fixes VFIO GPU hot-remove leaving group and container file descriptors open, causing re-add to fail with
"failed to open /dev/vfio/<group> group: Resource busy".Root Cause
During hot-add,
map_mmio_regions()createsArc<MmapRegion>entries (mmap'd from the VFIO device fd with MAP_SHARED) and clones them intoDeviceManager.mmio_regions. When the guest reprograms PCI BARs during boot,move_bar()updates the addresses inVfioPciDevice.common.mmio_regionsbut not in the DeviceManager's cloned copies.During hot-remove,
eject_device()callsretain()to remove the device's entries fromDeviceManager.mmio_regions, comparing byMmioRegion.startaddress. Since the addresses are desynchronized bymove_bar(), the comparison never matches and the entries are never removed. The survivingArc<MmapRegion>references keep the kernel struct file refcount on the VFIO device fd above zero (via the mmap VMA), even afterclose(fd). The kernel then refusesVFIO_GROUP_UNSET_CONTAINERwithEBUSYbecause it considers the device still in use.Fix
Replace the address-based
retain()with slot-based matching.UserMemoryRegion.slotnumbers are assigned at mmap creation time and are stable across BAR moves, making them a reliable identifier for cleanup.Two small public methods are added to
MmioRegionto expose slot information without changing field visibility.Disclaimer
As per the contributing guidelines, I thought I should write this here. This bug was debugged using Opus 4.6. The initial patch was also written with the aid of Opus 4.6. The final version was curated by a human (me).