feat: allow selective VFIO BAR mapping - #7991

@DamianB-BitFlipper It would help reviewers if you could explain "why" we need this feature in the PR description.
I would also recommend you to check the current vfio integration test suite and ascertain whether we could develop an integration test to cover your new changes. The commit messages should have a description. Whenever a PR introduces a new argument, it should also have documentation update commit in that series.
You could move this PR to draft and change its status when you are ready.

My bad, should have been a draft...

@saravan2 Do you know who would be most capable to review these changes, before I proceed to add integration changes and fix the commit history?

I just want to make sure that these changes make sense. I also updated the description, adding the purpose of this PR.

@DamianB-BitFlipper What about lazily mapping the BARs on first access, at 1G granularity?

It might be a bit of a skills issue from my end, that I didn't implement it this way.

But if I get some pointers as to how to do it, I'll happily try it out and test it too on our system.

@DamianB-BitFlipper I can review your changes. The maintainers would chime in and shape your PR to get it ready for merge.

@DemiMarie We can consider lazy bar mapping as a feature request because it requires significant changes to the MMIO exit path.

When mounting a GPU device with 100+GB of VRAM, a lot of time is spent simply during the IOMMU mmaping.

Can you check if you have --platform vfio_p2p_dma=on in your config ? Thats the only path I could think of that would issue synchronous VFIO_IOMMU_MAP_DMA to the host kernel.

Every other path I checked is lazy fault-based.

So if your ML applications never touch the large BARs, with vfio_p2p_dma=off there shouldn't be any penalty for mmaping the 100+GB VRAM BAR.

I do not pass it, but vfio_p2p_dma defaults to on. I didn't even know about this setting, and will give vfio_p2p_dma off a try.

I just did an AB test where I booted four VMs with a single H200 passthrough. Two were with vfio_p2p_dma off and no x_no_mmap_bars set. The other is with vfio_p2p_dma left on by default and x_no_mmap_bars selectively excluding the frame buffer.

Here are the attach times:

vfio_p2p_dma off -> 12.336837652s, 12.514751692s
using x_no_mmap_bars -> 5.039115267s, 5.056194296s

With these numbers in mind, I think this feature is quite useful.

For curiosity's sake, what was the time taken for vfio_p2p_dma left on by default and no x_no_mmap_bars ? It could serve a valuable datapoint to turn vfio_p2p_dma off by default for the next cloud-hypervisor release.

vfio_p2p_dma explicitly on and not using x_no_mmap_bars: 12.387654951s, 12.136960855s

This is peculiar, I double checked I compiled and ran the right code. I am; those are the timings.

@DamianB-BitFlipper Is it possible that your host distro is Rocky 9 or RHEL 9 based with a 5.1x kernel?

Older kernels do not have hugepage VFIO support, so the mmap of VRAM BAR forces the host kernel to perform ioremap and KVM memslot metadata allocation at 4K granularity. The ~7 seconds you are shaving off with x_no_mmap_bars PR is by skipping the mmap entirely.

Nope. I'm running debian and kernel 6.16.9. I tried really hard to enable hugepage mmap for the iommu, but couldn't.

My bad, that's the guest. On the host, I am running Ubuntu 24.04 and 6.8.0.

Linux 6.8 kernel does not have hugepage VFIO mmap support. It was introduced in kernel 6.11.

On 6.8, the ioremap inside vfio_pci_core_mmap and KVM memslot metadata allocation both operate at 4K granularity.

On a 6.11+ kernel with hugepage support enabled, the same mmap should be significantly faster and the savings from x_no_mmap_bars would be much smaller.

Interesting, I didn't know. Though changing the host's kernel might be out of our control unfortunately.

@rbradford Would the maintainers accept this PR that primarily helps cloud-hypervisor VMs boot and hotplug GPUs faster on hosts running older kernels ( pre 6.11 ) without hugepage VFIO mmap support ?

@rbradford Would the maintainers accept this PR that primarily helps cloud-hypervisor VMs boot and hotplug GPUs faster on hosts running older kernels ( pre 6.11 ) without hugepage VFIO mmap support ?

You could also discuss this in tomorrows Cloud Hypervisor Office Hour.

@DamianB-BitFlipper same link, same time!

On 6.8, the ioremap inside vfio_pci_core_mmap and KVM memslot metadata allocation both operate at 4K granularity

You can verify this thesis by comparing page table sizes grep VmPTE /proc/<CH PID>/status between master and x_no_mmap_bars on VM started with H200 passthrough.

When we dont mmap the large VRAM BAR on Linux 6.8 host kernel, the pagetable bloat would not be seen in CH and results in faster boot.

@DamianB-BitFlipper Since iommu mapping is not the source of latency, can you update the PR description with accurate information if you are going to table this in the office hour ?

We discussed today - all I think we need is a.) @saravan2's comment about it being page table related not IOMMU b.) Change to x_exclude_mmap_bars

Changes should be in!

I think @saravan2 recommended to widen to u64. See #7991 (comment)

I think @saravan2 recommended to widen to u64. See #7991 (comment)

Okey dokey - still have a whitespace issue to fix?

Fails on the VFIO test - timeout reached. Please investigate - might just need timeout increasing or could be a real issue

I'll give it a look

@rbradford I updated the integration test. Can you retry adding to the merge queue please?

@DamianB-BitFlipper Your new test caused integration test failures that kicked this change off the MQ - please investigate: https://github.com/cloud-hypervisor/cloud-hypervisor/actions/runs/25453310169/job/74677202205

@saravan2 made some suggestions in the review comment that should address the failing integration test.

I'll open a follow up PR

@saravan2 made some suggestions in the review comment that should address the failing integration test.

I'll open a follow up PR

#7991 (comment)

@DamianB-BitFlipper I had an offline conversation with @likebreath

The standard hugepages supported in IOMMU via iommu_pgsize did not accelerate vfio_dma_map portion.

That was because the PFN walk in drivers/vfio/vfio_iommu_type1.c, vaddr_get_pfns(), returned only "one" 4 KiB PFN at a time in 6.8 Linux kernel and forced the outer vfio_pin_pages_remote() loop to iterate on 4KiB granularity (instead of hugepage as I assumed based on this result from your earlier experiment in #7991 (comment))

https://elixir.bootlin.com/linux/v6.8/source/drivers/vfio/vfio_iommu_type1.c#L608-L722

The support to map in hugepage size was added later in Linux kernel in commit
torvalds/linux@0fd06844de5d

So as @likebreath mentioned in his comment the tradeoff for fast boot/hotplug achieved through x_exclude_mmap_bars is also via disabling p2p dma and the users cannot expect performant GPUDirect RDMA.

The current implementation we merged is correct and no further changes are necessary.