vmm: Prefault snapshot pages in background - #8150
Conversation
|
/cc @shayonj |
rbradford
left a comment
There was a problem hiding this comment.
What happens if the user triggers another snapshot or live migration whilst this thread is still alive (this was quite possibly a problem before the thread was added.)
This is a good question and actually this is now less of a problem with the new thread. Given we're actively bringing the pages in with the prefaulting thread, after a few seconds (minutes if the VM RAM size is huge), the uffd thread is going to complete. This was almost never the case before, unless the guest was actively touching every single page. About the underlying problem that you have mentioned, if not all the pages have been faulted yet when a new snapshot occurs, I'd expect the snapshot to be slowed down but still should succeed. The slowdown would come from the fact the VMM would be accessing the pages to copy them into the new snapshot file, but some pages might not be there yet, and this would trigger the uffd handler. Or am I mistaken and would an access to the guest RAM from the VMM not trigger the uffd handler? |
| let page_addr = range.host_addr + i * range.page_size; | ||
| let len = range.page_size as usize; | ||
|
|
||
| if let Err(e) = snapshot_file.read_at(&mut page_buf[..len], file_pos) { |
There was a problem hiding this comment.
Should you also use read_exact_at here?
| let mut pages_existed: u64 = 0; | ||
|
|
||
| for range in ranges { | ||
| let num_pages = range.length / range.page_size; |
There was a problem hiding this comment.
Better to use div_ceil for page count calculations (than floor division.) That would match the other code in this file.
| "UFFD prefault: done in {elapsed:.3?} — installed={pages_installed} \ | ||
| existed={pages_existed} total={total_pages}" | ||
| ); | ||
| stop_event.write(1).ok(); |
There was a problem hiding this comment.
I would prefer if this were called "done" or "completed" stop sounds like a verb - that would be used to stop the execution rather than to say it's completed.
| let mut pages_installed: u64 = 0; | ||
| let mut pages_existed: u64 = 0; |
There was a problem hiding this comment.
I also find the naming of these slightly confusing?
|
@rbradford I've updated the PR. It doesn't introduce a dedicated thread anymore, as it reuses the exiting one, faulting one page per iteration. And also introduced a bitmap to track the pages which have been previously faulted so that we don't copy them again from the prefault handler. |
72264a3 to
20d5a6f
Compare
rbradford
left a comment
There was a problem hiding this comment.
I like we don't have an extra thread.
Userfaultfd is a great mechanism for providing fast restore to Cloud Hypervisor VMs. But that means the price to pay for bringing pages in happens at runtime, which might slow down the guest when it's touching pages which haven't been brought in yet. By prefaulting the pages in the background, we're trying to get the best of both worlds. That means we still get a very fast restore with the uffd handler, but within a few seconds (depending on VM's RAM size), we also get the pages fully faulted and we can stop the uffd handler thread at that point. Signed-off-by: Sebastien Boeuf <[email protected]> Assisted-by: Claude:claude-opus-4-7
|
Apologies for being late to the party. I have been busy at work. I love the approach and skimmed it and looks good to me. I have nothing useful to add, if anything I can always follow up. Thank you for the contribution and giving this a good design shape 🙏🏾 |
Userfaultfd is a great mechanism for providing fast restore to Cloud
Hypervisor VMs. But that means the price to pay for bringing pages in
happens at runtime, which might slow down the guest when it's touching
pages which haven't been brought in yet.
By prefaulting the pages in the background, we're trying to get the best
of both worlds. That means we still get a very fast restore with the
uffd handler, but within a few seconds (depending on VM's RAM size), we
also get the pages fully faulted and we can stop the uffd handler thread
at that point.