arm64: correct the guest clock across snapshot/restore and migration - #8343
Conversation
|
@atishp04 CI is quite unhappy! |
aa3a213 to
db9793b
Compare
Fixed the linter issues and the CI is all green now. |
Not quite. |
Oops. I spoke too soon. Let me check. |
db9793b to
b3e324c
Compare
| // SAFETY: Safe because `mrs cntfrq_el0` only reads a read-only system | ||
| // register and touches no memory (nomem, nostack, preserves_flags). | ||
| unsafe { | ||
| std::arch::asm!( |
There was a problem hiding this comment.
This can just be a static method in arch. There is no hypervisor dependency here.
|
@rhakobyan Can you also take a look |
| /// boot vCPU (see `CpuManager::advance_timer`). | ||
| /// | ||
| #[cfg(target_arch = "aarch64")] | ||
| fn set_cntvct(&self, val: u64) -> cpu::Result<()> { |
There was a problem hiding this comment.
Upon reflection I actually think it would be better to expose set/get_one_reg rather than this specific register?
There was a problem hiding this comment.
@rbradford
Not sure what do you mean here.
Do you mean the caller should invoke set_one_reg with KVM_REG_ARM_TIMER_CNT instead of invoking set_cntvct ? I don't mind either way but can you clarify why do you think that is better that the current abstraction ?
Given that, the new common abstraction of snapshot_clock and restore_clock will exist in vm abstraction, invoking set/get_one_reg directly from there would be weird ?
There was a problem hiding this comment.
@rbradford Not sure what do you mean here.
Do you mean the caller should invoke set_one_reg with KVM_REG_ARM_TIMER_CNT instead of invoking set_cntvct ? I don't mind either way but can you clarify why do you think that is better that the current abstraction ?
Given that, the new common abstraction of snapshot_clock and restore_clock will exist in vm abstraction, invoking set/get_one_reg directly from there would be weird ?
What do you mean "vm abstraction" vmm/vm or hypervisor/vm? I agree with the former it would be weird but maybe work for the latter. I just wonder if it's too prescriptive when in fact just exposing set_one_reg/get_one_reg might be cleaner.
There was a problem hiding this comment.
hypervisor/vm. Let me push the v2 as it has changed a quite a bit due to the moving the snapshot/restore_clock to hypervisor/vm. If you still think set_one_reg/get_one_reg instead of get/set_cntvct/, I can change that in v3.
| // SAFETY: Safe because `mrs cntfrq_el0` only reads a read-only system | ||
| // register and touches no memory (nomem, nostack, preserves_flags). | ||
| unsafe { | ||
| std::arch::asm!( |
There was a problem hiding this comment.
This can just be a static method in arch. There is no hypervisor dependency here.
b3e324c to
ccaff60
Compare
ccaff60 to
2706b6e
Compare
sboeuf
left a comment
There was a problem hiding this comment.
LGTM, thanks for reworking through the Hypervisor abstraction!
|
@atishp04 kvm-ioctls/bindings releases made. |
@rbradford The vfio-ioctl is pinned to 0.6.1 which have kvm-ioctl 0.24.0. Is it okay to bump vfio-ioctl as well for this purpose ? I can send a PR if it is okay. |
7af013a to
72935fb
Compare
Sure thing. |
e640970 to
e24fd35
Compare
| #[cfg(any(target_arch = "x86_64", all(target_arch = "aarch64", feature = "kvm")))] | ||
| pub enum ClockState { | ||
| #[cfg(target_arch = "x86_64")] | ||
| X86(ClockData), |
There was a problem hiding this comment.
Do we even need this since on each arch its independent? e.g. you can't build for both?
There was a problem hiding this comment.
Agreed. While reviewing it again, I think we can alias create a single alias or just have ARM64 version of the TimerState name as ClockData to match x86 name.
A) Per-arch type alias
#[cfg(target_arch = "x86_64")]
pub type ClockState = ClockData; // existing kvmclock / HV ref-time enum
#[cfg(all(target_arch = "aarch64", feature = "kvm"))]
pub type ClockState = TimerState; // { cntvct, host_realtime_ns, cntfrq }
B) One name, no alias — make ClockData itself per-arch and drop ClockState/TimerState:
#[cfg(target_arch = "x86_64")]
pub enum ClockData { Kvm(..), Mshv(..) } // unchanged
#[cfg(all(target_arch = "aarch64", feature = "kvm"))]
pub struct ClockData { cntvct: u64, host_realtime_ns: u64, cntfrq: u64 }
The disadvantage of B is the name doesn't truly describe what the architecture supports for ARM64 but that's just semantic though. I prefer A though as future readers may get confused unless the look at the ARM64 version of internal structure.
Do you have any preference?
There was a problem hiding this comment.
We can also Rename ClockData to something more generic GuestClock as a third option but I am not sure if it is worth the churn.
e24fd35 to
f2066e9
Compare
rbradford
left a comment
There was a problem hiding this comment.
I think if you exploit the Option and default values I think you could remove a lot of the feature gates which would make the code more readable (I find too many compile time #[cfg(..)] a bit overwhelming (and this isn't performance critical code!).
| feature = "kvm", | ||
| not(feature = "mshv"), | ||
| any(target_arch = "x86_64", target_arch = "aarch64") |
There was a problem hiding this comment.
You could definitely simplify this - the other tests in the same boat are just not(feature = "mshv")
| host: host_cntfrq, | ||
| }); | ||
| } | ||
| let now_ns = std::time::SystemTime::now() |
There was a problem hiding this comment.
Trying to remove full paths like that.
There was a problem hiding this comment.
I also fixed the similar instances introduced in previously merged clock restore patch. I have folded that into the first patch. Happy to split it to a separate patch if CH prefers kernel style as commit preferences as well where no unrelated change must be present in the commit.
4c1a354 to
365434a
Compare
rbradford
left a comment
There was a problem hiding this comment.
Going to send to MQ - FYI be prepared for it to fail as this is the first time the ARM64 CI will have seen it.
Preserving the guest clock across pause/resume and snapshot/restore is currently open-coded in the VMM against the x86-only get_clock/set_clock. aarch64 needs the same correction but via a different mechanism (i.e. the architected counter, CNTVCT). Having a common backend-agnostic interface that VMM can drive uniformly allows us to keep the architecture details behind the Hypervisor abstraction. This commit only introduces the abstraction while the future commits will actually move the implementation to use it. Use this opportunity to fix the full path to get SystemTime as well. Suggested-by: Sebastien Boeuf <[email protected]> Signed-off-by: Atish Patra <[email protected]>
Currently, VM pause/resume/snapshot paths invoke architecture specific bits for guest clock udpates which ideally belongs to hypervisor layer. Route it through the snapshot_clock()/restore_clock() pair added in the previous commit instead, so the VMM no longer depends on an architecture specific clock API and the upcoming aarch64 backend can hook the same path without a parallel branch in vm.rs. Signed-off-by: Atish Patra <[email protected]>
Unlike x86, ARM64 has no kvmclock support to sync guest time upon required. However, the guest reads the architected virtual timer (CNTVCT_EL0) directly which can be modified by the VMM to update the time after snapshot restore. Since the CNTVCT is in ticks, we also need to read CNTFRQ (via mrs due to lack of ONEREG interface) to compute the ticks from wall clock difference. Because the counter is a vCPU register, the capture must run with the vCPUs quiesced, so the VMM now captures the clock just after cpu_manager.pause() through the boot vCPU. This is behaviorally identical for x86, whose clock is VM-wide. There is no restore/advance yet, so aarch64 guests still resume behind real time until the following commit. Signed-off-by: Atish Patra <[email protected]>
Currently, Cloud Hypervisor round-trips CNTVCT_EL0 through KVM_GET_REG_LIST/SET_ONE_REG, which leaves a cold-restored or migrated guest behind real UTC by the downtime. Same-host pause/resume self-corrects (the physical counter keeps running across the pause), so only restore and migration cases required the clock to catch up to wall clock time. Since ARM has no kernel helper, compute the difference in wall clock time and compute the ticks so that it can advance the CNTVCT correctly. It is set via vcpu0 only as it affects a single VM wide value after Linux 6.4. For older kernels, it was a truly vcpu value which needs to be invoked for every vcpu. Gated on all(target_arch = "aarch64", feature = "kvm"); x86 is unchanged. Basic manual test case (aarch64 + KVM) verified both in intra host and inter host snapshot save/restore: 1. Boot a Linux guest; in the guest, `date -u` tracks the host's UTC. 2. Pause and snapshot the VM (ch-remote pause; ch-remote snapshot file:///<dir>). 3. Leave it down for several minutes (the off-host interval). 4. Restore and resume into a fresh VMM (ch-remote restore source_url=file:///<dir>,resume=true). 5. In the guest, run `date -u` again and compare to the host: the guest now tracks current UTC, having advanced by ~the time it spent down. Before this change the restored guest reads behind real UTC by the downtime; after it, the guest clock is back in sync (to within the snapshot-to-restore sampling slop). Signed-off-by: Atish Patra <[email protected]>
Add a variation of _test_snapshot_restore that, after taking a snapshot, waits out a simulated off-host interval and then restores and resumes, asserting that the guest's wall clock has caught up to the host. This exercises the clock catch-up that each architecture provides on restore: kvmclock (KVM_CLOCK_REALTIME) on x86_64 today, and the CNTVCT advance on aarch64 with later commits. On x86_64 the guest is booted with clocksource=kvm-clock as the guest clock is caught up after pause/resume only in that mode. A tsc-clocksource guest's restored TSC freezes across the interval and would never catch up. Take this opportunity to improve the snapshot restore test as the existing bare boolean mechanism was bit hard to read with new test. Signed-off-by: Atish Patra <[email protected]>
Add an aarch64 test that pauses a running VM, waits out an interval, and resumes it on the same host, then asserts the guest wall clock still matches the host. On aarch64 the architected counter free-runs across the pause, so the guest self-corrects. The downtime and skew tolerance are shared with the snapshot clock test. x86_64 has its own kvmclock path and is covered by the snapshot clock test. Signed-off-by: Atish Patra <[email protected]>
365434a to
ae4d185
Compare
|
Basic CI is failing - partly because this hasn't been rebased in a long time and so not current with the I went ahead and rebased and fixed that clippy failure |
Thanks. I keep forgetting how fast CH main is moving these days! |
On x86, a snapshot-restored or migrated guest catches its CLOCK_REALTIME back up to real time via the kvmclock path (KVM_SET_CLOCK + KVM_CLOCK_REALTIME).
ARM64 has no such helper. Cloud Hypervisor just round-trips CNTVCT_EL0 which results in old time(behind real time by the entire downtime) at resume in a cold-restored or migrated guest resumes.
This series mirrors the x86 fix for arm64/KVM using KVM_REG_ARM_TIMER_CNT ONEREG inteface. At snapshot time, it records the guest counter together with the host wall clock and counter frequency; at restore/migration-receive, before the vCPUs run, it advances CNTVCT by the wall-clock time that elapsed while the VM was down. arm64 has no kernel-side helper, so the VMM does the arithmetic itself.
Tested with
We also considered the VM-wide counter-offset ioctl (KVM_CAP_COUNTER_OFFSET; available > Linux 6.4) as an alternative but advancing CNTVCT through the ONE_REG is the better fit here. It reuses the exact path snapshot/restore already uses for the guest counter restore writes CNTVCT via SET_ONE_REG, so the correction is just writing the advanced value through the same mechanism, instead of bolting on a second, capability-gated API.
Let us know if there is any other angle which favors the KVM_CAP_COUNTER_OFFSET
v1 → v2 summary
Backend-agnostic clock abstraction on the hypervisor Vm trait (snapshot_clock()/restore_clock() + ClockState) so the VMM drives save/restore uniformly and arch/backend details stay behind the hypervisor layer (Suggested-by Sebastien Boeuf.)
aarch64 multi-vCPU correctness gate the counter write on KVM_CAP_COUNTER_OFFSET — Linux ≥6.4 tracks the vtimer offset VM-wide (one boot-vCPU write), older kernels per-vCPU (write all). v1 wrote vcpu0 only.
Added a pause/resume integration test for ARM64
Testing