Thanks to visit codestin.com
Credit goes to github.com

Skip to content

arm64: correct the guest clock across snapshot/restore and migration - #8343

Merged
rbradford merged 6 commits into
cloud-hypervisor:mainfrom
atishp04:arm64_snapsthot_time
Jun 19, 2026
Merged

rbradford merged 6 commits into
cloud-hypervisor:mainfrom
atishp04:arm64_snapsthot_time

Conversation

@atishp04

@atishp04 atishp04 commented Jun 6, 2026

Copy link
Copy Markdown

On x86, a snapshot-restored or migrated guest catches its CLOCK_REALTIME back up to real time via the kvmclock path (KVM_SET_CLOCK + KVM_CLOCK_REALTIME).

ARM64 has no such helper. Cloud Hypervisor just round-trips CNTVCT_EL0 which results in old time(behind real time by the entire downtime) at resume in a cold-restored or migrated guest resumes.

This series mirrors the x86 fix for arm64/KVM using KVM_REG_ARM_TIMER_CNT ONEREG inteface. At snapshot time, it records the guest counter together with the host wall clock and counter frequency; at restore/migration-receive, before the vCPUs run, it advances CNTVCT by the wall-clock time that elapsed while the VM was down. arm64 has no kernel-side helper, so the VMM does the arithmetic itself.

  • KVM/arm64 only — x86 is unchanged and MSHV is unaffected.
  • an unreadable host clock or a cross-host CNTFRQ mismatch will fail now rather than resuming with a wrong clock.

Tested with

  1. new test_snapshot_check_guest_time (boot → snapshot → wait → restore → assert the guest's UTC caught up) on x86 and ARM64
  2. Manual with snapshot/restore on an arm64/KVM host.

We also considered the VM-wide counter-offset ioctl (KVM_CAP_COUNTER_OFFSET; available > Linux 6.4) as an alternative but advancing CNTVCT through the ONE_REG is the better fit here. It reuses the exact path snapshot/restore already uses for the guest counter restore writes CNTVCT via SET_ONE_REG, so the correction is just writing the advanced value through the same mechanism, instead of bolting on a second, capability-gated API.

Let us know if there is any other angle which favors the KVM_CAP_COUNTER_OFFSET

v1 → v2 summary

  • Backend-agnostic clock abstraction on the hypervisor Vm trait (snapshot_clock()/restore_clock() + ClockState) so the VMM drives save/restore uniformly and arch/backend details stay behind the hypervisor layer (Suggested-by Sebastien Boeuf.)

  • aarch64 multi-vCPU correctness gate the counter write on KVM_CAP_COUNTER_OFFSET — Linux ≥6.4 tracks the vtimer offset VM-wide (one boot-vCPU write), older kernels per-vCPU (write all). v1 wrote vcpu0 only.

  • Added a pause/resume integration test for ARM64

Testing

Feature behavior x86_64 aarch64
Snapshot → restore: clock catches up to wall time (intra-host) ✅ integration test + manual ✅ integration test + manual
Snapshot → restore: clock catches up to wall time (inter-host) X ✅ manual
Same-host pause → resume: no backward jump ✅ manual ✅ manual
Multi-vCPU consistency after restore ✅ manual ✅ manual

@atishp04
atishp04 requested a review from a team as a code owner June 6, 2026 01:46
@rbradford

Copy link
Copy Markdown
Member

@atishp04 CI is quite unhappy!

@atishp04
atishp04 force-pushed the arm64_snapsthot_time branch from aa3a213 to db9793b Compare June 8, 2026 19:55
@atishp04

atishp04 commented Jun 8, 2026

Copy link
Copy Markdown
Author

@atishp04 CI is quite unhappy!

Fixed the linter issues and the CI is all green now.

@rbradford

Copy link
Copy Markdown
Member

@atishp04 CI is quite unhappy!

Fixed the linter issues and the CI is all green now.

Not quite.

@atishp04

atishp04 commented Jun 9, 2026

Copy link
Copy Markdown
Author

@atishp04 CI is quite unhappy!

Fixed the linter issues and the CI is all green now.

Not quite.

Oops. I spoke too soon. Let me check.

@atishp04
atishp04 force-pushed the arm64_snapsthot_time branch from db9793b to b3e324c Compare June 9, 2026 19:18
Comment thread hypervisor/src/kvm/mod.rs Outdated
// SAFETY: Safe because `mrs cntfrq_el0` only reads a read-only system
// register and touches no memory (nomem, nostack, preserves_flags).
unsafe {
std::arch::asm!(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe use std::arch:asm;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can just be a static method in arch. There is no hypervisor dependency here.

Comment thread vmm/src/cpu.rs Outdated
Comment thread vmm/src/cpu.rs Outdated
Comment thread vmm/src/vm.rs Outdated
@rbradford

Copy link
Copy Markdown
Member

@rhakobyan Can you also take a look

Comment thread vmm/src/cpu.rs Outdated
Comment thread hypervisor/src/kvm/mod.rs Outdated
Comment thread hypervisor/src/kvm/mod.rs Outdated
Comment thread hypervisor/src/kvm/mod.rs
Comment thread hypervisor/src/kvm/mod.rs Outdated
Comment thread vmm/src/vm.rs Outdated
Comment thread hypervisor/src/kvm/mod.rs Outdated
Comment thread vmm/src/cpu.rs Outdated
Comment thread vmm/src/cpu.rs Outdated
Comment thread hypervisor/src/kvm/mod.rs
Comment thread hypervisor/src/kvm/mod.rs
/// boot vCPU (see `CpuManager::advance_timer`).
///
#[cfg(target_arch = "aarch64")]
fn set_cntvct(&self, val: u64) -> cpu::Result<()> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upon reflection I actually think it would be better to expose set/get_one_reg rather than this specific register?

@atishp04 atishp04 Jun 12, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rbradford
Not sure what do you mean here.

Do you mean the caller should invoke set_one_reg with KVM_REG_ARM_TIMER_CNT instead of invoking set_cntvct ? I don't mind either way but can you clarify why do you think that is better that the current abstraction ?

Given that, the new common abstraction of snapshot_clock and restore_clock will exist in vm abstraction, invoking set/get_one_reg directly from there would be weird ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rbradford Not sure what do you mean here.

Do you mean the caller should invoke set_one_reg with KVM_REG_ARM_TIMER_CNT instead of invoking set_cntvct ? I don't mind either way but can you clarify why do you think that is better that the current abstraction ?

Given that, the new common abstraction of snapshot_clock and restore_clock will exist in vm abstraction, invoking set/get_one_reg directly from there would be weird ?

What do you mean "vm abstraction" vmm/vm or hypervisor/vm? I agree with the former it would be weird but maybe work for the latter. I just wonder if it's too prescriptive when in fact just exposing set_one_reg/get_one_reg might be cleaner.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hypervisor/vm. Let me push the v2 as it has changed a quite a bit due to the moving the snapshot/restore_clock to hypervisor/vm. If you still think set_one_reg/get_one_reg instead of get/set_cntvct/, I can change that in v3.

Comment thread hypervisor/src/kvm/mod.rs Outdated
// SAFETY: Safe because `mrs cntfrq_el0` only reads a read-only system
// register and touches no memory (nomem, nostack, preserves_flags).
unsafe {
std::arch::asm!(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can just be a static method in arch. There is no hypervisor dependency here.

Comment thread hypervisor/src/kvm/mod.rs
Comment thread hypervisor/src/kvm/mod.rs

@mcfi mcfi left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks.

@atishp04
atishp04 force-pushed the arm64_snapsthot_time branch from b3e324c to ccaff60 Compare June 13, 2026 00:12
Comment thread hypervisor/src/kvm/mod.rs
@atishp04
atishp04 force-pushed the arm64_snapsthot_time branch from ccaff60 to 2706b6e Compare June 13, 2026 00:33

@sboeuf sboeuf left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for reworking through the Hypervisor abstraction!

Comment thread vmm/src/cpu.rs Outdated
@rbradford

Copy link
Copy Markdown
Member

@atishp04 kvm-ioctls/bindings releases made.

@atishp04

Copy link
Copy Markdown
Author

@atishp04 kvm-ioctls/bindings releases made.

@rbradford
It seems there are nested dependancies on kvm-ioctl.

The vfio-ioctl is pinned to 0.6.1 which have kvm-ioctl 0.24.0.
https://github.com/rust-vmm/vfio/blob/main/vfio-ioctls/Cargo.toml

Is it okay to bump vfio-ioctl as well for this purpose ? I can send a PR if it is okay.

@atishp04
atishp04 force-pushed the arm64_snapsthot_time branch 2 times, most recently from 7af013a to 72935fb Compare June 15, 2026 22:03
@atishp04
atishp04 requested a review from rbradford June 15, 2026 22:04
@rbradford

Copy link
Copy Markdown
Member

@atishp04 kvm-ioctls/bindings releases made.

@rbradford It seems there are nested dependancies on kvm-ioctl.

The vfio-ioctl is pinned to 0.6.1 which have kvm-ioctl 0.24.0. https://github.com/rust-vmm/vfio/blob/main/vfio-ioctls/Cargo.toml

Is it okay to bump vfio-ioctl as well for this purpose ? I can send a PR if it is okay.

Sure thing.

@atishp04
atishp04 force-pushed the arm64_snapsthot_time branch 2 times, most recently from e640970 to e24fd35 Compare June 16, 2026 22:39

@rbradford rbradford left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good!

Comment thread cloud-hypervisor/tests/integration.rs
Comment thread hypervisor/src/lib.rs Outdated
#[cfg(any(target_arch = "x86_64", all(target_arch = "aarch64", feature = "kvm")))]
pub enum ClockState {
#[cfg(target_arch = "x86_64")]
X86(ClockData),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we even need this since on each arch its independent? e.g. you can't build for both?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. While reviewing it again, I think we can alias create a single alias or just have ARM64 version of the TimerState name as ClockData to match x86 name.

A) Per-arch type alias
#[cfg(target_arch = "x86_64")]
pub type ClockState = ClockData; // existing kvmclock / HV ref-time enum
#[cfg(all(target_arch = "aarch64", feature = "kvm"))]
pub type ClockState = TimerState; // { cntvct, host_realtime_ns, cntfrq }

B) One name, no alias — make ClockData itself per-arch and drop ClockState/TimerState:
#[cfg(target_arch = "x86_64")]
pub enum ClockData { Kvm(..), Mshv(..) } // unchanged
#[cfg(all(target_arch = "aarch64", feature = "kvm"))]
pub struct ClockData { cntvct: u64, host_realtime_ns: u64, cntfrq: u64 }

The disadvantage of B is the name doesn't truly describe what the architecture supports for ARM64 but that's just semantic though. I prefer A though as future readers may get confused unless the look at the ARM64 version of internal structure.

Do you have any preference?

@atishp04 atishp04 Jun 17, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can also Rename ClockData to something more generic GuestClock as a third option but I am not sure if it is worth the churn.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think A is fine for now

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Updated the PR. PTAL.

@atishp04
atishp04 force-pushed the arm64_snapsthot_time branch from e24fd35 to f2066e9 Compare June 17, 2026 18:29

@rbradford rbradford left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if you exploit the Option and default values I think you could remove a lot of the feature gates which would make the code more readable (I find too many compile time #[cfg(..)] a bit overwhelming (and this isn't performance critical code!).

Comment thread cloud-hypervisor/tests/integration.rs Outdated
Comment on lines +8011 to +8013
feature = "kvm",
not(feature = "mshv"),
any(target_arch = "x86_64", target_arch = "aarch64")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could definitely simplify this - the other tests in the same boat are just not(feature = "mshv")

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment thread hypervisor/src/kvm/mod.rs Outdated
host: host_cntfrq,
});
}
let now_ns = std::time::SystemTime::now()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying to remove full paths like that.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also fixed the similar instances introduced in previously merged clock restore patch. I have folded that into the first patch. Happy to split it to a separate patch if CH prefers kernel style as commit preferences as well where no unrelated change must be present in the commit.

Comment thread hypervisor/src/vm.rs
@atishp04
atishp04 force-pushed the arm64_snapsthot_time branch 2 times, most recently from 4c1a354 to 365434a Compare June 18, 2026 21:03

@rbradford rbradford left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going to send to MQ - FYI be prepared for it to fail as this is the first time the ARM64 CI will have seen it.

@rbradford
rbradford enabled auto-merge June 18, 2026 21:28
Atish Patra added 6 commits June 18, 2026 22:56
Preserving the guest clock across pause/resume and snapshot/restore is
currently open-coded in the VMM against the x86-only
get_clock/set_clock. aarch64 needs the same correction but via a
different mechanism (i.e. the architected counter, CNTVCT). Having a
common backend-agnostic interface that VMM can drive uniformly allows us
to keep the architecture details behind the Hypervisor abstraction.

This commit only introduces the abstraction while the future commits
will actually move the implementation to use it.

Use this opportunity to fix the full path to get SystemTime as well.

Suggested-by: Sebastien Boeuf <[email protected]>
Signed-off-by: Atish Patra <[email protected]>
Currently, VM pause/resume/snapshot paths invoke architecture specific
bits for guest clock udpates which ideally belongs to hypervisor layer.

Route it through the snapshot_clock()/restore_clock() pair added in the
previous commit instead, so the VMM no longer depends on an architecture
specific clock API and the upcoming aarch64 backend can hook the same
path without a parallel branch in vm.rs.

Signed-off-by: Atish Patra <[email protected]>
Unlike x86, ARM64 has no kvmclock support to sync guest time upon
required. However, the guest reads the architected virtual timer
(CNTVCT_EL0) directly which can be modified by the VMM to update the
time after snapshot restore. Since the CNTVCT is in ticks, we also need
to read CNTFRQ (via mrs due to lack of ONEREG interface) to compute the
ticks from wall clock difference.

Because the counter is a vCPU register, the capture must run with the
vCPUs quiesced, so the VMM now captures the clock just after
cpu_manager.pause() through the boot vCPU. This is behaviorally
identical for x86, whose clock is VM-wide. There is no restore/advance
yet, so aarch64 guests still resume behind real time until the following
commit.

Signed-off-by: Atish Patra <[email protected]>
Currently, Cloud Hypervisor round-trips CNTVCT_EL0 through
KVM_GET_REG_LIST/SET_ONE_REG, which leaves a cold-restored or migrated
guest behind real UTC by the downtime. Same-host pause/resume
self-corrects (the physical counter keeps running across the pause), so
only restore and migration cases required the clock to catch up to wall
clock time.

Since ARM has no kernel helper, compute the difference in wall clock
time and compute the ticks so that it can advance the CNTVCT correctly.
It is set via vcpu0 only as it affects a single VM wide value after
Linux 6.4. For older kernels, it was a truly vcpu value which needs to
be invoked for every vcpu.

Gated on all(target_arch = "aarch64", feature = "kvm"); x86 is
unchanged.

Basic manual test case (aarch64 + KVM) verified both in intra host and
inter host snapshot save/restore:

1. Boot a Linux guest; in the guest, `date -u` tracks the host's UTC.
2. Pause and snapshot the VM (ch-remote pause; ch-remote snapshot
   file:///<dir>).
3. Leave it down for several minutes (the off-host interval).
4. Restore and resume into a fresh VMM (ch-remote restore
   source_url=file:///<dir>,resume=true).
5. In the guest, run `date -u` again and compare to the host: the guest
   now tracks current UTC, having advanced by ~the time it spent down.

Before this change the restored guest reads behind real UTC by the
downtime; after it, the guest clock is back in sync (to within the
snapshot-to-restore sampling slop).

Signed-off-by: Atish Patra <[email protected]>
Add a variation of _test_snapshot_restore that, after taking a snapshot,
waits out a simulated off-host interval and then restores and resumes,
asserting that the guest's wall clock has caught up to the host. This
exercises the clock catch-up that each architecture provides on restore:
kvmclock (KVM_CLOCK_REALTIME) on x86_64 today, and the CNTVCT advance on
aarch64 with later commits.

On x86_64 the guest is booted with clocksource=kvm-clock as the guest
clock is caught up after pause/resume only in that mode. A
tsc-clocksource guest's restored TSC freezes across the interval and
would never catch up.

Take this opportunity to improve the snapshot restore test as the
existing bare boolean mechanism was bit hard to read with new test.

Signed-off-by: Atish Patra <[email protected]>
Add an aarch64 test that pauses a running VM, waits out an interval, and
resumes it on the same host, then asserts the guest wall clock still
matches the host. On aarch64 the architected counter free-runs across
the pause, so the guest self-corrects.

The downtime and skew tolerance are shared with the snapshot clock test.
x86_64 has its own kvmclock path and is covered by the snapshot clock
test.

Signed-off-by: Atish Patra <[email protected]>
@rbradford
rbradford force-pushed the arm64_snapsthot_time branch from 365434a to ae4d185 Compare June 18, 2026 22:05
@rbradford

Copy link
Copy Markdown
Member

Basic CI is failing - partly because this hasn't been rebased in a long time and so not current with the VmOwnership change so even if you tested locally with cargo clippy --features mshv you wouldn't notice.

I went ahead and rebased and fixed that clippy failure

@atishp04

Copy link
Copy Markdown
Author

Basic CI is failing - partly because this hasn't been rebased in a long time and so not current with the VmOwnership change so even if you tested locally with cargo clippy --features mshv you wouldn't notice.

I went ahead and rebased and fixed that clippy failure

Thanks. I keep forgetting how fast CH main is moving these days!

@rbradford
rbradford added this pull request to the merge queue Jun 18, 2026
Merged via the queue into cloud-hypervisor:main with commit eb1c64e Jun 19, 2026
38 checks passed
@rbradford rbradford added the bug-fix Bug fix to include in release notes label Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-fix Bug fix to include in release notes

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

5 participants