hypervisor: kvm: preserve kvmclock realtime and fill if needed - #8256
Conversation
If `KVM_GET_CLOCK` already filled out the `realtime` field, it sets the `KVM_CLOCK_REALTIME` flag, but if we instead preserve this flag, the kernel will automatically adjust the kvmclock clock when calling `KVM_SET_CLOCK` based on the elapsed wall-clock time between pause and resume. This just requires removing the `reset_flags()` function, which allows the `KVM_CLOCK_REALTIME` flag to persist in the serialized clock state. However, the kernel does not always fill the `realtime` field, depending on clock source; in this case, fill `realtime` during pause based on the system time. This is not as precise as the automatic `KVM_GET_CLOCK` version, since we query the time slightly after the vCPU was paused, but it allows the clock to be resumed mostly in sync instead of being wildly off. In this case, we also set the `KVM_CLOCK_REALTIME` flag in the saved `struct kvmclock` so `KVM_SET_CLOCK` will adjust the clock on resume. Basic test case: 1. Run a VM with a Linux guest. 2. Pause the guest via `vm.pause` API. 3. Wait several minutes. 4. Resume the guest via `vm.resume` API. 5. Verify the guest time (e.g. via `date` command) is valid. 6. Verify guest is still using `kvm-clock` timesource: cat /sys/devices/system/clocksource/clocksource0/current_clocksource Before applying the patch, the guest clock would be off by the delta time between pause and resume; after the patch, the clock is (more or less) in sync with the correct wall-clock time. Old snapshots will not have the `KVM_CLOCK_REALTIME` flag populated, so they will not be affected by the new behavior. Signed-off-by: Daniel Verkamp <[email protected]>
e298f89 to
ed8f052
Compare
likebreath
left a comment
There was a problem hiding this comment.
This is a behavior change: kvmclock is now automatically adjusted on resume, restore, and live-migration. It's a quick way to keep the guest's wall clock correct, but it comes at the cost of guest-visible discontinuities that can trigger softlockup/RCU stall warnings, timer storms, etc.
I'd suggest separating the two concerns. Let pause/resume focus on guest execution continuity by keeping kvmclock frozen across the transition, and use a dedicated mechanism, say virtio-rtc,to notify the guest about the wall-clock shift. The guest can then adjust its wall clock without disturbing the rest of its timekeeping. This seems like a cleaner and more programmatic split of responsibility between the VMM and the guest.
I think softlockup is avoided due to
Agreed, I think this is the better long-term plan (especially since virtio-rtc can work on non-x86 platforms, unlike kvmclock). However, I am not sure of the status of virtio-rtc implementation in cloud-hypervisor, and I don't think it currently has a direct mechanism to notify the guest of a potential discontinuity (e.g. pause/resume) where re-reading the RTC should be performed (maybe alarm could be abused to do this, but I don't think that's what it was intended for); this might need an extra feature in the virtio-rtc spec, and I'm not sure even that would fully solve the problem, since the guest will not immediately update the clock on resume, only after guest code runs for some time with the old clock state (whereas the kvmclock state can be updated before resuming the guest). |
|
@likebreath Even though this is a behaviour change I think this is actually better - I think the clock lagging behind on pause/resume is almost certainly not the behaviour that the user wants. It is unfortunate that this not available on ARM immediately (we would need to extend the virtio-rtc spec to cover that) but I don't see that a reason not to improve things for the x86-64 users. In terms of timers (mis-)firing - is this different behaviour to NTP bringing the clock forward? It's also worth reminding ourselves that majority of the time explicit pause/resume is only really part of snapshot/restore and it's almost certainly the case that shifting to the current wallclock time is desired or implicitly part of live migration when in fact we would also not want that time to "disappear". |
Right. Good callout. We are already doing that from
I think so. NTP only adjusts To my understanding, the distinction matters because the two clocks have different contracts: For context, I referred QEMU's implementation (which keeps kvmclock frozen across pause/resume), and came to the conclusion that separating the two concerns is the more programmatic direction. That said, I may be missing context, and if there's a real use case demanding a quicker wall-clock fix, this isn't a blocker from my side. |
| } | ||
| } | ||
|
|
||
| pub fn set_realtime(&mut self, realtime: std::time::SystemTime) { |
There was a problem hiding this comment.
This(realtime) variable is only compiled if KVM is enabled. Shouldn't make this as kvm feature guarded?
Also, How this differs if the VMM provides an emulated RTC?
There was a problem hiding this comment.
- This does the right thing on MSHV as
ClockDatawont beClockData::Kvm - This actually more closely matches what happens with virtio-rtc. It always returns the current time when queried.
I don't think we should be thinking about this in terms of pause/resume as in that case the TSC continues to run. But rather frame this as what does a user reasonably expect the guest clock to be after restoring (or after a live migration). It would expect it to reflect the current time. Providing time catch-up after a migration is the intended use of
I don't think the concern about I think we should go ahead with this even though it is a change in behaviour - I would argue that this new behaviour is an improvement. |
If
KVM_GET_CLOCKalready filled out therealtimefield, it sets theKVM_CLOCK_REALTIMEflag, but if we instead preserve this flag, the kernel will automatically adjust the kvmclock clock when callingKVM_SET_CLOCKbased on the elapsed wall-clock time between pause and resume. This just requires removing thereset_flags()function, which allows theKVM_CLOCK_REALTIMEflag to persist in the serialized clock state.However, the kernel does not always fill the
realtimefield, depending on clock source; in this case, fillrealtimeduring pause based on the system time. This is not as precise as the automaticKVM_GET_CLOCKversion, since we query the time slightly after the vCPU was paused, but it allows the clock to be resumed mostly in sync instead of being wildly off. In this case, we also set theKVM_CLOCK_REALTIMEflag in the savedstruct kvmclocksoKVM_SET_CLOCKwill adjust the clock on resume.Basic test case:
Run a VM with a Linux guest.
Pause the guest via
vm.pauseAPI.Wait several minutes.
Resume the guest via
vm.resumeAPI.Verify the guest time (e.g. via
datecommand) is valid.Verify guest is still using
kvm-clocktimesource:cat /sys/devices/system/clocksource/clocksource0/current_clocksource
Before applying the patch, the guest clock would be off by the delta time between pause and resume; after the patch, the clock is (more or less) in sync with the correct wall-clock time.
Old snapshots will not have the
KVM_CLOCK_REALTIMEflag populated, so they will not be affected by the new behavior.