arch: x86_64: advertise mandatory Hyper-V partition privileges - #8380
Conversation
44925ea to
fc44a91
Compare
sboeuf
left a comment
There was a problem hiding this comment.
LGTM, but it would be good to have some Windows/Microsoft experts reviewing that behavior.
| | (1 << 5) // AccessHypercallMsrs | ||
| | (1 << 6) // AccessVpIndex |
There was a problem hiding this comment.
This breaks live migration and snapshot-restore. :-( We probably need some solution to that. A simple solution would be to introduce a hyperv_enlightenment_level integer in the state (default 0). Only include these at level >= 1 - start new VMs at level 1.
There was a problem hiding this comment.
@rbradford just curious, how do you know that? Can you explain the rationale behind the breakage?
There was a problem hiding this comment.
thinking about it again as well: During a migration/restore, you upgrade to a new VMM that can simply do more if I'm not mistaken - therefore I think it should be fine. but you can't migrate back.
There was a problem hiding this comment.
@rbradford just curious, how do you know that? Can you explain the rationale behind the breakage?
It does say it in the PR summary :-)
There was a problem hiding this comment.
error is like
["Failed to receive migratable component snapshot","Error checking cpu feature compatibility': CpuidCheckCompatibility"]
these should be the errors from CH
There was a problem hiding this comment.
@tonicmuroq Do you have access to the logs? There should be a log at the error level describing what the differences are between the CPUID values.
There was a problem hiding this comment.
You might have to check the logs on the destination as I think that is where you will find the failed CPUID compatibility check.
Does the live migration work without the changes introduced here? It could just be that the CPUs are not compatible. The ongoing CPU profiles work we are doing (#7068 (comment)) might be of help if that is the issue.
There was a problem hiding this comment.
@olivereanderson @rbradford my bad, just tested again, no errors. @olivereanderson was right, the reason i saw those errors was because i also introduced EAX bits 0/4 and max-leaf etc, it was not a clean fix, and final fix version should be this one.
|
@tonicmuroq I am very confused with regards to the state of your branch. The changes you introduce in this PR should be done in |
|
@olivereanderson oh no the fork is far behind, this changes everything, let me recheck this |
The Microsoft "Requirements for Implementing the Microsoft Hypervisor Interface" document marks exactly two privileges in CPUID leaf 0x40000003 EAX as "Must be set": AccessHypercallMsrs (bit 5) and AccessVpIndex (bit 6). Cloud Hypervisor advertised neither. Without bit 5, Windows guests abort enlightened-mode initialization before timer-API selection: HalpHvTimerApi is left NULL and every QueryPerformanceCounter call falls back to reading HV_X64_MSR_TIME_REF_COUNT (0x40000020), costing one VM exit per call. The guest never writes HV_X64_MSR_REFERENCE_TSC (0x40000021) to enable the reference TSC page, even though AccessPartitionReferenceTsc (bit 9) is advertised. With both bits set, Windows 10 22H2 and Windows 11 25H2 guests enable the reference TSC page at boot. Measured QueryPerformanceCounter throughput on a nested-KVM host went from ~71K calls/sec (14 us/call, one MSR exit each) to ~1.3M calls/sec (free, no exits); on bare metal from ~390K to ~1.9M calls/sec. Guest idle CPU and interrupt-service time drop correspondingly. Both MSR ranges are already handled in-kernel by KVM unconditionally, so no backend change is needed. Bisection across the full delta to QEMU's Hyper-V CPUID layout (vendor ID, max leaf, leaves 4-6 contents, build number) shows bit 5 is the only load-bearing change; bit 6 is included per the conformance document's mandate. Signed-off-by: Tonic Li <[email protected]> Signed-off-by: tonic <[email protected]>
fc44a91 to
45f3562
Compare
|
@olivereanderson rebased |
Related: #8383 — fixes the same "Windows on the slow clock path" symptom on the
vm.restorepath (restored TSC page goes stale). This PR is the cold-boot half; both are needed for Windows to use the Hyper-V reference TSC page.Problem
Windows guests on Cloud Hypervisor (
kvm_hyperv=on) never enable the Hyper-V reference TSC page. AlthoughAccessPartitionReferenceTsc(CPUID 0x40000003 EAX bit 9) is advertised, the guest never writesHV_X64_MSR_REFERENCE_TSC(0x40000021). EveryQueryPerformanceCountercall falls back to readingHV_X64_MSR_TIME_REF_COUNT(0x40000020) — one VM exit per clock read, millions per second on a busy guest.Boot-to-steady-state trace of a fresh Windows 11 25H2 guest on stock CH (90 s,
kvm:kvm_msrtracepoint):Windows happily uses the hypercall page, SynIC, and synthetic timers, but selectively refuses the TSC page. This is the issue anticipated in #4779.
Root cause
The Microsoft "Requirements for Implementing the Microsoft Hypervisor Interface" document marks exactly two partition privileges in CPUID leaf 0x40000003 EAX as "Must be set":
AccessHypercallMsrs(bit 5) andAccessVpIndex(bit 6). Cloud Hypervisor advertises neither.Without bit 5, Windows aborts enlightened-mode initialization before timer-API selection: the kernel's
HalpHvTimerApipointer is left NULL, andKeQueryPerformanceCounterpermanently takes therdmsr 0x40000020trap path (one VM exit per call) instead of the in-guest TSC-page read (~tens of ns, zero exits).We confirmed the gate by booting the same Windows image on the same host/kernel under QEMU (
-cpu host,hv-time,...): the guest writes0x40000021during early boot. Bisecting the full CPUID delta between QEMU's Hyper-V layout and Cloud Hypervisor's (vendor ID string, max leaf, build number, leaves 4–6 contents, EBX privilege bits, frequency-MSR bit, invariant-TSC passthrough) isolated bit 5 as the single load-bearing difference:Bit 6 is included in this PR because the conformance document mandates both. Every hypervisor on which Windows is known to use the TSC page already sets them: QEMU ORs
HV_HYPERCALL_AVAILABLEinto leaf 3 unconditionally whenever anyhv-*enlightenment is enabled, Propolis's default leaf-3 EAX isVP_INDEX | HYPERCALL("the minimum set of access rights that all Hyper-V-compatible hypervisors must grant"), and VirtualBox's GIM provider asserts both bits, citing the same Microsoft document.No backend change is needed: KVM handles the hypercall MSRs (0x40000000/0x40000001) and
HV_X64_MSR_VP_INDEXin-kernel unconditionally, and Windows already uses them opportunistically (it writes GUEST_OS_ID/HYPERCALL today despite the missing privilege bit — but its TSC-page init path checks the privilege mask strictly).Measured effect
All measurements with identical guest images (Windows 10 22H2 / Windows 11 25H2), identical noise suppression inside the guest, settled idle, PowerShell
Stopwatch.GetTimestamp()tight-loop (= QPC throughput). Hosts: nested = GCE n2 (Cascade Lake, KVM-on-KVM, kernel 6.8), bare metal = GCE c3-highmem-192-metal (Sapphire Rapids, kernel 6.17).QPC throughput (calls/sec)
Patched per-call cost lands at or below the PowerShell loop floor (~640–940 ns/iter) — the clock read itself is no longer measurable from the guest. For scale: plain GCE Windows on Google's hypervisor (working TSC page) measures ~944 K calls/sec on the same benchmark; the patched CH guests exceed it.
Host-side VM exits during QPC load (
perf kvm stat)0x40000020traffic disappears entirely; the boot trace shows the guest writingmsr_write 40000021 = 0xd001(TSC page GPA + enable bit), identical to its behavior under QEMU.Guest-side idle cost (settled, suppressed)
_TotalMicrosoft-Windows-Kernel-Powerevents/secWin11's high idle interrupt volume partially collapses once the clock path is fast (part of its background activity was induced by the slow clock); Windows 10 22H2 under the patch idles at 413–470 IRQ/s — the quietest configuration we measured anywhere.
Note for users restoring pre-patch snapshots
Snapshots store the vCPU CPUID and restore validates it (CpuidCheckCompatibility), so snapshots taken on a build without these bits will refuse to restore on a build with them (and vice-versa) — same behavior as any CPUID-affecting change. Freshly booted guests are unaffected. Additionally, a Windows guest that booted without the privilege decides against the TSC page once at boot; it must be cold-booted (not restored) under the patched CPUID to pick up the fast path.Re-tested it, it's okay.
Verification
tests: fresh Windows 10 22H2 + Windows 11 25H2 boots on nested-KVM and bare-metal hosts, kvm:kvm_msr trace confirms WRMSR 0x40000021 at boot and zero 0x40000020 traffic at steady state.kvm_hyperv=off(code path not taken). Withkvm_hyperv=on, the two added privilege bits correspond to MSRs KVM already implements; this matches what QEMU advertises to everyhv-*-enabled guest.