Thanks to visit codestin.com
Credit goes to github.com

Skip to content

arch: x86_64: advertise mandatory Hyper-V partition privileges - #8380

Merged
rbradford merged 1 commit into
cloud-hypervisor:mainfrom
cocoonstack:fix/hyperv-must-set-privileges
Jun 15, 2026
Merged

rbradford merged 1 commit into
cloud-hypervisor:mainfrom
cocoonstack:fix/hyperv-must-set-privileges

Conversation

@tonicmuroq

@tonicmuroq tonicmuroq commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Related: #8383 — fixes the same "Windows on the slow clock path" symptom on the vm.restore path (restored TSC page goes stale). This PR is the cold-boot half; both are needed for Windows to use the Hyper-V reference TSC page.

Problem

Windows guests on Cloud Hypervisor (kvm_hyperv=on) never enable the Hyper-V reference TSC page. Although AccessPartitionReferenceTsc (CPUID 0x40000003 EAX bit 9) is advertised, the guest never writes HV_X64_MSR_REFERENCE_TSC (0x40000021). Every QueryPerformanceCounter call falls back to reading HV_X64_MSR_TIME_REF_COUNT (0x40000020) — one VM exit per clock read, millions per second on a busy guest.

Boot-to-steady-state trace of a fresh Windows 11 25H2 guest on stock CH (90 s, kvm:kvm_msr tracepoint):

msr_read  0x40000020  (TIME_REF_COUNT, slow path):   3,249,387
msr_write 0x400000b1  (STIMER0_COUNT):                  75,070
msr_write 0x40000021  (REFERENCE_TSC enable):                0   ← never attempted

Windows happily uses the hypercall page, SynIC, and synthetic timers, but selectively refuses the TSC page. This is the issue anticipated in #4779.

Root cause

The Microsoft "Requirements for Implementing the Microsoft Hypervisor Interface" document marks exactly two partition privileges in CPUID leaf 0x40000003 EAX as "Must be set": AccessHypercallMsrs (bit 5) and AccessVpIndex (bit 6). Cloud Hypervisor advertises neither.

Without bit 5, Windows aborts enlightened-mode initialization before timer-API selection: the kernel's HalpHvTimerApi pointer is left NULL, and KeQueryPerformanceCounter permanently takes the rdmsr 0x40000020 trap path (one VM exit per call) instead of the in-guest TSC-page read (~tens of ns, zero exits).

We confirmed the gate by booting the same Windows image on the same host/kernel under QEMU (-cpu host,hv-time,...): the guest writes 0x40000021 during early boot. Bisecting the full CPUID delta between QEMU's Hyper-V layout and Cloud Hypervisor's (vendor ID string, max leaf, build number, leaves 4–6 contents, EBX privilege bits, frequency-MSR bit, invariant-TSC passthrough) isolated bit 5 as the single load-bearing difference:

Bisection round Config TSC page enabled?
stock EAX bits 1,2,3,9 no
stock + bit 5 + bit 6 yes
stock + bit 5 only yes
stock + bit 6 only no
QEMU-mirror minus bits 5,6 everything else from QEMU's layout no

Bit 6 is included in this PR because the conformance document mandates both. Every hypervisor on which Windows is known to use the TSC page already sets them: QEMU ORs HV_HYPERCALL_AVAILABLE into leaf 3 unconditionally whenever any hv-* enlightenment is enabled, Propolis's default leaf-3 EAX is VP_INDEX | HYPERCALL ("the minimum set of access rights that all Hyper-V-compatible hypervisors must grant"), and VirtualBox's GIM provider asserts both bits, citing the same Microsoft document.

No backend change is needed: KVM handles the hypercall MSRs (0x40000000/0x40000001) and HV_X64_MSR_VP_INDEX in-kernel unconditionally, and Windows already uses them opportunistically (it writes GUEST_OS_ID/HYPERCALL today despite the missing privilege bit — but its TSC-page init path checks the privilege mask strictly).

Measured effect

All measurements with identical guest images (Windows 10 22H2 / Windows 11 25H2), identical noise suppression inside the guest, settled idle, PowerShell Stopwatch.GetTimestamp() tight-loop (= QPC throughput). Hosts: nested = GCE n2 (Cascade Lake, KVM-on-KVM, kernel 6.8), bare metal = GCE c3-highmem-192-metal (Sapphire Rapids, kernel 6.17).

QPC throughput (calls/sec)

Host Guest stock CH patched CH gain
nested GCE n2 Win11 25H2 71,366 (14,016 ns/call) 1,300,966 (~free) 18×
bare metal c3 Win10 22H2 390,000 (2,547 ns/call) 1,810,000–1,900,000 (527–553 ns/call) 4.7×
bare metal c3 Win11 25H2 390,000 (2,546 ns/call) 1,760,000–1,860,000 (536–570 ns/call) 4.6×

Patched per-call cost lands at or below the PowerShell loop floor (~640–940 ns/iter) — the clock read itself is no longer measurable from the guest. For scale: plain GCE Windows on Google's hypervisor (working TSC page) measures ~944 K calls/sec on the same benchmark; the patched CH guests exceed it.

Host-side VM exits during QPC load (perf kvm stat)

stock CH patched CH
MSR_READ share of all exits 93–99% (87–167 K exits/s) zero samples (both OSes)
remaining exits HLT + EXTERNAL_INTERRUPT (~700–930/s) + EPT_MISCONFIG

0x40000020 traffic disappears entirely; the boot trace shows the guest writing msr_write 40000021 = 0xd001 (TSC page GPA + enable bit), identical to its behavior under QEMU.

Guest-side idle cost (settled, suppressed)

Metric stock nested Win11 patched nested Win11 stock metal Win11 patched metal Win11
idle CPU _Total 5.7% 1.49% 0.7–0.8% 0–1.1%
interrupts/sec 4,098 2,067 4,079 ~2,270
implied per-IRQ service time ~105 µs ~26 µs ~17 µs-class
Microsoft-Windows-Kernel-Power events/sec ~13,100 ~6,500 ~7,900 ~2,640

Win11's high idle interrupt volume partially collapses once the clock path is fast (part of its background activity was induced by the slow clock); Windows 10 22H2 under the patch idles at 413–470 IRQ/s — the quietest configuration we measured anywhere.

Note for users restoring pre-patch snapshots

Snapshots store the vCPU CPUID and restore validates it (CpuidCheckCompatibility), so snapshots taken on a build without these bits will refuse to restore on a build with them (and vice-versa) — same behavior as any CPUID-affecting change. Freshly booted guests are unaffected. Additionally, a Windows guest that booted without the privilege decides against the TSC page once at boot; it must be cold-booted (not restored) under the patched CPUID to pick up the fast path.
Re-tested it, it's okay.

Verification

  • tests: fresh Windows 10 22H2 + Windows 11 25H2 boots on nested-KVM and bare-metal hosts, kvm:kvm_msr trace confirms WRMSR 0x40000021 at boot and zero 0x40000020 traffic at steady state.
  • Linux guests: unaffected with kvm_hyperv=off (code path not taken). With kvm_hyperv=on, the two added privilege bits correspond to MSRs KVM already implements; this matches what QEMU advertises to every hv-*-enabled guest.

@sboeuf sboeuf left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but it would be good to have some Windows/Microsoft experts reviewing that behavior.

Comment thread arch/src/x86_64/mod.rs
Comment on lines +792 to +793
| (1 << 5) // AccessHypercallMsrs
| (1 << 6) // AccessVpIndex

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This breaks live migration and snapshot-restore. :-( We probably need some solution to that. A simple solution would be to introduce a hyperv_enlightenment_level integer in the state (default 0). Only include these at level >= 1 - start new VMs at level 1.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice catch!

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rbradford just curious, how do you know that? Can you explain the rationale behind the breakage?

@phip1611 phip1611 Jun 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thinking about it again as well: During a migration/restore, you upgrade to a new VMM that can simply do more if I'm not mistaken - therefore I think it should be fine. but you can't migrate back.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rbradford just curious, how do you know that? Can you explain the rationale behind the breakage?

It does say it in the PR summary :-)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

error is like

["Failed to receive migratable component snapshot","Error checking cpu feature compatibility': CpuidCheckCompatibility"]

these should be the errors from CH

@olivereanderson olivereanderson Jun 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tonicmuroq Do you have access to the logs? There should be a log at the error level describing what the differences are between the CPUID values.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might have to check the logs on the destination as I think that is where you will find the failed CPUID compatibility check.

Does the live migration work without the changes introduced here? It could just be that the CPUs are not compatible. The ongoing CPU profiles work we are doing (#7068 (comment)) might be of help if that is the issue.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@olivereanderson @rbradford my bad, just tested again, no errors. @olivereanderson was right, the reason i saw those errors was because i also introduced EAX bits 0/4 and max-leaf etc, it was not a clean fix, and final fix version should be this one.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, great!

@olivereanderson

Copy link
Copy Markdown
Contributor

@tonicmuroq I am very confused with regards to the state of your branch. The changes you introduce in this PR should be done in required_common_cpuid_updates and not in generate_common_cpuid, but your branch doesn't even seem to have the required_common_cpuid_updates function in arch/src/x86_64/mod.rs .

@tonicmuroq

Copy link
Copy Markdown
Contributor Author

@olivereanderson oh no the fork is far behind, this changes everything, let me recheck this

The Microsoft "Requirements for Implementing the Microsoft Hypervisor
Interface" document marks exactly two privileges in CPUID leaf
0x40000003 EAX as "Must be set": AccessHypercallMsrs (bit 5) and
AccessVpIndex (bit 6). Cloud Hypervisor advertised neither.

Without bit 5, Windows guests abort enlightened-mode initialization
before timer-API selection: HalpHvTimerApi is left NULL and every
QueryPerformanceCounter call falls back to reading
HV_X64_MSR_TIME_REF_COUNT (0x40000020), costing one VM exit per call.
The guest never writes HV_X64_MSR_REFERENCE_TSC (0x40000021) to enable
the reference TSC page, even though AccessPartitionReferenceTsc (bit 9)
is advertised.

With both bits set, Windows 10 22H2 and Windows 11 25H2 guests enable
the reference TSC page at boot. Measured QueryPerformanceCounter
throughput on a nested-KVM host went from ~71K calls/sec (14 us/call,
one MSR exit each) to ~1.3M calls/sec (free, no exits); on bare metal
from ~390K to ~1.9M calls/sec. Guest idle CPU and interrupt-service
time drop correspondingly.

Both MSR ranges are already handled in-kernel by KVM unconditionally,
so no backend change is needed. Bisection across the full delta to
QEMU's Hyper-V CPUID layout (vendor ID, max leaf, leaves 4-6 contents,
build number) shows bit 5 is the only load-bearing change; bit 6 is
included per the conformance document's mandate.

Signed-off-by: Tonic Li <[email protected]>
Signed-off-by: tonic <[email protected]>
@tonicmuroq
tonicmuroq force-pushed the fix/hyperv-must-set-privileges branch from fc44a91 to 45f3562 Compare June 15, 2026 10:41
@tonicmuroq

Copy link
Copy Markdown
Contributor Author

@olivereanderson rebased

@tonicmuroq
tonicmuroq requested a review from rbradford June 15, 2026 10:51

@rbradford rbradford left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@rbradford
rbradford enabled auto-merge June 15, 2026 13:48
@rbradford
rbradford added this pull request to the merge queue Jun 15, 2026
Merged via the queue into cloud-hypervisor:main with commit ce9416a Jun 15, 2026
41 checks passed
@CMGS
CMGS deleted the fix/hyperv-must-set-privileges branch June 16, 2026 02:41
@rbradford rbradford added the bug-fix Bug fix to include in release notes label Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-fix Bug fix to include in release notes

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

5 participants