Codestin Search App

rbradford · 2026-02-23T23:52:37Z

Add a core_scheduling option to --cpus with three modes of operation.
This feature takes advantage of a kernel feature that restricts scheduling
of processes on the SMT threads on the same core. This is useful for
mitigating certain classes of side-channel attacks and has better
performance that disabling SMT on the CPU.

vm (default): All vCPU threads share one core scheduling cookie.
They may be co-scheduled on SMT siblings while host threads are
excluded - this has minimal performance impact and can even
potentially improve performance from co-location.
vcpu: Each vCPU gets a unique cookie preventing any two vCPUs from
sharing SMT siblings. This has the strongest isolation but at some
compromise of performance.
off: No core scheduling applied (old behaviour).

In VM mode the cookie is created on the calling thread before spawning
vCPU threads so they inherit the same cookie. In vCPU mode each vCPU
thread the cookie is created when the thread starts.

EINVAL from prctl is silently ignored so this works transparently on
kernels older than 5.14 that lack PR_SCHED_CORE.

Full details of this kernel feature can be found at:
https://docs.kernel.org/admin-guide/hw-vuln/core-scheduling.html

This implementation was inspired by crosvm's implementation - in
particular the enable_core_scheduling() function.

Signed-off-by: Rob Bradford [email protected]

danielverkamp · 2026-02-24T00:09:20Z

@@ -1318,6 +1352,11 @@ impl CpuManager {
            self.vcpus_pause_signalled.load(Ordering::SeqCst)
        );

+        // Check the insertion flag to identify if this a hotplug or not.
+        if self.config.core_scheduling == CoreScheduling::Vm && !inserting {
+            enable_core_scheduling()?;


If the calling thread is used for other purposes after launching the vCPU threads (e.g. handling device emulation), it will still have the same core scheduling cookie as the vCPUs, which may or may not be desirable.

If it is undesirable, then probably spawning a temporary intermediate thread that enables PR_CORE_SCHED and spawns the vCPU threads would be the simplest way to avoid it.

Great point - it's not as simple as just starting an intermediate vCPU thread - since we also support vCPU hotplug. Let me look at the PR_SCHED_CORE_SHARE_TO as a way to make all the vCPU threads share a cookie.

parthy · 2026-02-24T12:21:42Z

Cool to see activity on this topic! We have been looking into core scheduling support for libvirt/CHV a while ago and actually started porting the qemu implementation in libvirt to the CHV driver. If you don't mind, I have a few questions with respect to this PR.

What is the rationale regarding other threads belonging to the same VM (virtio, API, ...)? Are they covered by the "Vm" mode? Or are they deemed not relevant? If at some point the core scheduling configuration should expand beyond just vCPUs, it might be awkward to have the setting in the CPU section. In the libvirt/qemu driver, there is a specific option to group the entire VMM with all its threads together, not just the vCPUs.
Looking at libvirt/qemu, applying the core scheduling configuration externally seems feasible. Was it a conscious choice to add this to CHV directly instead of relying on the component spawning CHV to take care of it?
Maybe related to the previous questions, is there a specific use case this PR is targeted at? Or is it just a useful feature to have?

rbradford · 2026-02-24T12:54:43Z

Cool to see activity on this topic! We have been looking into core scheduling support for libvirt/CHV a while ago and actually started porting the qemu implementation in libvirt to the CHV driver. If you don't mind, I have a few questions with respect to this PR.

What is the rationale regarding other threads belonging to the same VM (virtio, API, ...)? Are they covered by the "Vm" mode? Or are they deemed not relevant? If at some point the core scheduling configuration should expand beyond just vCPUs, it might be awkward to have the setting in the CPU section. In the libvirt/qemu driver, there is a specific option to group the entire VMM with all its threads together, not just the vCPUs.

Looking at libvirt/qemu, applying the core scheduling configuration externally seems feasible. Was it a conscious choice to add this to CHV directly instead of relying on the component spawning CHV to take care of it?

Maybe related to the previous questions, is there a specific use case this PR is targeted at? Or is it just a useful feature to have?

Since this is about the SMT security issue you wouldn't want a vCPU thread to be co-scheduled on the same core as e.g. a virtio-block thread as the latter is more privileged.

The reason to add it to CH rather than have the management layer do it is that this a good default security practice (that's why it's defaulted to VM) and it's straightforward to implement there with the cookies (no racing with the thread start times). This isn't about pinning threads for NUMA or performance reasons - I still believe that is best done with support from the management layer.

parthy · 2026-02-24T13:05:52Z

Since this is about the SMT security issue you wouldn't want a vCPU thread to be co-scheduled on the same core as e.g. a virtio-block thread as the latter is more privileged.

But isn't the virtio-* thread still belonging to the same VM and therefore dealing with the same "protection domain" (i.e., the single VM)? Assigning them to separate scheduling groups would protect the guest's vCPU from snooping on its own disk traffic, or am I getting this wrong?

Looking at it from the other side, doesn't leaving the virtio thread unassigned mean that other unrelated processes on the same host can theoretically run on a sibling core and snoop on the disk traffic?

rbradford · 2026-02-24T14:23:24Z

Since this is about the SMT security issue you wouldn't want a vCPU thread to be co-scheduled on the same core as e.g. a virtio-block thread as the latter is more privileged.

But isn't the virtio-* thread still belonging to the same VM and therefore dealing with the same "protection domain" (i.e., the single VM)? Assigning them to separate scheduling groups would protect the guest's vCPU from snooping on its own disk traffic, or am I getting this wrong?

I would argue that the e.g. virtio-block threads are more privileged than the the vCPU threads which are mostly running the guest code.

Looking at it from the other side, doesn't leaving the virtio thread unassigned mean that other unrelated processes on the same host can theoretically run on a sibling core and snoop on the disk traffic?

If those threads are untrusted VMM vCPU threads they won't ever be scheduled on the same core with other threads.

sboeuf · 2026-02-25T09:55:45Z

+        // EINVAL: kernel < 5.14 where PR_SCHED_CORE is unknown.
+        // ENODEV: CONFIG_SCHED_CORE is enabled but SMT is not present/enabled,
+        //         so core scheduling is not applicable.
+        // Both mean core scheduling is unavailable; silently ignore.


I'm not sure silently ignoring is the best approach here. If a user explicitly uses this new core_scheduling option, they have a reason and by ignoring the error here, they will be situation where the user think the core scheduling is applied when it's not.
I'd be in favor of throwing an error, but if you think this is too strong, at least let's print a warning.

It's turned on by default - my worry is that it might be excessive to throw a warning. The use probably knows that they're running a very old kernel version and are forced to do that for a reason.

Ah I missed the fact this was turned on by default. In that case I understand the reason for ignoring the error.
But maybe a debug!() or info!() would make sense.

The use probably knows that they're running a very old kernel version and are forced to do that for a reason.

That's not the only scenario where this can happen. What prompted #7335 was not me running a kernel too old for Landlock, but running a very new kernel where CONFIG_SECURITY_LANDLOCK had not been enabled by my distribution (Asahi Linux in this case). Core scheduling is also gated behind a config option, and users are unlikely to know off-hand which of the many thousands of optional kernel features they have enabled.

I think this should at least be an info log, but probably a warn. Users who expect it not to work because of their kernel can simply explicitly turn it off to silence the warning.

Added a warn! - thanks for all the thoughts on this guys,

Small nitpick: The comment still states that it's silently ignored.

And also excludes the case where CONFIG_SCHED_CORE is unset — in this case behaviour would be the same as <5.14 I think.

sboeuf · 2026-02-25T09:58:46Z

+    if ret == -1 {
+        let err = io::Error::last_os_error();
+        match err.raw_os_error() {
+            Some(libc::EINVAL) | Some(libc::ENODEV) => {}


sboeuf · 2026-02-25T10:04:46Z

@@ -1117,6 +1191,64 @@ impl CpuManager {
                        }
                    }

+                    // Set up core scheduling before seccomp locks down prctl.


As of curiosity, what happens if the CPU affinity set up above conflicts with the core_scheduling? I guess Linux notices and throws an error when we invoke prctl(PR_SCHED_CORE, ...)?

I don't think a conflict per se can occur here, because the two mechanisms are very different (affinity is a static restriction, core scheduling is a dynamic property that influences scheduler decisions based on what threads are in the system). The worst case I can think of is affinity settings pinning different groups to sibling cores, which would result in vCPUs running in an alternating schedule rather than in parallel.

Ah good point. Let's say you set the affinity to be CPUs [0-1] (being 2 threads on the same core) for both vCPU0 and vCPU1 (on an hypothetical 2 vCPUs VM), that means with a strict core scheduling applied, you end up scheduling one vCPU while the other is idle waiting to be scheduled. This would kill parallelism, but as you said this wouldn't be seen as a conflict and therefore wouldn't be caught from the kernel. Thanks for the clarification!

That's at least my understanding. If a thread is to be scheduled alongside an untrusted thread, that second one would be evicted and either replaced by another trusted one (even lower priority) or the idle thread in the worst case:

The scheduler tries its best to find tasks that trust each other such that all tasks selected to be scheduled are of the highest priority in a core. However, it is possible that some runqueues had tasks that were incompatible with the highest priority ones in the core. Favoring security over fairness, one or more siblings could be forced to select a lower priority task if the highest priority task is not trusted with respect to the core wide highest priority task. If a sibling does not have a trusted task to run, it will be forced idle by the scheduler (idle thread is scheduled to run).

parthy · 2026-02-25T12:26:38Z

I would argue that the e.g. virtio-block threads are more privileged than the the vCPU threads which are mostly running the guest code.

I always thought of the core scheduling to be a question of a trust gradient rather than a privilege gradient. That's also how the kernel documentation describes it:

As mentioned in Usage, tasks with the same cookie value are assumed to trust each other and share a core.

Are you saying that the vCPU thread does not trust its VM's virtio-block thread? Or what is the scenario which is prevented by having them in different groups?

If those threads are untrusted VMM vCPU threads they won't ever be scheduled on the same core with other threads.

I'm interested in the protection of the virtio-block thread against other untrusted threads in the system. Right now, the virtio-block thread would have no group assigned, is that correct? So it could run on a sibling core with any other untrusted thread, potentially exposing it to side-channel attacks.

I don't want to blow this discussion up unnecessarily, just sharpen my understanding of the feature and how it's meant to be implemented in here. If this discussion is better held somewhere else, I'm very open to do that ;)

rbradford · 2026-02-25T18:14:40Z

I would argue that the e.g. virtio-block threads are more privileged than the the vCPU threads which are mostly running the guest code.

I always thought of the core scheduling to be a question of a trust gradient rather than a privilege gradient. That's also how the kernel documentation describes it:

As mentioned in Usage, tasks with the same cookie value are assumed to trust each other and share a core.

Are you saying that the vCPU thread does not trust its VM's virtio-block thread? Or what is the scenario which is prevented by having them in different groups?

No, it's the other way around, we don't trust the vCPU thread, running user controlled code, to run as a sibling with the virtio-block thread (or any other system thread) which is running VMM controlled code. If you started lots of VMs with just 1 vCPU assigned then you might not be fully utilising your hardware as those cores will only run the one vCPU thread.

If those threads are untrusted VMM vCPU threads they won't ever be scheduled on the same core with other threads.

I'm interested in the protection of the virtio-block thread against other untrusted threads in the system. Right now, the virtio-block thread would have no group assigned, is that correct? So it could run on a sibling core with any other untrusted thread, potentially exposing it to side-channel attacks.

I you don't trust the virtio-block thread in your VMM - you could also consider using affinity to group all the processes used by the VMM into a single set of cores. Or use core scheduling at a higher level to group all the VMM threads together.

Add a core_scheduling option to --cpus with three modes of operation. This feature takes advantage of a kernel feature that restricts scheduling of processes on the SMT threads on the same core. This is useful for mitigating certain classes of side-channel attacks and has better performance that disabling SMT on the CPU. - vm (default): All vCPU threads share one core scheduling cookie. They may be co-scheduled on SMT siblings while host threads are excluded - this has minimal performance impact and can even potentially improve performance from co-location. - vcpu: Each vCPU gets a unique cookie preventing any two vCPUs from sharing SMT siblings. This has the strongest isolation but at some compromise of performance. - off: No core scheduling applied (old behaviour). This isolation is done by the kernel maintaining a "cookie" - threads with the same cookie can share the same core. In vCPU mode each vCPU thread the cookie is created when the thread starts and each gets a unique cookie. For VM mode the first vCPU thread (the leader) will create the cookie. All other vCPU threads started (via hotplug or during boot) will have that cookie shared to it. EINVAL/ENODEV from prctl is silently ignored so this works transparently on kernels older than 5.14 that lack PR_SCHED_CORE or when SMT disabled. Full details of this kernel feature can be found at: https://docs.kernel.org/admin-guide/hw-vuln/core-scheduling.html This implementation was inspired by crosvm's implementation - in particular the enable_core_scheduling() function. This is challenging to test via integration testing but the logging of the received cookie shows it working: VM case: cloud-hypervisor: 0.243102s: <vcpu1> INFO:vmm/src/cpu.rs:1247 -- vCPU 1: core scheduling cookie = 0x33e4c167 cloud-hypervisor: 0.243102s: <vcpu0> INFO:vmm/src/cpu.rs:1247 -- vCPU 0: core scheduling cookie = 0x33e4c167 vCPU case: cloud-hypervisor: 0.089356s: <vcpu0> INFO:vmm/src/cpu.rs:1247 -- vCPU 0: core scheduling cookie = 0x13993ad6 cloud-hypervisor: 0.089380s: <vcpu1> INFO:vmm/src/cpu.rs:1247 -- vCPU 1: core scheduling cookie = 0xd48e86e Signed-off-by: Rob Bradford <[email protected]>

parthy · 2026-02-26T08:41:04Z

I think I see the difference in thinking about this now: My mental model about assigning a group to a thread was to protect it from other threads in the system by making sure it only runs alongside other threads it can trust. With your explanation, it sounds more like "exiling" threads you don't trust (because it's running user-controlled code). I guess both aspects have their place and feed into the final decision what groups are assigned. Thank you for the detailed explanations!

Would it make sense to add some more documentation around this rationale or is it mostly obvious to everyone?

rbradford requested a review from a team as a code owner February 23, 2026 23:52

rbradford force-pushed the 202602/core-scheduling branch 3 times, most recently from abd9c17 to 5fa2003 Compare February 23, 2026 23:57

danielverkamp reviewed Feb 24, 2026

View reviewed changes

rbradford force-pushed the 202602/core-scheduling branch from 5fa2003 to 09a7728 Compare February 24, 2026 00:11

rbradford marked this pull request as draft February 24, 2026 00:22

rbradford force-pushed the 202602/core-scheduling branch from 09a7728 to 2051270 Compare February 24, 2026 15:52

sboeuf reviewed Feb 25, 2026

View reviewed changes

rbradford force-pushed the 202602/core-scheduling branch from 2051270 to 5f5bb4f Compare February 26, 2026 08:33

rbradford marked this pull request as ready for review February 26, 2026 08:33

sboeuf approved these changes Feb 26, 2026

View reviewed changes

rbradford added this pull request to the merge queue Feb 26, 2026

Merged via the queue into cloud-hypervisor:main with commit 3f800d2 Feb 26, 2026
35 of 42 checks passed

rbradford moved this to ✅ Done in Cloud Hypervisor Roadmap Apr 25, 2026

rbradford added this to Cloud Hypervisor Roadmap Apr 25, 2026

rbradford deleted the 202602/core-scheduling branch June 12, 2026 10:36

Uh oh!

Conversation

rbradford commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rbradford Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

parthy commented Feb 24, 2026

Uh oh!

rbradford commented Feb 24, 2026

Uh oh!

parthy commented Feb 24, 2026

Uh oh!

rbradford commented Feb 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

parthy commented Feb 25, 2026

Uh oh!

rbradford commented Feb 25, 2026

Uh oh!

parthy commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rbradford commented Feb 23, 2026 •

edited

Loading

rbradford Feb 24, 2026 •

edited

Loading

parthy commented Feb 26, 2026 •

edited

Loading