vmm: raise the (v)CPU limit on kvm/x86_64 - #7299

Note: I have a draft of several new itegration tests here: posk-io@0a250e1, but they fail when run from a VM (i.e. they fail making a nested VM with >= 254 vCPUs). I don't have a spare baremetal host to run the tests at the moment.

Another reason could be that the kernel in the VM image that is used to run the tests has a smaller CONFIG_NR_CPUS. Anyway, manual testing indicates the PR/patch series works on AMD and Intel hosts with up to 1024 vCPUs (the limit that the host KVM imposes in my tests).

Follow-up of #7299 (comment): I just noticed that using > 512 vCPUs on my AMD Ryzen 7 7840U w/ Radeon 780M Graphics causes:

Error: Cloud Hypervisor exited with the following chain of errors:
  0: Error booting VM
  1: The VM could not boot
  2: Error from CPU manager
  3: Error creating vCPU
  4: Failed to create Vcpu
  5: Invalid argument (os error 22)

This surprises me. I expected that at least 4096 vCPUs should be supported. Aye, @tpressure?

General remark: we should not generate an MPTable if we have more than 255 vCPUs but solely rely on the MADT.

I'm also missing the part where you expose KVM_FEATURE_MSI_EXT_DEST_ID . Is this missing on purpose?

Follow-up of #7299 (comment): I just noticed that using > 512 vCPUs on my AMD Ryzen 7 7840U w/ Radeon 780M Graphics causes:

Error: Cloud Hypervisor exited with the following chain of errors:
  0: Error booting VM
  1: The VM could not boot
  2: Error from CPU manager
  3: Error creating vCPU
  4: Failed to create Vcpu
  5: Invalid argument (os error 22)

This surprises me. I expected that at least 4096 vCPUs should be supported. Aye, @tpressure?

On my (Intel) system, it works fine with 768 vCPUs. I will check with my AMD system tomorrow (evening). Did you make sure to increase your ulimit correctly? I'm using ulimit -n 10000.

I'm also missing the part where you expose KVM_FEATURE_MSI_EXT_DEST_ID . Is this missing on purpose?

This was done in a previous PR:

General remark: we should not generate an MPTable if we have more than 255 vCPUs but solely rely on the MADT.

This was done in a previous PR:

Update: I was able to boot VMs with 1024 vCPUs on both Intel and AMD hosts. For some reason max KVM host limit is 1024 on machines I have access to. Does anybody know if this is tunable?

I'll lower the max vCPU limit in the PR to 8192, as this seems to be the Linux kernel's max, and do the cosmetic fixes suggested tomorrow. Will also experiment with the integration test (although it may be flaky, as it is unclear how the kvm host nr_cpu limit is determined).

Update: I was able to boot VMs with 1024 vCPUs on both Intel and AMD hosts. For some reason max KVM host limit is 1024 on machines I have access to. Does anybody know if this is tunable?

I'll lower the max vCPU limit in the PR to 8192, as this seems to be the Linux kernel's max, and do the cosmetic fixes suggested tomorrow. Will also experiment with the integration test (although it may be flaky, as it is unclear how the kvm host nr_cpu limit is determined).

kvm has a KVM_MAX_VCPUS macro which can be defined by CONFIG_KVM_MAX_NR_VCPUS, so the best method I suppose is not setting a max value in cloud hypervisor, instead, it's better to query the value from kvm.

For example, qemu did like this:
https://github.com/qemu/qemu/blob/master/accel/kvm/kvm-all.c#L2692

Update: I was able to boot VMs with 1024 vCPUs on both Intel and AMD hosts. For some reason max KVM host limit is 1024 on machines I have access to. Does anybody know if this is tunable?
I'll lower the max vCPU limit in the PR to 8192, as this seems to be the Linux kernel's max, and do the cosmetic fixes suggested tomorrow. Will also experiment with the integration test (although it may be flaky, as it is unclear how the kvm host nr_cpu limit is determined).

kvm has a KVM_MAX_VCPUS macro which can be defined by CONFIG_KVM_MAX_NR_VCPUS, so the best method I suppose is not setting a max value in cloud hypervisor, instead, it's better to query the value from kvm.

For example, qemu did like this: https://github.com/qemu/qemu/blob/master/accel/kvm/kvm-all.c#L2692

The current implementation already checks with KVM_MAX_VCPUS - see Error::MaximumVcpusExceeded. We kind of need a constant limit (say MAX_SUPPORTED_CPUS) to clarify the different limits on different architectures.

Will also experiment with the integration test (although it may be flaky, as it is unclear how the kvm host nr_cpu limit is determined).

That would be great to have. The challenge is that the most majority of our integration tests run on fairly small Azure VMs (8/16 vCPUs), and I'd assume that won't serve the purpose for launching a VM with 255+ vCPUs on top of it. We do have a couple baremetal x86_64 self-host runners, and we may can experiment with these machines. Not a blocker for landing this PR.

@posk-io I see we are very close to conclude on this series with landing this PR (just two small comments above). It would be good to include the increased vCPU limits in our next release (scheduled next Thursday). Thanks.

@posk-io I see we are very close to conclude on this series with landing this PR (just two small comments above). It would be good to include the increased vCPU limits in our next release (scheduled next Thursday). Thanks.

Ack. Give me a couple of hours (traveling - not in my usual habitat).

Very nit but please strip the Intel from the PR name, as this works for Intel and AMD. This will help developers in the future to look through old PRs without getting confused

Very nit but please strip the Intel from the PR name, as this works for Intel and AMD. This will help developers in the future to look through old PRs without getting confused

Done.