Description

Follow the Guide, build PVM host kernel and PVM guest kernel, start VM via Cloud Hypervisor, I can boot VM with default configurations, but if I change the rootfs into pmem, system boot failed.
There are such failed messages.

[ 0.994008] Run /sbin/init as init process
[ 1.001957] Starting init: /sbin/init exists but couldn't execute it (error -14)
[ 1.003975] Run /etc/init as init process
[ 1.005183] EXT4-fs error (device pmem0p1): __ext4_find_entry:1684: inode torvalds#41: comm init: reading directory lblock 0
[ 1.008044] Starting init: /etc/init exists but couldn't execute it (error -5)
[ 1.010054] Run /bin/init as init process
[ 1.011261] EXT4-fs warning (device pmem0p1): dx_probe:822: inode #1553: lblock 0: comm init: error -5 reading directory block
[ 1.014336] Starting init: /bin/init exists but couldn't execute it (error -5)
[ 1.016247] Run /bin/sh as init process
[ 1.017334] EXT4-fs warning (device pmem0p1): dx_probe:822: inode #1553: lblock 0: comm init: error -5 reading directory block
[ 1.020439] Starting init: /bin/sh exists but couldn't execute it (error -5)
[ 1.022364] Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
[ 1.026248] CPU: 0 PID: 1 Comm: init Not tainted 6.7.0-rc6-virt-pvm-guest+ torvalds#55
[ 1.028268] Hardware name: Cloud Hypervisor cloud-hypervisor, BIOS 0
[ 1.030055] Call Trace:
[ 1.030879]
[ 1.031556] dump_stack_lvl+0x43/0x60
[ 1.032689] panic+0x2b2/0x2d0
[ 1.033597] ? rest_init+0xc0/0xc0
[ 1.034551] kernel_init+0x112/0x120
[ 1.035626] ret_from_fork+0x2b/0x40
[ 1.036606] ? rest_init+0xc0/0xc0
[ 1.037537] ret_from_fork_asm+0x11/0x20
[ 1.038627]
[ 1.039490] Kernel Offset: disabled
[ 1.040556] ---[ end Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. ]---

Step to reproduce

1. Build PVM host kernel and PVM guest kernel
   Following the guide pvm-get-started-with-kata.md

2. Guest VM resource from Guide
   cloud-hypervisor v37
   VM image from Guide

3. Start VM and Create snapshot
   #Start VM failed with PMEM rootfs
cloud-hypervisor.v37 \
--api-socket ch.sock \
--log-file vmm.log \
--cpus boot=1 \
--kernel vmlinux.virt-pvm-guest \
--cmdline 'console=ttyS0 root=/dev/pmem0p1 rw clocksource=kvm-clock pti=off' \
--memory size=1G,hugepages=off,shared=false,prefault=off \
--pmem id=pmem_0,discard_writes=on,file=ubuntu-22.04-pvm-kata.raw \
-v --console off --serial tty

   #Start VM success with Virtio Block rootfs
cloud-hypervisor.v37 \
--api-socket ch.sock \
--log-file vmm.log \
--cpus boot=1 \
--kernel vmlinux.virt-pvm-guest \
--cmdline 'console=ttyS0 root=/dev/vda1 rw clocksource=kvm-clock pti=off' \
--memory size=1G,hugepages=off,shared=false,prefault=off \
--disk id=disk_0,path=ubuntu-22.04-pvm-kata.raw \
-v --console off --serial tty

4. Configurations to work around the issue
   After some investigation, I found the issue caused by the MMIO address space provided by Cloud Hypervisor, virtio-pmem device is using MMIO address, which is at the end of physical address space
   And this will exceed PVM guest virtual default address space, then trigger some unexpected fails.
   So after I change CPU configuration into --cpus boot=1,max_phys_bits=43, I can boot the VM with rootfs on virto-pmem.