This was originally cilium/ebpf#1734

vimto uses virtio-console to send control messages from the host to the init process in the guest, and to return stdout / stderr of the guest to the host. In ebpf-go we sporadically see unit tests getting stuck writing to that stdout. I've managed to reduce the problem to the following:

taskset -c 0 vimto -smp cpus=2 -kernel ghcr.io/cilium/ci-kernels:5.15 exec -- sh -c 'seq 1 100000 | while read i; do echo "line $i"; done'

• The qemu process is pinned to a single CPU
• The VM has two vCPU
• Each individual write has a chance of getting stuck, so we need lots of writes

Inside the VM, the kernel is stuck in:

[<0>] wait_port_writable+0x139/0x2d0
[<0>] port_fops_write+0x88/0x130
[<0>] vfs_write+0xf3/0x450
[<0>] ksys_write+0x6d/0xe0
[<0>] do_syscall_64+0x9e/0x1a0
[<0>] entry_SYSCALL_64_after_hwframe+0x77/0x7f
1 0x1 0x7ffe92ebda50 0xb 0x0 0x0 0x0 0x7ffe92ebd9f0 0x7f984861577e

On more recent kernels (seems like 6.1 and newer) it's possible to work around this by setting ioeventfd=off. Unfortunately this then breaks older kernels, but in a different spot. Instead of getting stuck while writing to stdout, the init process hangs while reading the control message from the host:

[    1.312132] Run /home/runner/go/bin/vimto as init process
Error: read command: read /dev/vport2p1: i/o timeout
[    3.144910] ACPI: Preparing to enter system sleep state S5