Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Tags: lmb/vimto

Tags

v0.4.0

Toggle v0.4.0's commit message
init: work around the mother of all race conditions

ebpf-go CI has been plagued by sporadic hangs, where tests simply
time out while trying to write status information to stdout.

The bug manifests when issuing blocking writes to a virtio console
while also polling it. The way we trigger the bug is quite involved:

- init opens tha port via os.OpenFile. This sets O_NONBLOCK on the
  fd, and registers the os.File with the poller.
- The port is passed to the child process via exec.Cmd.Stdout.
  This internally calls os.File.Fd(), which clears O_NONBLOCK
  but doesn't remove the file from the poller.
- The child process receives a blocking stdout. Writing to it
  will issue a blocking write to the virtio-console port,
  specifically port_fops_write() in virtio_console.c.
- port_fops_write() calls wait_port_writable(). This puts the calling
  thread to sleep if the virtqueue is full, by waiting
  on port->waitqueue.

We now enter the race window.

- The host processes the guest's write, frees up some space in the
  virtqueue and issues an interrupt to the guest.
- This interrupt races with a call to port_fops_poll() issued by
  the init process's Go runtime. That function invokes
  will_write_block(), which consumes all used buffers from
  the virtqueue.
- The interrupt handler vring_interrupt() checks whether the
  virtqueue has any unused buffers via more_used().
  Since all buffers have just been consumed by port_fops_poll()
  the interrupt is dropped.

At this point we still have a writer stuck in port_fops_write()
waiting for a wakeup that never comes.

The workaround for this issue is to close the stdio file in init,
thereby removing it from the runtime poller.

Fixes: #29

v0.3.1

Toggle v0.3.1's commit message
Revert "work around race in virtio serial console"

This reverts commit 475850d.

It seems that kernels < 6.1 suffer from a separate lost wakeup
problem which manifests in reads of the control message timing
out:

    [    1.312132] Run /home/runner/go/bin/vimto as init process
    Error: read command: read /dev/vport2p1: i/o timeout
    [    3.144910] ACPI: Preparing to enter system sleep state S5

v0.3.0

Toggle v0.3.0's commit message
work around race in virtio serial console

The ebpf-go CI has been plagued by a non-deterministic hang of
unit tests. It affects all packages and manifests as a write to
stdout getting stuck, followed by the test timing out. This
triggers a goroutine dump, which in turn unblocks the stuck write
to stdout.

Its possible to reproduce this behaviour using the following
commandline:

    taskset -c 0 vimto -smp cpus=2 -kernel ghcr.io/cilium/ci-kernels:6.15.3 \
      exec -- sh -c 'seq 1 1000000 | while read i; do echo "line $i"; done'

After a few seconds the output will freeze. Inspecting the stack of
the executing program shows something like the following:

    [<0>] wait_port_writable+0x139/0x2d0
    [<0>] port_fops_write+0x88/0x130
    [<0>] vfs_write+0xf3/0x450
    [<0>] ksys_write+0x6d/0xe0
    [<0>] do_syscall_64+0x9e/0x1a0
    [<0>] entry_SYSCALL_64_after_hwframe+0x77/0x7f
    1 0x1 0x7ffdf4878c80 0x9 0x0 0x0 0x0 0x7ffdf4878c20 0x7f592daed77e

As far as I can tell it is critical that execution is restricted to
a single CPU on the host side, while qemu presents two vCPU to the VM.

Passing ioeventfd=off to the serial console device works around
this problem.

See cilium/ebpf#1734 for more details.

v0.2.0

Toggle v0.2.0's commit message
document restrictions in readme

v0.1.0

Toggle v0.1.0's commit message
don't extract image for each script test

The script test runner sets a new TMPDIR for each script. This changes
the location of the cache. Hence every invocation of vimto does a new
fetch, which adds up to a bit of time.

Add an explicit test for OCI behaviour and reuse the cached kernel in
other tests.