Scripts for running GHA runners in ephemeral firecracker micro VMs. Ephemeral VMs are preferable because there is no cleanup to be done after a job run, hence there is also no risk of accidentially leaking data between job runs.
- The VM image is built from an ordinary docker file
- After each job run, the VM simply reboots, which will halt it's execution
systemdwill take care of restarting VMs immediatly, boot time is less than one second- Because VMs disk drive is a
tmpfsresiding entirely in RAM, no state is persistet between runs. Alternatively, e.g. if not enough memory is available on the VM host, this could also be made a COW which is thrown away after each run instead, with minor changes. - Isolation from the host is done with
jailer. - One VM host can run multiple microVMs at the same time. However each runner needs it's own GHA runner token.
dockerqemu-img- squashfs-tools
sudo- Unpriviledged user for jailer
- Download the latest runner package:
curl -O -L https://github.com/actions/runner/releases/download/v2.303.0/actions-runner-linux-x64-2.303.0.tar.gz - Obtain a firecracker supported kernel (5.10) and save it as
vmlinux.bin - Build the root filesystem:
./build-root.sh token root_passwd runner-name-X && vm rootfs.img rootfsX.img(replace X with a number) - Repeat the previous step for as many runners you want
- Change
gha.serviceto use your network interface (defaultenp0s3), data and script location (default/root/gha) and workdir (default/srv/jailer/firecracker) - Change
config.jsonto desired VM configuration. Do not changenetwork-interfacessections or it will break the network setup withfcnet.service. Do not change thedrivessection or it will break automatic setupgha.service. - Enable the systemd services
gha.serviceandfcnet.service
These would be nice to fix for good but I ENOTIME:
- Netfilter (NFT) doesn't work for some reason, but docker needs it -> switched to legacy iptables (ugly)
- Network configuration is an ugly hack
- DNS inside VM only works via TCP
- Uses the undocummented
--onceflag with the GHA runner binary but it is apparently depricated and removed in the future. I'm not sure how else we can make this work, since--ephemeralwill delete the runnner after one job and a new token needs to be obtained somehow? The GHA docs don't loose a word to this problem.