BoxLite bench methodology

Result

BoxLite N=100 spawn measured at 113,209 ms on the same dev box forkd was measured on (Ubuntu 24.04 / Linux 6.14 / 20 vCPU / 30 GiB / KVM). 0 of 100 sandboxes succeeded the workload (import numpy) because python:3.12-slim doesn't ship numpy; this matches the condition Docker, gVisor, and OpenSandbox were measured under, so the wall-clock figure is comparable across container-shaped runners.

Setup

python3.13 -m pip install --user boxlite==0.9.3

(BoxLite ships a manylinux wheel — no separate boxlite-cli / cargo install step is needed for the Python SDK path. The optional Rust CLI boxlite-cli requires protoc and a Rust toolchain; we don't depend on it for benchmarking.)

Workload

bench/boxlite-bench.py (also wired into compare-all.py) issues N concurrent boxlite.SimpleBox(image="python:3.12-slim") contexts via asyncio.gather, runs python -c "import numpy; print(numpy.zeros(5).tolist())" inside each, and tears down the Box on context exit.

import asyncio, boxlite

async def one(_i):
    async with boxlite.SimpleBox(image="python:3.12-slim") as box:
        r = await box.exec("python", "-c",
            "import numpy; print(numpy.zeros(5).tolist())")
        return getattr(r, "exit_code", 0) == 0

results = asyncio.run(asyncio.gather(*(one(i) for i in range(100))))

Scaling profile observed

N	total wall-clock	per-box mean
3	5,581 ms	1,860 ms
10	7,339 ms	734 ms
50	23,355 ms	467 ms
100	113,209 ms	1,132 ms

Per-box mean improves from N=3 to N=50 as parallelism amortises overhead, then degrades at N=100 as the host saturates on disk I/O (BoxLite reports console.log errors mentioning "Slow disk I/O during rootfs setup" and "Guest agent failed to start" for some boxes at N=100).

Notes

BoxLite's positioning ("the SQLite of sandbox" — embeddable, no daemon, cross-platform via Hypervisor.framework on macOS) is optimised for one long-lived Box per workload rather than N concurrent fresh Boxes. Cold-spawning 100 microVMs concurrently on a single 30-GiB host is not the workload BoxLite was designed for; the measurement is included here for comparability with the other runners on the same task, not as a critique of the design.