Codestin Search App

maxpain · 2026-06-03T06:19:37Z

Problem

In Socket serial mode (--serial socket=...), Cloud Hypervisor only installs the
serial device's output sink when a client connects. Output produced before that —
kernel boot messages, cloud-init, etc. — is dropped, so a console client that
attaches after boot (e.g. a web serial console in an IaaS) sees a blank screen
with no prior output. Only PTY mode buffers output (via SerialBuffer). See #7907.

Fix

Reuse the existing SerialBuffer (1 MiB ring) for Socket mode:

Install a persistent SerialBuffer as the Socket device's output sink at
SerialManager construction, discarding downstream (io::sink()) until a
client connects — so output is captured into the ring even with no client.
On connect: retarget the buffer at the accepted client and flush the backlog
before live output resumes.
On disconnect: keep the ring, so the next client still gets the history.
The accepted socket is made non-blocking via fcntl (not
UnixStream::set_nonblocking, whose ioctl(FIONBIO) is not in the
serial-manager seccomp filter) so a slow/stalled client can't stall the vCPU
thread — SerialBuffer re-buffers on WouldBlock.
Allow sendto in the serial-manager seccomp filter: replaying the backlog is
the first time that thread writes to the socket.

Two commits: the serial_buffer API addition (set_out + unit tests), then the
vmm wiring. PTY mode is unchanged.

Verification

Built --features kvm, run with seccomp on (default). Boot a guest with
--serial socket=... and no client attached; once it is past boot and idle,
connect a late client:

	late client receives
before	0 bytes (boot output dropped)
after	~29 KB — full boot log replayed (`Linux version …` → `Kernel panic … VFS: Unable to mount root fs`)

Also verified: connecting during boot streams live output (vCPU write-through
works under seccomp); reconnect after disconnect still replays retained history
(unit test); --serial pty still boots; cargo build / rustfmt / clippy are
clean and the serial_buffer unit tests pass.

This is the first of two PRs for the socket-console work; #7974 (console persists
across guest reboot) will build on this persistent buffer.

Fixes #7907

🤖 Prepared with Claude Code; all changes were reviewed and verified by the author (see the Assisted-by: commit trailers).

phip1611

Generally I think this is a good think! I'm not an expert of the serial_manager code but I think we should make some refactorings here to ensure maintainability. For example, a enum SerialManagerInner could carry the information for the output sync. We should get rid of dozens of Option<T> - this is bad API design and increases cognitive complexity and decreases maintainability

phip1611 · 2026-06-03T10:08:28Z

        (libc::SYS_recvfrom, vec![]),
        (libc::SYS_rt_sigprocmask, vec![]),
        (libc::SYS_rt_sigreturn, vec![]),
+        // Replaying the buffered backlog to a freshly-connected client is the


I think this rather verbose comment doesn't belong here - especially as also all other syscalls do not have comment explaining their context

Removed — sendto is now bare like the surrounding syscalls.

phip1611 · 2026-06-03T10:12:46Z

        }))
    }

+    // Put a file descriptor into non-blocking mode using fcntl(). We avoid


excessive comment which should not be merged like this.

non valid rust doc

not minimal

Put a file descriptor into non-blocking mode using fcntl(). is replicating what the code does without much value add

which is not permitted by the SerialManager -> this is likely to be outdated eventually - further you can change the seccomp filter (which you even did!)

Trimmed to just the non-obvious part: UnixStream::set_nonblocking() would issue ioctl(FIONBIO), which this thread's seccomp filter rejects, so we use fcntl instead. Dropped the rest.

arctic-alpaca · 2026-06-03T10:26:36Z

+    fn set_fd_nonblocking(fd: RawFd) -> Result<()> {
+        // SAFETY: FFI calls with a valid fd.
+        let ret = unsafe {
+            let flags = libc::fcntl(fd, libc::F_GETFL);
+            libc::fcntl(fd, libc::F_SETFL, flags | libc::O_NONBLOCK)
+        };
+        if ret < 0 {
+            return Err(Error::SetNonBlocking(std::io::Error::last_os_error()));
+        }
+        Ok(())
+    }


A RawFd doesn't guarantee a valid fd, the SAFETY comment is wrong.

Fixed — the helper now takes BorrowedFd<'_> instead of RawFd, so the fd is guaranteed live for the duration of the call and the SAFETY comment is actually true.

maxpain · 2026-06-03T11:10:49Z

Thanks both — pushed a revision addressing the feedback:

Bundle the socket Options (@phip1611): socket_path, the ring buffer and the write-through flag are now a single SocketConsole struct (Some only in socket mode), with attach_client/detach_client methods, so a subset can't be set without the rest. I kept it a struct rather than a full SerialManagerInner enum because transport already carries the PTY/Socket/Tty discriminant and a second enum would duplicate it — happy to go further if you'd rather.
SAFETY on the non-blocking helper (@arctic-alpaca): you're right, a RawFd doesn't prove validity. The helper now takes BorrowedFd<'_>, so the fd is guaranteed live for the call and the SAFETY note is actually true.
Excessive comments (@phip1611): trimmed the fcntl helper to just the non-obvious bit (why not set_nonblocking() — it issues ioctl(FIONBIO), which this thread's seccomp filter rejects) and dropped the sendto comment in the filter.

While re-verifying I also fixed two of my own inaccuracies: the set_out doc and a test name implied full-history replay, but flush() drains the ring (each byte delivered once), so I reworded them and added a test pinning that already-drained bytes are not resent. Verified end-to-end against the CLOUDHV firmware: a client connecting after boot gets the buffered banner, including under --seccomp true (the sendto replay path), and the socket file is removed on exit.

phip1611 · 2026-06-03T12:12:24Z

    pty_write_out: Option<Arc<AtomicBool>>,
-    socket_path: Option<PathBuf>,
+    // `Some` only in socket mode; `None` otherwise.
+    socket_console: Option<SocketConsole>,


This should be split into two commits for better reviewability.

new struct SocketConsole

add the new members

Done — split into the feature commit (socket output buffering) and a separate commit that bundles the state into SocketConsole. I put the feature first because a struct-only commit would be unused (dead code) until it's wired in, so this way every commit builds on its own. Three commits total, including the serial_buffer: set_out() change.

rbradford · 2026-06-03T12:22:08Z

Cool! I think it would be worth thinking about if we could also get socket support for virtio-console too? And make sure we don't preclude that in this design.

maxpain · 2026-06-03T12:46:16Z

Good point — the design leans into that rather than precluding it. The reusable piece is SerialBuffer::set_out() in the serial_buffer crate: it retargets the downstream writer while keeping the ring, which is exactly what a socket needs (point at the accepted client + flush on connect, point at a sink on disconnect). It's device-agnostic and already shared — virtio-console even builds a SerialBuffer itself today for PTY (virtio-devices/src/console.rs), with the same out: Option<Box<dyn Write + Send>> + write_out: Option<Arc<AtomicBool>> shape as the serial path.

So virtio-console-over-socket can reuse the same buffer-while-detached / retarget-and-replay-on-connect pattern. What's serial-specific here is the accept/epoll glue in serial_manager (the SocketConsole bundle) — virtio-console has its own output-queue thread, so it'd need its own accept/retarget wiring, but the buffering core transfers unchanged and nothing here bakes UART assumptions into the buffer. Happy to factor that glue into a shared helper when your virtio-console socket work lands.

rbradford

I think the code is good but the comments are too much. This is to be expected from an AI generated code (I spend a lot of time cleaning them up in the PRs I produce). I don't need the comment to tell me what the code is doing - I can see that. They should be limited to what is non-obvious.

rbradford · 2026-06-03T12:50:29Z

+    /// Replace the downstream writer without discarding buffered bytes.
+    ///
+    /// On connect the buffer is pointed at the new client (its backlog is then
+    /// flushed to it); on disconnect it is pointed at a discarding sink. Either
+    /// way the buffered bytes are left in place, so output produced while no
+    /// client is attached is preserved for the next client to connect.


I don't think this comment is helpful - I feel like this is not reflecting what this method does and instead reflects some other previous behaviour. These kind of comments are unfortunately typical of AI generated code. It's always best to check them and ask it to make sure the comments reflect the code as it currently stands.

rbradford · 2026-06-03T12:50:57Z

    handle: Option<thread::JoinHandle<()>>,
    pty_write_out: Option<Arc<AtomicBool>>,
-    socket_path: Option<PathBuf>,
+    // `Some` only in socket mode; `None` otherwise.


I think this is self evident from the name.

rbradford · 2026-06-03T12:51:19Z

+}
+
+/// A [`Write`] handle to a shared [`SerialBuffer`], so the serial device (which
+/// owns its `out` sink) and the serial-manager thread (which retargets the


rbradford · 2026-06-03T12:51:57Z

+/// Socket-console plumbing, present only in `ConsoleTransport::Socket` mode.
+/// Bundled so the fields move together: either all are set or none is, rather
+/// than separate `Option`s that could end up disagreeing.


Again - this is a comment reflecting what has changed vs what the code is right now.

rbradford · 2026-06-03T12:52:59Z

+    /// connected and replays it on connect. Shared with the serial device's
+    /// `out` sink and retargeted by the epoll thread.
+    buffer: Arc<Mutex<SerialBuffer>>,
+    /// Gates write-through to the connected client; `false` while detached.


I think the comment for buffer is good - the rest just look like AI added fluff.

maxpain · 2026-06-03T13:15:03Z

Thanks — trimmed all of these to the minimum and pushed. Removed the socket_console field and SocketConsole struct/field comments (self-evident), cut the set_out and SharedSerialBuffer docs to one line each, and kept only the buffer comment you flagged as good.

rbradford · 2026-06-03T14:35:55Z

+    /// Attach a freshly-accepted client: point the buffer at it, enable
+    /// write-through, and replay the captured backlog before live output.


I don't think we need a comment to summarise what the code does - we can read that.

Done — dropped the doc comments on attach_client/detach_client, the SocketConsole struct doc, and the two call-site narrations, and trimmed the new() and non-blocking comments to just the non-obvious bit. Kept only the genuinely explanatory ones (the buffer field, SharedSerialBuffer, and the fcntl-vs-set_nonblocking seccomp note). Pushed.

rbradford

Thanks for making those changes I asked for - much appreciated :

rbradford · 2026-06-03T19:58:06Z

+    // Use fcntl rather than UnixStream::set_nonblocking(): the latter issues
+    // ioctl(FIONBIO), which this thread's seccomp filter does not allow.
+    fn set_nonblocking(fd: BorrowedFd<'_>) -> Result<()> {
+        let fd = fd.as_raw_fd();
+        // SAFETY: `fd` is borrowed from a live owner, so valid for this call.
+        let ret = unsafe {
+            let flags = libc::fcntl(fd, libc::F_GETFL);
+            libc::fcntl(fd, libc::F_SETFL, flags | libc::O_NONBLOCK)
+        };
+        if ret < 0 {
+            return Err(Error::SetNonBlocking(std::io::Error::last_os_error()));
+        }
+        Ok(())
+    }
+


Thanks! But I kinda feel like the right thing to do is probably allow that ioctl in the seccomp filters. We are able to be precise about which ioctl we allow. WDYT?

Good call — done. Switched to UnixStream::set_nonblocking(true) and added a precise ioctl rule restricted to FIONBIO in serial_manager_thread_rules, mirroring the existing create_api_ioctl_seccomp_rule() (the HTTP-API thread already does exactly this): or![and![Cond::new(1, ArgLen::Dword, Eq, FIONBIO as _)?]]. On Linux set_nonblocking() issues ioctl(fd, FIONBIO, &1), so that's the one syscall the change introduces. Dropped the manual fcntl helper (and its unsafe/BorrowedFd).

Verified under --seccomp true with a real ~38 KB Ubuntu kernel boot: the client gets the full serial log (first kernel line → end), no SIGSYS.

rbradford

Thanks! 🚀

Code changed since review.

rbradford · 2026-06-04T11:21:05Z

@maxpain I think the cloud-hypervisor::integration common_parallel::test_serial_socket_interaction test is now taking a lot longer than before and failing sometimes. It used to take < 1 minute.

rbradford · 2026-06-05T11:01:07Z

@maxpain The test is still taking too long - https://github.com/cloud-hypervisor/cloud-hypervisor/actions/runs/27008281728/job/79706428355

maxpain · 2026-06-05T11:05:21Z

@maxpain The test is still taking too long - https://github.com/cloud-hypervisor/cloud-hypervisor/actions/runs/27008281728/job/79706428355

I'll fix this soon. Thanks!

SerialBuffer owns its downstream writer privately, with no way to replace it. Buffering the Socket console requires keeping one buffer alive across client connects and disconnects and pointing it at each newly accepted client (or a discarding sink when none is connected) without dropping bytes buffered while no client was attached. Add set_out(), which swaps the writer while leaving the buffered contents intact, plus unit tests covering accumulate-while-detached, replay on connect, live pass-through, delivery of while-detached output to the next client, and that bytes already drained by one client are not resent to the next. Signed-off-by: Max Makarov <[email protected]> Assisted-by: Claude:claude-opus-4-8 [Claude Code]

In Socket serial mode the device output sink was only installed once a client connected, so output produced beforehand (kernel boot messages, cloud-init) was dropped, and a client attaching after boot saw a blank screen. Only PTY mode wrapped the sink in a SerialBuffer. Install a persistent SerialBuffer as the Socket device's output sink at SerialManager construction (discarding downstream via io::sink() until a client connects), so output is captured into the 1 MiB ring even with no client attached. On connect, retarget the buffer at the accepted client and flush the backlog before live output resumes; on disconnect, keep buffering so output produced while no client is attached is delivered to the next one. The accepted socket is made non-blocking via set_nonblocking() so a slow client cannot stall the vCPU thread (SerialBuffer re-buffers on WouldBlock). The serial-manager thread gains two syscalls under seccomp: sendto (replaying the backlog is the first time it writes to the socket) and ioctl restricted to FIONBIO, which is what set_nonblocking() issues. Fixes: cloud-hypervisor#7907 Signed-off-by: Max Makarov <[email protected]> Assisted-by: Claude:claude-opus-4-8 [Claude Code]

maxpain · 2026-06-06T06:06:56Z

@rbradford Fixed — and it turned out to be entirely in the test, not the serial code.

Root cause: the test's socat pty echoed back whatever was written to it (raw alone didn't disable echo here). With the buffer in place a late client receives the whole boot backlog on connect; socat wrote that into the pty, the pty echoed it straight back, and it got fed to the guest as serial input. ~68 KB of the guest's own boot log looped into its UART → input overrun, serial-getty respawning, login never completing — so the read loop ran to the nextest timeout. That's both the "taking too long" and the intermittent failures. Before the buffer, pre-connect output was just dropped, so there was nothing to echo back — which is why it only surfaced now.

Test-only fix:

socat pty,...,raw,echo=0 — break the echo loopback.
read the pty concurrently with typing the login, so the replayed backlog is drained instead of back-pressuring socat.
pty_read() reads 4 KB chunks without the per-read sleep and the loop drains everything each round (and is bounded), so tens of KB of backlog no longer takes minutes.

The serial code is unchanged from what you approved (I dropped some experimental serial-manager changes I'd pushed while chasing this). integration-x86-64-pr just ran it green on this push in 24.5s — back under the minute it used to take (locally it's ~17s).

rbradford · 2026-06-06T11:50:42Z

+        let mut received = false;
+        loop {
+            match ptyc.try_recv() {
+                Ok(line) => {
+                    received = true;
+                    prev = prev + &line;
+                    if prev.contains("test_pty_console") {
+                        return;
+                    }
                }
+                Err(mpsc::TryRecvError::Empty) => break,
+                Err(_) => panic!("No login on pty"),
            }
-            Err(mpsc::TryRecvError::Empty) => {
-                empty += 1;
-                assert!(empty <= 5, "No login on pty");
-            }
-            _ => {
-                panic!("No login on pty")
-            }
+        }
+        if received {
+            empty = 0;
+        } else {
+            empty += 1;
+            assert!(empty <= 5, "No login on pty");
        }
    }
 }


If we receive 20 lines, and but none of them match "test_pty_console" we still exit from the loop without error. Could this be more defensively written?

Good catch — fixed: falling out of the bounded loop is now an explicit panic!("No login on pty") instead of a silent return, and I dropped the now-redundant empty counter. Squashed into the test commit.

With socket serial output now buffered and replayed on connect, a late-connecting client receives the whole boot backlog. The pty interaction test had three problems with that: - pty_read() slept a second between 512-byte reads and the loop consumed one chunk per two-second tick, far too slow to drain the backlog. Read in larger chunks without the per-read sleep and drain everything available each round; bound the loop so a missing marker can't run to the harness timeout. - it wrote the login keystrokes before reading, so the unread backlog back-pressured the sender and the keystrokes never reached the prompt. Start reading concurrently with typing instead. - the socat pty was created with echo on, so the replayed backlog was echoed back to the guest as serial input, flooding it (UART input overrun, login never completing). Create the pty with echo=0. Signed-off-by: Max Makarov <[email protected]> Assisted-by: Claude:claude-opus-4-8 [Claude Code]

maxpain requested a review from a team as a code owner June 3, 2026 06:19

This was referenced Jun 3, 2026

Serial/console socket connection should persist across VM reboot #7974

Open

virtio-blk: Add rotation_rate support for SSD/HDD identification #7970

Closed

phip1611 requested changes Jun 3, 2026

View reviewed changes

arctic-alpaca reviewed Jun 3, 2026

View reviewed changes

maxpain force-pushed the fix-7907-socket-console-buffer branch from ef01fe9 to 759cd5a Compare June 3, 2026 11:09

phip1611 previously requested changes Jun 3, 2026

View reviewed changes

maxpain force-pushed the fix-7907-socket-console-buffer branch from 759cd5a to 5eb92ab Compare June 3, 2026 12:27

rbradford requested a review from phip1611 June 3, 2026 12:47

rbradford reviewed Jun 3, 2026

View reviewed changes

maxpain force-pushed the fix-7907-socket-console-buffer branch from 5eb92ab to c9d9152 Compare June 3, 2026 13:10

rbradford requested changes Jun 3, 2026

View reviewed changes

maxpain force-pushed the fix-7907-socket-console-buffer branch from c9d9152 to 59a3807 Compare June 3, 2026 14:55

rbradford reviewed Jun 3, 2026

View reviewed changes

maxpain force-pushed the fix-7907-socket-console-buffer branch from 59a3807 to 90ad11b Compare June 3, 2026 22:25

rbradford approved these changes Jun 4, 2026

View reviewed changes

rbradford enabled auto-merge June 4, 2026 10:10

rbradford added this pull request to the merge queue Jun 4, 2026

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Jun 4, 2026

maxpain force-pushed the fix-7907-socket-console-buffer branch 2 times, most recently from dc346c2 to 6af406f Compare June 5, 2026 05:24

rbradford added this pull request to the merge queue Jun 5, 2026

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Jun 5, 2026

rbradford added this pull request to the merge queue Jun 5, 2026

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Jun 5, 2026

maxpain marked this pull request as draft June 6, 2026 02:48

maxpain added 2 commits June 6, 2026 04:48

maxpain force-pushed the fix-7907-socket-console-buffer branch from 6af406f to e37d9ea Compare June 6, 2026 05:19

maxpain marked this pull request as ready for review June 6, 2026 06:06

rbradford approved these changes Jun 6, 2026

View reviewed changes

maxpain force-pushed the fix-7907-socket-console-buffer branch from e37d9ea to 7475663 Compare June 7, 2026 00:47

rbradford approved these changes Jun 7, 2026

View reviewed changes

rbradford added this pull request to the merge queue Jun 8, 2026

Merged via the queue into cloud-hypervisor:main with commit 7f6df9e Jun 8, 2026
41 checks passed

maxpain deleted the fix-7907-socket-console-buffer branch June 8, 2026 12:14

rbradford added this to Cloud Hypervisor Roadmap Jul 8, 2026

github-project-automation Bot moved this to ✅ Done in Cloud Hypervisor Roadmap Jul 8, 2026

rbradford added the new-feature New feature to include in release notes label Jul 10, 2026

		/// Attach a freshly-accepted client: point the buffer at it, enable
		/// write-through, and replay the captured backlog before live output.

Uh oh!

Conversation

maxpain commented Jun 3, 2026

Problem

Fix

Verification

Uh oh!

phip1611 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxpain commented Jun 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rbradford commented Jun 3, 2026

Uh oh!

maxpain commented Jun 3, 2026

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxpain commented Jun 3, 2026

Uh oh!

rbradford Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rbradford commented Jun 4, 2026

Uh oh!

Uh oh!

Uh oh!

rbradford commented Jun 5, 2026

Uh oh!

maxpain commented Jun 5, 2026

Uh oh!

maxpain commented Jun 6, 2026

Uh oh!

rbradford Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

rbradford Jun 3, 2026 •

edited

Loading

rbradford Jun 6, 2026 •

edited

Loading