Introduce uhyve-interface v2 and MmioWrite-based hypercalls on x86_64 #1145

n0toose · 2025-10-25T15:34:48Z

Experimental branch/pull request built on top of #754

codecov · 2025-10-25T15:38:15Z

Codecov Report

❌ Patch coverage is 44.70046% with 120 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.26%. Comparing base (c1b5916) to head (87ff950).

Files with missing lines	Patch %	Lines
src/linux/x86_64/kvm_cpu.rs	23.40%	72 Missing ⚠️
src/hypercall.rs	60.00%	36 Missing ⚠️
uhyve-interface/src/lib.rs	50.00%	4 Missing ⚠️
uhyve-interface/src/v1/mod.rs	0.00%	3 Missing ⚠️
uhyve-interface/src/v2/mod.rs	0.00%	3 Missing ⚠️
src/vm.rs	83.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1145      +/-   ##
==========================================
- Coverage   76.70%   75.26%   -1.45%     
==========================================
  Files          29       31       +2     
  Lines        3490     3626     +136     
==========================================
+ Hits         2677     2729      +52     
- Misses        813      897      +84

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

n0toose · 2025-10-25T16:08:23Z

src/hypercall.rs

+				let sysunlink = unsafe { mem.get_ref_mut(data).unwrap() };
+				Hypercall::FileUnlink(sysunlink)
+			}
+			HypercallAddress::Exit => Hypercall::Exit(data.as_u64() as i32),


... I frankly forgot why I just decided to pass the integer instead of dereferencing a pointer, but perhaps it might be worth thinking about?

I'm not sure if I understand.
Just passing the exit code directly is fine here - no further information is necessary when exiting.

src/hypercall.rs

jounathaen · 2025-10-27T08:25:12Z

src/arch/x86_64/paging/mod.rs

 		};
 	}
+
+	let _ = unsafe {


A comment explaining this is likely needed.

A comment explaining this is likely needed.

Agreed; it's just that things do actually break if we use a non-relocatable image as of now, so I can't really justify that it's safe using a // SAFETY: comment just yet :D

jounathaen · 2025-10-27T08:25:47Z

src/arch/x86_64/paging/mod.rs

+
+	let _ = unsafe {
+		pagetable_mapping.identity_map(
+			PhysFrame::<Size2MiB>::from_start_address(PhysAddr::new(0)).unwrap(),


Would it also work when only mapping a 4KiB Page? Would that impact performance?

Would that impact performance?

You mentioned you wanted measurements for the MMIO-based hypercalls, is this in-scope?

Would be interesting to check the difference. Mapping a single 4KiB page is better from a memory point of view, but 2MiB pages are faster to access. Just a note here for later measurements.

src/hypercall.rs

src/macos/x86_64/vcpu.rs

src/hypercall.rs

src/linux/x86_64/kvm_cpu.rs

src/hypercall.rs

fogti · 2025-10-30T04:36:24Z

src/hypercall.rs

+			return peripherals.serial.output(bytes);
+		} else if !file_map.fdmap.is_fd_present(syswrite.fd.into_raw_fd()) {
+			// We don't write anything if the file descriptor is not available,
+			// but this is OK for now, as we have no means of returning an error code


Given that this is a new interface version that isn't currently deployed, wouldn't it make sense to adjust the write hypercall params to also have a way of returning non-fatal errors?

What do you envision?
I'm all open for extra fields here — the reasoning behind the current params is to be close to Linux/POSIX write. Since that is the de-facto interface to all applications as well as the file system interface on the Uhyve side, we save transformations when keeping things close.

I'm solely complaining about the lack of being able to emit non-fatal write errors back to the guest (currently all errors from writes afaik terminate the guest).

I discussed this with @jounathaen last week and was mildly unsure as to what type of errors we could report here; it's not like we can use POSIX as a reference, right?

Why not? Like, we could pick any arbitrary standard, we just have to make sure that it fits the constants defined in hermit-abi and isn't extremely cumbersome to use from the kernel side.

The main errors that would make sense to forward are EPIPE, ECONNRESET (for host-side IPC stuff, i.e. anything that uses pipes or sockets on host-side) and ENOSPC (we can't expect the guest to know how much disk space it has available).

fogti · 2025-10-30T04:40:24Z

uhyve-interface/src/v2/mod.rs

+/// The discriminants of this enum are the respective addresses, so one can get the code by calling
+/// e.g., `HypercallAddress::Exit as u64`.
+#[non_exhaustive]
+#[repr(u64)]


Would this (e.g. HypercallAddress::Exit as u64) also work with #[repr(u16)] or such?

Well, yes. But as this is intended to represent a GuestPhysAddr, I've opted for an u64 to reduce as 64 statements in the call.

I don't think it makes a difference, and subsequently #[repr(u16)] should be preferred: https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=610243c9a13c895b8140a3a22bddc5d4

uhyve-interface/src/v2/parameters.rs

fogti · 2025-10-30T13:46:20Z

Given that this PR depends upon another PR that is currently marked as a draft, it makes to sense to mark this one as non-draft, yet.

fogti · 2025-10-30T15:10:17Z

tbqh, d1d002c took me way too long, but I think it does massively cut down on code duplication.

fogti · 2025-10-30T15:15:37Z

Appears to work on linux now 🎉

fogti

Apart from fixing the MacOS / Darwin part, this looks good now.

src/linux/x86_64/kvm_cpu.rs

n0toose · 2025-11-01T17:52:28Z

Apart from fixing the MacOS / Darwin part, this looks good now.

The test-kernels are also a bit of a problem. It would make sense to compile an older version of the kernel to test whether the I/O port-based hypercalls still work in those cases, right?

(P.S. I'm aware of the suggestion regarding that exit still)

n0toose · 2025-11-01T17:53:16Z

We also aren't making the paging conditional - there are some corner cases (particularly when disabling ASLR) where this breaks, I think. So we're not quite done yet.

n0toose · 2025-11-07T16:32:06Z

Okay, now that the "abuse the macOS CI because I don't have a machine" strat worked, I should probably fix the mess I made trying to frantically rebase this, check whether I wiped some work during rebasing accidentally, squash and continue.

src/arch/aarch64/mod.rs

uhyve-interface/src/v2/parameters.rs

src/macos/aarch64/vcpu.rs

fogti

Overall LGTM, a few weirdnesses remain.

src/hypercall.rs

fogti · 2025-11-19T15:53:37Z

src/hypercall.rs

+	if let Ok(hypercall_port) = HypercallAddress::try_from(addr as u64) {
+		Some(match hypercall_port {
+			HypercallAddress::FileClose => {
+				let sysclose = unsafe { mem.get_ref_mut::<CloseParams>(data).unwrap() };


Why do we sometimes have to explicitly denote the parameter type (e.g. ::<CloseParams> and sometimes not?

I don't think this was answered yet...

src/hypercall.rs

src/stats.rs

src/vm.rs

uhyve-interface/src/v2/parameters.rs

This globally enforces a suggestion by Ellen in: hermit-os#1145 Helped-by: Ellen Emilia Anna Zscheile <[email protected]>

This is based on a suggestion by Ellen: hermit-os#1145 (comment) Helped-by: Ellen Emilia Anna Zscheile <[email protected]>

fogti · 2025-11-29T18:42:01Z

src/hypercall.rs

+					if bytes_read >= 0 {
+						bytes_read as i64
+					} else {
+						-1


Shouldn't we use some errno number here?

fogti · 2025-11-29T18:43:05Z

src/vm.rs

 use std::{
 	env, fmt, fs, io,
-	num::NonZeroU32,
+	num::{NonZero, NonZeroU32},


We could probably get rid of the NonZeroU32, too.

fogti · 2025-11-29T18:44:38Z

uhyve-interface/src/v2/parameters.rs

 	/// Number of bytes to read into the buffer.
-	pub len: usize,
+	pub len: u64,
 	/// Number of bytes read on success. `-1` on failure.


What about -EFAULT?

This is based on a suggestion by Ellen: hermit-os#1145 (comment) Helped-by: Ellen Emilia Anna Zscheile <[email protected]>

This globally enforces a suggestion by Ellen in: #1145 Helped-by: Ellen Emilia Anna Zscheile <[email protected]>

This is based on a suggestion by Ellen: #1145 (comment) Helped-by: Ellen Emilia Anna Zscheile <[email protected]>

Co-authored-by: Panagiotis "Ivory" Vasilopoulos <[email protected]>

- bump uhyve-interface to 0.2.0 - perform virtaddr -> physaddr conversions in hermit for read, write. - use adjust serial_port based on version of uhyve-interface used - Exit hypercalls should encode the exit code instead of a pointer to the integer. ExitArgs has been removed. - v2: add v2 support in src/stats.rs, futureproof - v2: use u64, i64 instead of usize, isize - v2 (x86_64): allocate page for MmioExits uhyve-interface (v2) should now use MmioWrite to invoke hypercalls, so as to allow for using pointers to 64-bit addresses (ports restrict us to 32-bit). This restriction should not apply for serial writes, although it also remains possible for the kernel to print out messages. Co-authored-by: Ellen Emilia Anna Zscheile <[email protected]> Co-authored-by: Jonathan Klimt <[email protected]>

n0toose mentioned this pull request Oct 25, 2025

feat(uhyve): use v2 interface, MmioWrite exits on x86_64 hermit-os/kernel#2013

Draft

n0toose commented Oct 25, 2025

View reviewed changes

src/hypercall.rs Show resolved Hide resolved

jounathaen reviewed Oct 27, 2025

View reviewed changes

fogti mentioned this pull request Oct 28, 2025

feat(isolation/fd): implement virtual file descriptors #1140

Merged

n0toose force-pushed the uhyve_if_v2-mmiowrite branch 2 times, most recently from 520e639 to 3aa5f14 Compare October 28, 2025 12:40

fogti requested changes Oct 29, 2025

View reviewed changes

src/hypercall.rs Outdated Show resolved Hide resolved

src/hypercall.rs Outdated Show resolved Hide resolved

fogti reviewed Oct 29, 2025

View reviewed changes

src/hypercall.rs Outdated Show resolved Hide resolved

jounathaen reviewed Oct 29, 2025

View reviewed changes

src/macos/x86_64/vcpu.rs Outdated Show resolved Hide resolved

fogti reviewed Oct 29, 2025

View reviewed changes

src/hypercall.rs Outdated Show resolved Hide resolved

fogti reviewed Oct 30, 2025

View reviewed changes

src/linux/x86_64/kvm_cpu.rs Outdated Show resolved Hide resolved

fogti reviewed Oct 30, 2025

View reviewed changes

src/hypercall.rs Show resolved Hide resolved

fogti reviewed Oct 30, 2025

View reviewed changes

uhyve-interface/src/v2/parameters.rs Outdated Show resolved Hide resolved

fogti self-requested a review October 30, 2025 13:45

fogti marked this pull request as draft October 30, 2025 13:45

fogti force-pushed the uhyve_if_v2-mmiowrite branch from d1d002c to 3d13bb4 Compare October 30, 2025 15:12

fogti approved these changes Oct 30, 2025

View reviewed changes

fogti reviewed Oct 31, 2025

View reviewed changes

src/linux/x86_64/kvm_cpu.rs Outdated Show resolved Hide resolved

n0toose force-pushed the uhyve_if_v2-mmiowrite branch from 3b25ec9 to 7287905 Compare November 7, 2025 16:08

fogti reviewed Nov 14, 2025

View reviewed changes

src/arch/aarch64/mod.rs Outdated Show resolved Hide resolved

n0toose force-pushed the uhyve_if_v2-mmiowrite branch from b4da38a to 2c77b78 Compare November 14, 2025 14:20

n0toose marked this pull request as ready for review November 14, 2025 15:54

n0toose changed the title ~~Uhyve Interface V2 + MmioWrite-based hypercalls~~ Introduce uhyve-interface v2 and MmioWrite-based hypercalls on x86_64 Nov 14, 2025

This comment was marked as resolved.

Sign in to view

fogti mentioned this pull request Nov 19, 2025

annotate(hypercall/lseek): make sure missing constants cause compile-time error #1170

Merged

jounathaen reviewed Nov 19, 2025

View reviewed changes

uhyve-interface/src/v2/parameters.rs Outdated Show resolved Hide resolved

jounathaen reviewed Nov 19, 2025

View reviewed changes

uhyve-interface/src/v2/parameters.rs Outdated Show resolved Hide resolved

jounathaen reviewed Nov 19, 2025

View reviewed changes

uhyve-interface/src/v2/parameters.rs Outdated Show resolved Hide resolved

jounathaen reviewed Nov 19, 2025

View reviewed changes

uhyve-interface/src/v2/parameters.rs Outdated Show resolved Hide resolved

jounathaen reviewed Nov 19, 2025

View reviewed changes

uhyve-interface/src/v2/parameters.rs Outdated Show resolved Hide resolved

jounathaen reviewed Nov 19, 2025

View reviewed changes

src/macos/aarch64/vcpu.rs Outdated Show resolved Hide resolved

fogti requested changes Nov 19, 2025

View reviewed changes

n0toose added a commit to n0toose/uhyve that referenced this pull request Nov 29, 2025

style: use generic NonZero

5660204

This globally enforces a suggestion by Ellen in: hermit-os#1145 Helped-by: Ellen Emilia Anna Zscheile <[email protected]>

n0toose mentioned this pull request Nov 29, 2025

style: use generic NonZero #1178

Merged

n0toose added a commit to n0toose/uhyve that referenced this pull request Nov 29, 2025

style: use generic NonZero

33ab945

This globally enforces a suggestion by Ellen in: hermit-os#1145 Helped-by: Ellen Emilia Anna Zscheile <[email protected]>

n0toose added a commit to n0toose/uhyve that referenced this pull request Nov 29, 2025

fix(hypercall): move write standard output case outside of loop

c48f719

This is based on a suggestion by Ellen: hermit-os#1145 (comment) Helped-by: Ellen Emilia Anna Zscheile <[email protected]>

n0toose mentioned this pull request Nov 29, 2025

fix(hypercall): move write standard output case outside of loop #1179

Merged

fogti reviewed Nov 29, 2025

View reviewed changes

src/hypercall.rs

if bytes_read >= 0 {

bytes_read as i64

} else {

-1

Copy link

Contributor

fogti Nov 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we use some errno number here?

fogti reviewed Nov 29, 2025

View reviewed changes

n0toose added a commit to n0toose/uhyve that referenced this pull request Nov 29, 2025

fix(hypercall): move write standard output case outside of loop

87081db

This is based on a suggestion by Ellen: hermit-os#1145 (comment) Helped-by: Ellen Emilia Anna Zscheile <[email protected]>

github-merge-queue bot pushed a commit that referenced this pull request Dec 1, 2025

style: use generic NonZero

fd12053

This globally enforces a suggestion by Ellen in: #1145 Helped-by: Ellen Emilia Anna Zscheile <[email protected]>

github-merge-queue bot pushed a commit that referenced this pull request Dec 1, 2025

fix(hypercall): move write standard output case outside of loop

919de9c

This is based on a suggestion by Ellen: #1145 (comment) Helped-by: Ellen Emilia Anna Zscheile <[email protected]>

n0toose force-pushed the uhyve_if_v2-mmiowrite branch from a1aa479 to 175d06f Compare December 1, 2025 22:52

jounathaen and others added 4 commits December 2, 2025 00:16

feat(interface): initial version -> v1 folder

be39b27

Co-authored-by: Panagiotis "Ivory" Vasilopoulos <[email protected]>

docs(interface): add README file

fbe9473

feat(tests): adjust test-kernels for uhyve-interface v2

954e7e1

n0toose force-pushed the uhyve_if_v2-mmiowrite branch from 175d06f to 954e7e1 Compare December 1, 2025 23:34

Introduce uhyve-interface v2 and MmioWrite-based hypercalls on x86_64 #1145

Are you sure you want to change the base?

Introduce uhyve-interface v2 and MmioWrite-based hypercalls on x86_64 #1145

Uh oh!

Conversation

n0toose commented Oct 25, 2025

Uh oh!

codecov bot commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fogti commented Oct 30, 2025

Uh oh!

fogti commented Oct 30, 2025

Uh oh!

fogti commented Oct 30, 2025

Uh oh!

fogti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

n0toose commented Nov 1, 2025

Uh oh!

n0toose commented Nov 1, 2025

Uh oh!

n0toose commented Nov 7, 2025

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fogti left a comment

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Oct 25, 2025 •

edited

Loading