I think we want something like this. This yield-for makes mixing async and sync code in userspace easier. Currently, it is fairly easy to write all async or all sync code in userspace, but when mixing them async callbacks can lead to unexpected call chains when using yield_for() in userspace. With this yield for syscall, sync code knows no other callbacks will happen.

The decision to make is whether to go with this lightweight version, or something more integrated (or something else).

Instead (or in addition to), we could add yield-for-blocking, which would remove the upcall entirely, and instead pass the upcall arguments to userspace via the return arguments of yield-for-blocking.

Advantages of yield-for-blocking:

1. Userspace sync code is easy to write:

   allow();
   command();
   ret = yield_for_blocking(UPCALL_ID, &arg1, &arg2, &arg3);
   unallow();

   instead of:

   void callback(e, arg1, arg2, arg3) {
       arg1=arg1;
       arg2=arg2;
       arg3=arg3;
   }
   
   allow();
   subscribe(callback);
   command();
   yield_for(UPCALLID);
   unallow();

2. Eliminates the context switches for subscribe and the upcall.

Disadvantages:

1. Semantic confusion: schedule_upcall() in a capsule may not, actually, schedule an upcall any more. Instead the "upcall" happens as the return to a yield.
2. If userspace does call subscribe, but then only uses yield-for-blocking, that upcall would never get called, which is odd. But with a mix of yield-for-blocking and yield, the upcall would get called sometimes. This inconsistency complicates the interface.

Having articulated this, I see the appeal of yield-for. I think Pat pointed out that the complexity in userspace code is fairly easy to hide in a library layer, so that driver code would look like the yield-for-blocking case. Perhaps the prudent move is to address the complexity of writing sync and async code in userspace with yield-for, and not try to introduce blocking semantics before we know the aysnc version is insufficient.

A quick note, UPCALL_ID is made up of driver_number and subscribe_number in practice, so it has to be two arguments. The upcall signature is currently three integer arguments and a user data pointer; while the blocking call can eliminate the last argument, as the pointer can just be local to the userland callsite, one of the arguments would have to overwrite one passed in. That makes this pseudocode more awkward:

allow();
command();
// Can't do this:   ret = yield_for_blocking(UPCALL_ID, &arg1, &arg2, &arg3);
int subscribe_num_and_arg1 = SUBSCRIBE_NUMBER;
ret = yield_for_blocking(DRIVER_NUMBER, &subscribe_num_and_arg1, &arg2, &arg3);
unallow();

There are probably other games that could be played to ease this, but it's all more complex than the lightweight change I think we should start with as-proposed here.

allow();
command();
// Can't do this:   ret = yield_for_blocking(UPCALL_ID, &arg1, &arg2, &arg3);
int subscribe_num_and_arg1 = SUBSCRIBE_NUMBER;
ret = yield_for_blocking(DRIVER_NUMBER, &subscribe_num_and_arg1, &arg2, &arg3);
unallow();

But the arg1 arg2 arg3 are going back up (kernel to userspace) and driver num and subscribe num are going down (userspace to kernel).

Oh, you were thinking some more library magic here? I guess that works so long as the lower layer does the remapping and the pointer writing. I don't think there is any way to pass all the pointers for arg1-arg3 in the syscall and have the kernel do the write, but that's not necessary.... I think the below would work?

// libtock.c
int yield_for_blocking(struct upcall_id, &arg1, &arg2, &arg3) {
    register uint32_t r0 __asm__ ("r0") = YIELD_FOR_BLOCKING_ID;
    register uint32_t r1 __asm__ ("r1") = upcall_id.drv_num;
    register uint32_t r2 __asm__ ("r2") = upcall_id.sub_num;
    register int retval __asm__ ("r0");
    register int rv1 __asm__ ("r1");
    register int rv2 __asm__ ("r2");
    register int rv3 __asm__ ("r3");
    __asm__ volatile (
    "svc 0"
    : "=r" (retval), "=r" (rv1), "=r" (rv2), "=r" (rv3)
    : "r" (r0), "r" (r1), "r" (r2)
    : "memory");

    if (retval & YIELD_FOR_ACTUALLY_YIELDED_FLAG_MASK) {
        *arg1 = rv1;
        *arg2 = rv2;
        *arg3 = rv3;
    }
    return retval;
}

Yes that is what I was imagining.

The C wrapper could just take five arguments though instead of a struct:

int yield_for_blocking(driver_num, subscribe_num, &arg1, &arg2, &arg3) {

True, though, since we're size-optimizing, if that function didn't get inlined everywhere, adding the fifth argument would require spilling to the stack [assuming stuct upcall_id were passed by ref, unlike the implicit copy I wrote above..] — for something low-level and apparently on the hot path like this, I'd push to limit to four args generally.

True, though, since we're size-optimizing, if that function didn't get inlined everywhere, adding the fifth argument would require spilling to the stack [assuming stuct upcall_id were passed by ref, unlike the implicit copy I wrote above..] — for something low-level and apparently on the hot path like this, I'd push to limit to four args generally.

This function seems like something that should get inlined everywhere, provided that it is being built with reasonable optimization settings, which should be the case for any project particularly concerned about size. So I would advocate for the five argument function definition, though I don't feel super strongly about it. If we used your approach, wouldn't we need to pass a pointer to upcall_id in order to not spill to the stack?

That said, I think that the disadvantages Brad described are valid:

Disadvantages:

Semantic confusion: schedule_upcall() in a capsule may not, actually, schedule an upcall any more. Instead the "upcall" happens as the return to a yield.
If userspace does call subscribe, but then only uses yield-for-blocking, that upcall would never get called, which is odd. But with a mix of yield-for-blocking and yield, the upcall would get called sometimes. This inconsistency complicates the interface.

These are both disadvantages that do not exist with blocking_command() as Alyssa described it at TockWorld.

yield_for(), in cases where the return value matters, does not remove any system calls compared to the current status quo, or the need for a callback function. It just allows userspace to ensure that only one callback will ever be called. Note that libtock-rs today only even allows one outstanding callback, so this seems to be of very limited usefulness in libtock-rs.

yield_for_blocking(), as Brad described, removes the context switch for subscribe (and unsubscribe in libtock-rs), as well as the need to implement a callback. But it comes with the two disadvantages that Brad mentioned, and still requires one more context switch than blocking_command(). The potential for mixing of functions that call yield_for_blocking() and subscribe() takes away a lot of the purported advantage here of "making it easier to simplify correctness for userspace apps", since there would be a new class of bug if apps mix async and sync driver methods that use the same callbacks.

wouldn't we need to pass a pointer to upcall_id in order to not spill to the stack?

Yes, that's what I meant by "[assuming struct upcall_id were passed by ref, unlike the implicit copy I wrote above..]"

I'm generally wary about relying on compiler optimizations for performance correctness, but I think we can shelve this side-hypothetical for now.

Note that libtock-rs today only even allows one outstanding callback

Why is that limitation in place? That's a pretty crippling limitation for a lot of applications — can't have something like a periodic timer and a event-based sensor subscription live at the same time? That's not really an acceptable or realistic limitation for the long term.

If the issue is concurrent execution concerns of callbacks, yield_for as-proposed can be used to ensure that only one callback stack/chain is active at any given time.

I'm not opposed to something of the spirit of yield_for_blocking in addition to yield_for, but I think that is a more complex and nuanced interface for the reasons we've articulated, and that the simpler yield_for is useful on its own merits.

One thing I had been ruminating on was whether there would be value in something closer to a select/poll style interface, or in this context a yeild_for_any_of([array of upcall ids]) (with return value of index of which array entry fired).

In practice, I think the ergonomics around first allowing the array and then sending the next syscall with the pointer is awkward, especially for the likely common-case of just wanting to yield on one specific upcall. I'd be inclined to steer clear of this more complex interface until it's proven necessary.

Note that libtock-rs today only even allows one outstanding callback

Why is that limitation in place? That's a pretty crippling limitation for a lot of applications — can't have something like a periodic timer and a event-based sensor subscription live at the same time? That's not really an acceptable or realistic limitation for the long term.

I'm not sure it is? In my little dabble yesterday it seems like subscribe in libtock-rs is essentially the with operator in python: the subscribe is removed when the context goes away, but you can nest "with"s. Maybe that doesn't actually work in practice, but that's the impression the libtock-rs API gives.

But it comes with the two disadvantages that Brad mentioned, and still requires one more context switch than blocking_command().

It's really hard to discuss blocking command without something like this PR. What exactly is blocking command? Critically, how does a capsule give the blocking command its return data? Also, can someone write its #### Disadvantages section? I kinda made it easy by writing my own for the idea I was supporting.

It seems like the major disadvantage is going to be: when writing a capsule driver, after my interrupt fires, do I call grant.sched_upcall() or kernel.command_return()? And when using a capsule, can I call subscribe or do I have to call blocking command?

Imagine we implement yield-for-blocking and it turns out that we like it, and who knows maybe we rename grant.sched_upcall() to grant.deliver() or something and we end up with a nice way to merge an async kernel with a convenient sequential userspace.

It seems like it might not be too big of a step to implement command-yield-for-blocking, which is a command immediately followed by a yield-for-blocking in a single system call (how exactly to do that tbd).

I say this because that to me seems more plausible than re-writing all of our capsules to have both an async and a sync version (if in fact that is what is required for blocking_command).

Note that libtock-rs today only even allows one outstanding callback

Why is that limitation in place? That's a pretty crippling limitation for a lot of applications — can't have something like a periodic timer and a event-based sensor subscription live at the same time? That's not really an acceptable or realistic limitation for the long term.

libtock-rs allows multiple outstanding callbacks (e.g. you can wait for multiple sensors concurrently, with a timeout alarm), but it does not allow for independent tasks. That is, you cannot have a callback running in the background that blinks an LED continuously while your code does something completely unrelated to that.

There is more context on the decision to not support independent tasks at tock/libtock-rs#334. The short version is that supporting that requires making a lot of important design decisions that I did not have the data to make at the time.

I also want to re-evaluate whether futures are acceptable in libtock-rs. Over the past few years, I've received a lot of suggestions on how to do more lightweight Future-based APIs than I achieved when I benchmarked the code size impacts of Future. I haven't had the time to investigate those suggestions, but I want to do so, and if any of them pan out then that would we wonderful. Using futures would allow us to support independent tasks without having to make those painful design tradeoffs.

That state of libtock-rs is much more sane.

w.r.t. to an eventual more async world, there's some more complex stuff in libtock-c apps, but really nothing that would demand more capability than what is sounds like libtock-rs already supports. I do think one of the good outcomes from TockWorld6 is a push for the core team to get more parity in libtock-c / libtock-rs examples, so if I'm wrong, I suspect we'll see soon :).

I'm strongly supportive of something very close to this.

High bit

Yield-WaitFor solves several related problems concisely and elegantly:

1. It allows for true blocking semantics, most notably for debug statements in callbacks, where "simulated" blocking using the user-space yield_for relies heavily on user-written callbacks not, themselves calling yield_for recursively, e.g.. This is a big semantic improvement to userspace.

2. It can dramatically reduce the code size required of user-space applications that using a synchronous I/O API---they could avoid callbacks all-together, avoid corresponding subscribe calls, and reduces the number of system calls invoked per operation for some common operations. This seems particularly useful in Rust applications where having callbacks always subscribed in the background (without unsubscribing them) may be unsound.

3. It doesn't require any additional functionality in capsules to support this. Everything is entirely differentiated in the process scheduler. This is important because it means that any code size implications in the kernel are fixed, and it means that userspace doesn't need to wait for a particular capsule to adapt to the new API in order to use this new yield variant.

Some more details separated so they can be quoted separately

More on 1: it's worth looking through our various libtock-c examples and drivers and seeing where we would actually replace calls to the current library yield_for with a new Yield-WaitFor. In practice, these have slightly different semantics, so it's not obvious it would be everywhere, but it might be everywhere or almost everywhere.

More on 2: with some specialization in userspace as well as some sort of combined system-call system call ("please do a command followed by a yield-waitfor") I think we can get nearly optimal code size in applications (compared to a functionality-specific specialized blocking call). So we should view this as a step towards even more code size optimization.

(There are no!) Disadvantages

@bradjc points out some disadvantages. These are very well worth considering. I think an evolution of this proposal avoids these pitfalls.

1. Semantic confusion: schedule_upcall() in a capsule may not, actually, schedule an upcall any more. Instead the "upcall" happens as the return to a yield.

Yes. As suggested later in the thread, either renaming or aliasing schedule_upcall to deliver may help avoid this confusion. I think this is just a naming issue, which we should resolve, but doesn't break this proposal.

2. If userspace does call subscribe, but then only uses yield-for-blocking, that upcall would never get called, which is odd. But with a mix of yield-for-blocking and yield, the upcall would get called sometimes. This inconsistency complicates the interface.

Agree completely. Yield-WaitFor should use the existence of a subscribed callback to determine whether to execute a callback or return directly to the system call sight with the values passed to schedule_upcall. If a process has a callback subscribed, that callback should always execute. If it doesn't, Yield-WaitFor should always return the values that would have been passed to a callback.

There is a remaining question of what to do with r0-r3 if a callback was executed. I think, in this case, it might be reasonable to allow the callback (currently a void) to determine what those values should be.

Naming versions of the Yield variant

We've now described several variations of the proposed yield variant and I believe I've added to the confusion by conflating names for them above. Let's refer to them as follows:

• Yield-WaitFor is the original proposal as described in this revision of the TRD text
• Yield-WaitFor-NoCallback (proposed by @bradjc) never executes a callback, regardless of whether one is registered, and instead returns the values that would be passed to the callback as return values to the system call site.
• Yield-WaitFor-CallbackIfPresent (proposed implicitly by me) always executes a callback if one is present, otherwise it returns directly to the callsight returning the values that would have been passed to the callback in registers r0-r3.
• Yield-WaitFor-OptionalSubscribe (proposed by @kupiakos) yield passes an upcall handler if it wants to use an upcall (which replaces any existing subscribed upcall handler), otherwise upcall values are passed as return values to the system call site.

What is the usecase for calling subscribe and using Yield-WaitFor-CallbackIfPresent? Maybe writing async code with ordering semantics where the code can know which order the callbacks will happen?

What is the usecase for calling subscribe and using Yield-WaitFor-CallbackIfPresent? Maybe writing async code with ordering semantics where the code can know which order the callbacks will happen?

One simple example may be logging statistics on how often something happens.

If the callback is able to return different values to the syscall site in r0-r3, it might be useful for filtering or customizing the result...

Just general modularity.

Indeed, I suspect this will rarely be used.

I've updated this to match the Yield-WaitFor-CallbackIfPresent proposed by @alevy.

There are some design choices not fully elucidated in the RFC yet, as I'd like us to decide what we want to do about #3577 (comment) before fleshing that text out completely.

PoC code is updated (again, compile-tested only) to realize the proposed interface.

If the callback is able to return different values to the syscall site in r0-r3, it might be useful for filtering or customizing the result...

Is Yield-WaitFor-CallbackIfPresent either or, or both? I think it is either: either the upcall is called or yield returns r0-r3, but not both. So if you use Yield-WaitFor-CallbackIfPresent with a subscribe then it can't also use r0-r3. Or are you saying those would be some other values instead of the upcall return values? But, reading the TRD again, I don't think we can both call an upcall and return values.

With Yield-WaitFor-CallbackIfPresent, is it true that userspace can't determine at runtime if r0-r3 are valid, or if an upcall executed?

So libtock tries to write:

int my_command_sync() {
  driver_command();
  asm("yield wait-for-callbackifpresent");
  // did an upcall execute?
  // or is r0-r3 valid?
  return ??
}

But the then the user at some point called subscribe(my_driver) and then forgot about it. So when the user calls my_command_sync() it seems to work ok, but the values are wrong.

Is this possible or am I missing something?

With Yield-WaitFor-CallbackIfPresent, is it true that userspace can't determine at runtime if r0-r3 are valid, or if an upcall executed?

As currently written, this is correct, yes. It assumes that userspace tracks whether it has subscribed a callback or not correctly, and if userspace cares that a callback ran, the callback can set a flag.

We could use yield-param-3 to do the same pointer-indicator thing as Yield-NoWait if desired.

With Yield-WaitFor-CallbackIfPresent, is it true that userspace can't determine at runtime if r0-r3 are valid, or if an upcall executed?

As currently written, this is correct, yes. It assumes that userspace tracks whether it has subscribed a callback or not correctly, and if userspace cares that a callback ran, the callback can set a flag.

We could use yield-param-3 to do the same pointer-indicator thing as Yield-NoWait if desired.

I propose we pass all yield return arguments via a pointer specified by yield-param-3 unconditionally.

Otherwise, I don't know how we would use Yield-WaitFor-CallbackIfPresent in userspace libraries other than calling subscribe(driver, upcall, 0) before Yield-WaitFor-CallbackIfPresent every time.

OR, I prefer Yield-WaitFor-NoCallback to avoid this potential error entirely (and it would not be a fun bug to debug I suspect).

@tock/core-wg This is definitely P-Significant, and I marked it so, so we need another approval. Having said that, I think this is pretty good to go.

@alexandruradovici Comments adapted. I think all good.

In general, I don't love the structure of the Task enum, which is now kind pulling double-duty, and thus requires a bit more documentation (as you've added) to make clear how it's actually used. Having said that, I think it's worth deferring refactoring the code until IPC is potentially reworked, which might unify Task::IPC and Task::FunctionCall, and overall result in cleaner code.

How did we merge a significant PR in 3 days with only 2 approvals?

Did anyone else test this? I ran exactly one test on one board with one process. And nothing on riscv.

We need 2 approvals for a p-significants

We need 2 approvals for a p-significants

Can we write this down somewhere? That is not my understanding nor what our documentation says

Significant pull requests require review by the entire core team. Each
core team member is expected to respond within one week.

I guess this PR is a bit of a grey area because it was a draft for a year, but it wasn't clear how close that draft was to mergable, and core members should have a week to look at significant PRs. After a week 2 approvals makes sense.

Ok, sorry.

My sense was that it had sat since your rebase and testing, I confirmed with similar testing, and there has been for a while either consensus or apathy about the design (I think I'm the most dissenting voice about the design specifically). And we'll do more testing for release and now that it's merged.

We can/should clarify that language. Maybe your interpretation is right and I was wrong, maybe my interpretation was right, but either way it's not clear

We can/should clarify that language. Maybe your interpretation is right and I was wrong, maybe my interpretation was right, but either way it's not clear

I don't know whether requiring approvals from all core team members is feasible in practice, but we should definitely allow for more time from the point where reviews from the WG are requested, to when someone is able to hit merge.

How did we merge a significant PR in 3 days with only 2 approvals?

It's also still an RFC