Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix(capsule): bind run loops to typed owner context - #1380

Merged
joshuajbouw merged 1 commit into
astrid-runtime:mainfrom
jvsteiner:fix/1197-run-loop-per-principal
Aug 14, 2026
Merged

joshuajbouw merged 1 commit into
astrid-runtime:mainfrom
jvsteiner:fix/1197-run-loop-per-principal

Conversation

@jvsteiner

@jvsteiner jvsteiner commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Linked Issue

Closes #1224. Refs #1197 and #1454.

Summary

Autonomous #[astrid::run] capsules entered their run export without an invocation principal, leaving owner KV, home/tmp, secrets, env, profile, budgets, log, and cancellation at the neutral floor. On the authority-scoped runtime architecture from #1487, this port installs the typed principal runtime owner's context before the run task is published. Explicit SystemResident runtimes remain neutral.

Changes

  • Recognize autonomous owner context only from typed RuntimeScope::Principal; never infer authority from default, manifest shape, or first-loader state.
  • Require and resolve the owning principal profile for principal-resident run exports.
  • Derive owner home, environment, and log paths from the kernel-supplied capsule context.
  • Install the already UID-scoped KV, home/tmp, secrets, profile, env, budgets, log, and cancellation context before entering the guest run export.
  • Reject SystemResident and mismatched capsule RuntimeIds in the owner-context installer.
  • Preserve existing unstamped compatibility callers and recv-driven publisher context behavior.

Verification

  • cargo test -p astrid-capsule -- --quiet — 633 passed
  • cargo test -p astrid-kernel --lib -- --quiet — 321 passed
  • cargo clippy -p astrid-capsule --all-features -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check
  • Regressions prove Alice's run-loop KV survives replacement, default and Bob cannot observe or overwrite it, owner home/tmp/secrets/log/profile/env are installed, supplied-home env isolation holds, and system/wrong-capsule scopes fail closed.
  • Independent adversarial review: clean

Scope

This closes autonomous run-loop owner context. #1454 remains separate: the current recv ABI has no reliable handler-return boundary, so automatically restoring owner context after a publisher-scoped recv would be an approximation rather than a sound lifecycle edge.

AI / Tool Assistance

Assisted-by: Claude:Opus-4.8

Assisted-by: Codex:GPT-5

Jamie Steiner and Claude developed the original fix and tests. Codex ported and hardened the change against #1487's typed RuntimeId/RuntimeScope architecture, expanded isolation and replacement regressions, and performed adversarial review. Joshua reviewed the resulting changes and validation and authorized the signed commit, retaining responsibility for the design and merge decision.

Checklist

  • Linked to an issue
  • CHANGELOG.md updated (entry under [Unreleased] — or [Unreleased] rolled into a version section for a release PR; not applicable to docs/CI-only changes)
  • I understand every change in this PR and can explain its design, risks, and validation.
  • I reviewed and tested any meaningful tool-generated output included in this PR.
  • Every non-bot, non-merge commit has a matching Signed-off-by trailer.

@jvsteiner
jvsteiner marked this pull request as ready for review July 26, 2026 12:27
@jvsteiner
jvsteiner requested a review from joshuajbouw as a code owner July 26, 2026 12:27
@jvsteiner

Copy link
Copy Markdown
Contributor Author

@joshuajbouw ready for review — run-loop capsules get their per-principal KV/env/secrets overlay before the guest run export (#1197, #1224).

@joshuajbouw

Copy link
Copy Markdown
Member

Review outcome: technically worth advancing. The runtime, Windows, CodeQL, and CI jobs are green. Please update the PR body to the repository template headings (Linked Issue, Summary, Changes, Verification/Test Plan, Checklist) and add the required changelog entry; those are the remaining failing gates. The stated run + ipc::recv publisher-identity semantics should remain explicit or be covered by a follow-up issue.

@joshuajbouw

Copy link
Copy Markdown
Member

Adversarial review against current main found that this PR's owner-overlay intent is correct, but the current hash-shared runtime identity makes the patch unsafe to merge as scoped.

The kernel deduplicates one mutable live runtime per verified WASM hash and constructs the first runtime under synthetic default. Installing KV, secrets, home, environment, profile, and budgets into that Store can therefore expose default authority to a capsule installed only for a non-default principal and to every other view of the same hash. “First installer owns it” would remain load-order- and restart-dependent.

Tracked as prerequisite #1486. That issue preserves content-addressed bytes and compilation while separating principal-resident Stores, instances, subscriptions, processes, health, and lifecycle by authority scope. It explicitly covers boot, live load, attach-existing-view, derived-agent spawn, reload/restart, unload/delete, stdio MCP, persistent processes, and system-resident services.

After #1486 lands, this PR should be rebased and retain its core change: install the established PrincipalResident(owner) overlays before calling the run export, with a non-default persistence/isolation regression.

joshuajbouw added a commit that referenced this pull request Aug 13, 2026
## Linked Issue

Closes #1486.

Prerequisite for #1380. This PR does not supersede Jamie's run-loop
owner-context work: after this lands, #1380 can be rebased so its owner
overlays are installed into a genuine immutable-UID principal runtime
rather than a hash-shared/default-owned Store.

## Summary

Astrid used the verified WASM content hash as both immutable artifact
identity and mutable runtime identity. Content addressing is correct for
deduplicating bytes and compilation, but sharing an executable Store,
guest memory, subscriptions, processes, readiness, and cancellation
across principals is not a valid authority boundary.

This separates compiled artifact reuse from live runtime ownership.
Identical verified artifacts still compile once, while every principal
receives a distinct executable runtime keyed by immutable `PrincipalUid`
and an incarnation generation. Explicit operator-owned system services
remain intentional singletons.

## Changes

- Add `RuntimeScope`, `RuntimeKey`, and generation-bearing `RuntimeId`;
preserve `WasmHash` as artifact identity and existing wire/source UUID
formats.
- Share verified Wasmtime `Engine`/`Component`/`InstancePre` artifacts
while isolating mutable Stores, instances, pools, guest state, run
tasks, resource tables, and authority overlays per principal.
- Key registry views, dispatcher queues, readiness, health, replacement,
unload, and source resolution by runtime generation so stale work cannot
address a replacement.
- Stage route publication until readiness, retire old routes before
replacement publication, and scope principal-resident subscriptions to
their owner plus system events.
- Make live install/reload prepare, ready, atomically replace, and
retire runtime generations; serialize load/reload/delete lifecycle
admission.
- Restrict `SystemResident` creation and replacement to the
operator/default install root and operator-owned `[[uplinks]]` policy;
workspace config cannot widen that allowlist.
- Isolate stdio MCP identifiers and processes per runtime generation.
Teardown now retries to confirmed manager absence, awaits rmcp transport
termination, and owns full Unix process groups or Windows Job Objects.
- Add rollback and cleanup guarantees for composite engines, publication
failures, process handles, derived principals, and agent deletion.
- Preserve legacy registry/hash/UUID compatibility where unambiguous and
fail closed where old unscoped lookup would cross authorities.

## Impact and sequencing

Existing capsule artifacts, manifests, WIT, and SDK contracts require no
migration. On restart, executable capsules instantiate under corrected
principal authority while compiled code remains deduplicated.

Once this PR lands:

1. Rebase #1380 onto `main`.
2. Retain its run-loop owner-context installation against the new
`Principal(owner_uid)` runtime.
3. Add the non-default run-loop persistence/resource regression
described in #1486.

## Verification

- `cargo test -p astrid-capsule -p astrid-mcp --lib -- --quiet`: 627
capsule tests passed; 124 MCP tests passed with one subprocess fixture
intentionally ignored outside its driver.
- `cargo test -p astrid-kernel --lib -- --quiet`: 313 passed.
- `cargo test -p astrid-events -p astrid-config --lib -- --quiet`: 80
event and 113 config tests passed.
- Real MCP teardown regression completed a handshake, exercised a
TERM-resistant descendant, forced rmcp process-tree termination, and
proved `stop().await` returned only after descendant absence while an
unrelated peer remained available.
- `cargo clippy -p astrid-mcp -p astrid-capsule -p astrid-events -p
astrid-config -p astrid-kernel -p astrid-daemon --all-features
--all-targets -- -D warnings` passed.
- `cargo check -p astrid-mcp --target aarch64-pc-windows-msvc` passed;
Windows Job Object execution remains delegated to Windows CI.
- `cargo fmt --all -- --check`, `git diff --check`, commit signature,
DCO trailer, and repository file-size limits passed.
- Three independent adversarial review tracks covered
security/lifecycle, Wasmtime/MCP teardown, and public compatibility; all
concluded clean after fixes.

The broad workspace suite passed through the changed runtime/kernel/MCP
surfaces; one unrelated pre-existing macOS seatbelt test stalled and was
interrupted rather than treated as evidence for this change.

## AI / Tool Assistance

Assisted-by: Codex: GPT-5

Codex assisted with the runtime-identity implementation, concurrency and
teardown regressions, and adversarial review. Joshua reviewed the design
and changes, directed the authority model and compatibility constraints,
and validated the resulting implementation through the test, clippy,
formatting, cross-compilation, and review passes listed above.

## Checklist

- [x] Linked to an issue
- [x] CHANGELOG.md updated (entry under `[Unreleased]` — or
`[Unreleased]` rolled into a version section for a release PR; not
applicable to docs/CI-only changes)
- [x] I understand every change in this PR and can explain its design,
risks, and validation.
- [x] I reviewed and tested any meaningful tool-generated output
included in this PR.
- [x] Every non-bot, non-merge commit has a matching `Signed-off-by`
trailer.

---------

Signed-off-by: Joshua J. Bouw <[email protected]>
@joshuajbouw
joshuajbouw force-pushed the fix/1197-run-loop-per-principal branch from 673bfe6 to 59614f0 Compare August 14, 2026 01:17
@joshuajbouw joshuajbouw changed the title fix(capsule): install owner per-principal overlays before the run export (#1197, #1224) fix(capsule): bind run loops to typed owner context Aug 14, 2026
@joshuajbouw
joshuajbouw force-pushed the fix/1197-run-loop-per-principal branch from 59614f0 to ce48004 Compare August 14, 2026 14:40
@joshuajbouw
joshuajbouw force-pushed the fix/1197-run-loop-per-principal branch 2 times, most recently from 813e025 to 2d27a98 Compare August 14, 2026 19:07
Install a principal-resident run loop's immutable owner KV, home, secrets, env, profile, budgets, log, and cancellation context before entering the guest run export. Keep explicit system runtimes neutral and preserve unstamped compatibility callers.

Closes astrid-runtime#1224; refs astrid-runtime#1197 and astrid-runtime#1454.

Co-authored-by: Jamie Steiner <[email protected]>

Co-authored-by: Claude Opus 4.8 <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
@joshuajbouw
joshuajbouw force-pushed the fix/1197-run-loop-per-principal branch from 2d27a98 to 3b01eeb Compare August 14, 2026 20:11
@joshuajbouw
joshuajbouw merged commit a10e634 into astrid-runtime:main Aug 14, 2026
30 checks passed
@jvsteiner
jvsteiner deleted the fix/1197-run-loop-per-principal branch August 15, 2026 15:07
jvsteiner added a commit to jvsteiner/astrid that referenced this pull request Aug 15, 2026
A run-loop (#[astrid::run]) capsule hosting a loopback TCP server was pinned to
a single Store, so it handled requests serially — parallel clients queued behind
each other. The guest is single-threaded with blocking host I/O, so cooperative
concurrency inside one instance is impossible; the only route is N instances.

Add `bind_workers` (CapabilitiesDef): a run-loop capsule declaring net_bind and
no host_process runs N worker Stores. Each executes run() and shares ONE bound
listener via a per-capsule registry (Approach B), blocking on accept() — the OS
accept queue load-balances. SO_REUSEPORT was rejected: on macOS it does not
load-balance, it delivers every connection to the most-recent bind.

N=1 is byte-identical to prior behaviour. Interceptors + workers>1 is forced to
1 with a warn, since N subscriptions would double-process every event.

Rebased onto current main rather than cherry-picked: the run-loop lifecycle was
rebuilt underneath the original branch (activation watch channel, mid-run
cancellation racing call_async, run loop as async task). Per worker:

  * activation — one watch::Sender, each worker takes its own subscribe()
    receiver, so a single publish releases all N; a worker cancelled before
    activation returns without ever locking its Store.
  * cancellation — the same cancel_token clone to every worker, each keeping the
    biased select! so a compute-bound guest unwinds per worker.
  * owner context — astrid-runtime#1380's install_run_loop_owner_context is applied to EVERY
    worker Store; a worker without it would serve with no principal authority.
  * run_handles/ready_rxs become Vecs; unload aborts all, wait_ready awaits all.

Three defects fixed that the original branch carried:

  * A wildcard port must never be shared. Port 0 means "any ephemeral port", so
    two such requests are different addresses; keying the registry on the
    requested port conflated them, and a pooled capsule binding port 0 four
    times would have received one socket four times. Caught by the existing
    quota test. Sharing is restricted to concrete ports — what workers bind.
  * The quota bounds sockets, not workers. MAX_ACTIVE_TCP_LISTENERS is 4, so
    charging per worker would let bind_workers=8 exhaust it with a single port.
    The charge is taken only when a socket is created, and listener_count became
    Option<_> so only the binding worker releases it (N slots would otherwise
    underflow the counter N-1 times).
  * `localhost` is normalized to the loopback literal BEFORE the registry key,
    or two workers naming the same address produce two entries and race for one
    OS port.

discovery.rs loses its hand-promotion of bind_workers: astrid-runtime#1381 replaced the
field-by-field merge with an exhaustive destructure that has no `..`, so the new
field could not compile until merge_from handled it.

bind_workers is deliberately NOT a held capability. It grants no authority — it
parameterises an already-granted net_bind. Listing a worker count in held_names
would misreport the security posture in `astrid capsule show` and the audit
trail, so held_names_and_has_agree_when_all_held excludes it by name.

Evidence, not assertion (e2e/concurrency.sh + e2e/fixtures/astrid-capsule-
concurrency, a fixture that blocks 500ms per request):

  bind_workers=1   504 1007 1510 2013 2515 ms   spread 2011ms  serialized
  bind_workers=5   505  505  505  505  505 ms   spread    0ms  parallel
  after daemon stop: connect refused — no worker survived

Teardown is asserted by connecting, not binding: a bind probe is confounded by
TIME_WAIT from the client connections and reported a survivor when lsof showed
nothing listening.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Signed-off-by: Jamie Steiner <[email protected]>
joshuajbouw added a commit to jvsteiner/astrid that referenced this pull request Aug 19, 2026
A run-loop (#[astrid::run]) capsule hosting a loopback TCP server was pinned to
a single Store, so it handled requests serially — parallel clients queued behind
each other. The guest is single-threaded with blocking host I/O, so cooperative
concurrency inside one instance is impossible; the only route is N instances.

Add `bind_workers` (CapabilitiesDef): a run-loop capsule declaring net_bind and
no host_process runs N worker Stores. Each executes run() and shares ONE bound
listener via a per-capsule registry (Approach B), blocking on accept() — the OS
accept queue load-balances. SO_REUSEPORT was rejected: on macOS it does not
load-balance, it delivers every connection to the most-recent bind.

N=1 is byte-identical to prior behaviour. Interceptors + workers>1 is forced to
1 with a warn, since N subscriptions would double-process every event.

Rebased onto current main rather than cherry-picked: the run-loop lifecycle was
rebuilt underneath the original branch (activation watch channel, mid-run
cancellation racing call_async, run loop as async task). Per worker:

  * activation — one watch::Sender, each worker takes its own subscribe()
    receiver, so a single publish releases all N; a worker cancelled before
    activation returns without ever locking its Store.
  * cancellation — the same cancel_token clone to every worker, each keeping the
    biased select! so a compute-bound guest unwinds per worker.
  * owner context — astrid-runtime#1380's install_run_loop_owner_context is applied to EVERY
    worker Store; a worker without it would serve with no principal authority.
  * run_handles/ready_rxs become Vecs; unload aborts all, wait_ready awaits all.

Three defects fixed that the original branch carried:

  * A wildcard port must never be shared. Port 0 means "any ephemeral port", so
    two such requests are different addresses; keying the registry on the
    requested port conflated them, and a pooled capsule binding port 0 four
    times would have received one socket four times. Caught by the existing
    quota test. Sharing is restricted to concrete ports — what workers bind.
  * The quota bounds sockets, not workers. MAX_ACTIVE_TCP_LISTENERS is 4, so
    charging per worker would let bind_workers=8 exhaust it with a single port.
    The charge is taken only when a socket is created, and listener_count became
    Option<_> so only the binding worker releases it (N slots would otherwise
    underflow the counter N-1 times).
  * `localhost` is normalized to the loopback literal BEFORE the registry key,
    or two workers naming the same address produce two entries and race for one
    OS port.

discovery.rs loses its hand-promotion of bind_workers: astrid-runtime#1381 replaced the
field-by-field merge with an exhaustive destructure that has no `..`, so the new
field could not compile until merge_from handled it.

bind_workers is deliberately NOT a held capability. It grants no authority — it
parameterises an already-granted net_bind. Listing a worker count in held_names
would misreport the security posture in `astrid capsule show` and the audit
trail, so held_names_and_has_agree_when_all_held excludes it by name.

Evidence, not assertion (e2e/concurrency.sh + e2e/fixtures/astrid-capsule-
concurrency, a fixture that blocks 500ms per request):

  bind_workers=1   504 1007 1510 2013 2515 ms   spread 2011ms  serialized
  bind_workers=5   505  505  505  505  505 ms   spread    0ms  parallel
  after daemon stop: connect refused — no worker survived

Teardown is asserted by connecting, not binding: a bind probe is confounded by
TIME_WAIT from the client connections and reported a survivor when lsof showed
nothing listening.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Signed-off-by: Jamie Steiner <[email protected]>
Co-authored-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
joshuajbouw added a commit to jvsteiner/astrid that referenced this pull request Aug 20, 2026
A run-loop (#[astrid::run]) capsule hosting a loopback TCP server was pinned to
a single Store, so it handled requests serially — parallel clients queued behind
each other. The guest is single-threaded with blocking host I/O, so cooperative
concurrency inside one instance is impossible; the only route is N instances.

Add `bind_workers` (CapabilitiesDef): a run-loop capsule declaring net_bind and
no host_process runs N worker Stores. Each executes run() and shares ONE bound
listener via a per-capsule registry (Approach B), blocking on accept() — the OS
accept queue load-balances. SO_REUSEPORT was rejected: on macOS it does not
load-balance, it delivers every connection to the most-recent bind.

N=1 is byte-identical to prior behaviour. Interceptors + workers>1 is forced to
1 with a warn, since N subscriptions would double-process every event.

Rebased onto current main rather than cherry-picked: the run-loop lifecycle was
rebuilt underneath the original branch (activation watch channel, mid-run
cancellation racing call_async, run loop as async task). Per worker:

  * activation — one watch::Sender, each worker takes its own subscribe()
    receiver, so a single publish releases all N; a worker cancelled before
    activation returns without ever locking its Store.
  * cancellation — the same cancel_token clone to every worker, each keeping the
    biased select! so a compute-bound guest unwinds per worker.
  * owner context — astrid-runtime#1380's install_run_loop_owner_context is applied to EVERY
    worker Store; a worker without it would serve with no principal authority.
  * run_handles/ready_rxs become Vecs; unload aborts all, wait_ready awaits all.

Three defects fixed that the original branch carried:

  * A wildcard port must never be shared. Port 0 means "any ephemeral port", so
    two such requests are different addresses; keying the registry on the
    requested port conflated them, and a pooled capsule binding port 0 four
    times would have received one socket four times. Caught by the existing
    quota test. Sharing is restricted to concrete ports — what workers bind.
  * The quota bounds sockets, not workers. MAX_ACTIVE_TCP_LISTENERS is 4, so
    charging per worker would let bind_workers=8 exhaust it with a single port.
    The charge is taken only when a socket is created, and listener_count became
    Option<_> so only the binding worker releases it (N slots would otherwise
    underflow the counter N-1 times).
  * `localhost` is normalized to the loopback literal BEFORE the registry key,
    or two workers naming the same address produce two entries and race for one
    OS port.

discovery.rs loses its hand-promotion of bind_workers: astrid-runtime#1381 replaced the
field-by-field merge with an exhaustive destructure that has no `..`, so the new
field could not compile until merge_from handled it.

bind_workers is deliberately NOT a held capability. It grants no authority — it
parameterises an already-granted net_bind. Listing a worker count in held_names
would misreport the security posture in `astrid capsule show` and the audit
trail, so held_names_and_has_agree_when_all_held excludes it by name.

Evidence, not assertion (e2e/concurrency.sh + e2e/fixtures/astrid-capsule-
concurrency, a fixture that blocks 500ms per request):

  bind_workers=1   504 1007 1510 2013 2515 ms   spread 2011ms  serialized
  bind_workers=5   505  505  505  505  505 ms   spread    0ms  parallel
  after daemon stop: connect refused — no worker survived

Teardown is asserted by connecting, not binding: a bind probe is confounded by
TIME_WAIT from the client connections and reported a survivor when lsof showed
nothing listening.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Signed-off-by: Jamie Steiner <[email protected]>
Co-authored-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
joshuajbouw added a commit to jvsteiner/astrid that referenced this pull request Aug 20, 2026
A run-loop (#[astrid::run]) capsule hosting a loopback TCP server was pinned to
a single Store, so it handled requests serially — parallel clients queued behind
each other. The guest is single-threaded with blocking host I/O, so cooperative
concurrency inside one instance is impossible; the only route is N instances.

Add `bind_workers` (CapabilitiesDef): a run-loop capsule declaring net_bind and
no host_process runs N worker Stores. Each executes run() and shares ONE bound
listener via a per-capsule registry (Approach B), blocking on accept() — the OS
accept queue load-balances. SO_REUSEPORT was rejected: on macOS it does not
load-balance, it delivers every connection to the most-recent bind.

N=1 is byte-identical to prior behaviour. Interceptors + workers>1 is forced to
1 with a warn, since N subscriptions would double-process every event.

Rebased onto current main rather than cherry-picked: the run-loop lifecycle was
rebuilt underneath the original branch (activation watch channel, mid-run
cancellation racing call_async, run loop as async task). Per worker:

  * activation — one watch::Sender, each worker takes its own subscribe()
    receiver, so a single publish releases all N; a worker cancelled before
    activation returns without ever locking its Store.
  * cancellation — the same cancel_token clone to every worker, each keeping the
    biased select! so a compute-bound guest unwinds per worker.
  * owner context — astrid-runtime#1380's install_run_loop_owner_context is applied to EVERY
    worker Store; a worker without it would serve with no principal authority.
  * run_handles/ready_rxs become Vecs; unload aborts all, wait_ready awaits all.

Three defects fixed that the original branch carried:

  * A wildcard port must never be shared. Port 0 means "any ephemeral port", so
    two such requests are different addresses; keying the registry on the
    requested port conflated them, and a pooled capsule binding port 0 four
    times would have received one socket four times. Caught by the existing
    quota test. Sharing is restricted to concrete ports — what workers bind.
  * The quota bounds sockets, not workers. MAX_ACTIVE_TCP_LISTENERS is 4, so
    charging per worker would let bind_workers=8 exhaust it with a single port.
    The charge is taken only when a socket is created, and listener_count became
    Option<_> so only the binding worker releases it (N slots would otherwise
    underflow the counter N-1 times).
  * `localhost` is normalized to the loopback literal BEFORE the registry key,
    or two workers naming the same address produce two entries and race for one
    OS port.

discovery.rs loses its hand-promotion of bind_workers: astrid-runtime#1381 replaced the
field-by-field merge with an exhaustive destructure that has no `..`, so the new
field could not compile until merge_from handled it.

bind_workers is deliberately NOT a held capability. It grants no authority — it
parameterises an already-granted net_bind. Listing a worker count in held_names
would misreport the security posture in `astrid capsule show` and the audit
trail, so held_names_and_has_agree_when_all_held excludes it by name.

Evidence, not assertion (e2e/concurrency.sh + e2e/fixtures/astrid-capsule-
concurrency, a fixture that blocks 500ms per request):

  bind_workers=1   504 1007 1510 2013 2515 ms   spread 2011ms  serialized
  bind_workers=5   505  505  505  505  505 ms   spread    0ms  parallel
  after daemon stop: connect refused — no worker survived

Teardown is asserted by connecting, not binding: a bind probe is confounded by
TIME_WAIT from the client connections and reported a survivor when lsof showed
nothing listening.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Signed-off-by: Jamie Steiner <[email protected]>
Co-authored-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
joshuajbouw added a commit that referenced this pull request Aug 20, 2026
…1231) (#1540)

## Linked Issue

Closes #1231

## Summary

A run-loop (`#[astrid::run]`) capsule hosting a loopback TCP server is
pinned to a **single Store**, so it serves connections serially — while
it blocks on one request's upstream I/O, every other connection waits.
The guest is single-threaded with blocking host I/O, so cooperative
concurrency inside one instance is impossible; the only route is N
instances.

`bind_workers` lets such a capsule run N worker Stores, each executing
`run()` against **one** shared bound listener and blocking on `accept()`
— the OS accept queue load-balances. `N = 1` is byte-identical to
current behaviour.

Lands on top of #1380, as set out when #1321 was split into focused PRs.
Rebased onto current `main` rather than cherry-picked: the run-loop
lifecycle was rebuilt underneath the original branch (activation watch
channel, mid-run cancellation racing `call_async`, run loop as an async
task), so this is a reimplementation against that machinery.

## Changes

- **`bind_workers` (`[capabilities]`)** — a run-loop capsule declaring
`net_bind` and no `host_process` runs N worker Stores, clamped to
`instance_pool_size`. Forced to 1 with a warning when the capsule
declares interceptors, since N subscriptions would double-process every
event.
- **Shared listener (Approach B)** — the first worker to `bind_tcp`
binds and inserts into a per-capsule `DashMap<(host, port),
Arc<TcpListener>>`; siblings clone the `Arc`. The bind runs under the
shard lock so racing workers serialize rather than failing `EADDRINUSE`
(macOS sets no `SO_REUSEADDR`). `SO_REUSEPORT` was rejected: on macOS it
does not load-balance, it delivers every connection to the most-recent
bind.
- **Per-worker activation** — one `watch::Sender`; each worker takes its
own `subscribe()` receiver, so a single publish releases all N. A worker
cancelled before activation returns without ever locking its Store.
- **Per-worker cancellation and teardown** — the same `cancel_token`
clone to every worker, each keeping the existing `biased` `select!` so a
compute-bound guest unwinds. `run_handles`/`ready_rxs` become `Vec`s;
`unload` aborts all, `wait_ready` awaits all.
- **Owner context on every worker Store** — #1380's
`install_run_loop_owner_context` is applied to each, not just the first;
a worker without it would serve with no principal authority.
- **A wildcard port is never shared** — port `0` means "any ephemeral
port", so two such requests are different addresses. Keying the registry
on the requested port conflated them, and a pooled capsule binding port
0 four times would have received one socket four times. Caught by the
existing quota test.
- **The listener quota bounds sockets, not workers** —
`MAX_ACTIVE_TCP_LISTENERS` is 4, so charging per worker would let
`bind_workers = 8` exhaust it with a single port. The charge is taken
only when a socket is created, and `TcpListenerSlot::listener_count`
became `Option<_>` so only the binding worker releases it.
- **`localhost` is normalized before the registry key** — otherwise two
workers naming the same address produce two entries and race for one OS
port.
- **`discovery.rs` loses its hand-promotion of `bind_workers`** — #1381
replaced the field-by-field merge with an exhaustive destructure, so
`merge_from` gains one arm and the workaround is deleted.
- **`bind_workers` is deliberately not a held capability** — it grants
no authority, it parameterises an already-granted `net_bind`. Listing a
worker count in `held_names` would misreport the security posture in
`astrid capsule show` and the audit trail, so
`held_names_and_has_agree_when_all_held` excludes it by name.
- **New fixture and harness** —
`e2e/fixtures/astrid-capsule-concurrency` and `e2e/concurrency.sh`.

## Verification

`cargo test -p astrid-capsule -p astrid-capsule-types` — 688 tests, all
green. `cargo fmt --all --check` clean; `cargo clippy --workspace
--all-targets -- -D warnings` clean.

**End-to-end.** `e2e/concurrency.sh` runs the fixture (which blocks 500
ms per request) twice against an isolated `ASTRID_HOME`, changing one
manifest line between runs:

```
astrid#1231: serial baseline (bind_workers = 1)
  elapsed ms: 504 1007 1510 2013 2515
astrid#1231: concurrent (bind_workers = 5)
  elapsed ms: 505 505 505 505 505
astrid#1231: asserting
  baseline spread   2011ms  (>= 500ms, serialized)
  concurrent spread 0ms  (<  500ms, parallel)
astrid#1231: asserting every worker died
  connect refused on 18231 — no worker survived the stop
astrid#1231: PASS
```

The baseline is the issue's own symptom reproduced: each client queues
~505 ms behind the one before it.

To run it:

```sh
cargo build --bin astrid --bin astrid-daemon
(cd e2e/fixtures/astrid-capsule-concurrency && cargo build --release)
./e2e/concurrency.sh
```

**Not wired into CI** — it needs a built runtime, a wasm fixture and a
live daemon, and the existing `wasm_e2e` fixtures are prebuilt artifacts
absent from the tree. Happy to wire it up if you want that; it seemed
like its own change.

Two deliberate choices in the harness, both of which bit me first:

- It asserts **completion spread**, not absolute wall-clock. A shared
runner can be slow without being serial, and "all five finished within
one request-time of each other" is the property under test.
- Teardown is asserted by **connecting, not binding**. A bind probe is
confounded by `TIME_WAIT` from the client connections and reported a
surviving worker when `lsof` showed nothing listening at all.

**Unit coverage:**

| Claim | Test |
|---|---|
| Workers share one bound socket |
`worker_stores_dedupe_onto_one_bound_socket` |
| `localhost` and the literal are one address |
`localhost_and_loopback_literal_share_one_registry_entry` |
| Quota charged once, released once | the two above, plus the existing
quota test |
| A wildcard port never shares |
`tcp_listener_quota_is_independent_and_released_on_drop` (existing) |
| `bind_workers` merges component→root | `merge_from_unions_every_field`
|
| A silent component does not erase it |
`merge_from_preserves_root_bind_workers_when_component_is_silent` |

## AI / Tool Assistance

Assisted-by: CLAUDE:claude-opus-4.8

Substantial. The rebase-and-reimplementation, the fixture capsule, the
harness, the tests and this description were all produced with Claude,
working from a stale branch (`feat/capsule-bind-tcp-on-main`, authored
against old `main`) that no longer applied.

How it was reviewed and validated:

- **Every claim in Verification was executed, not asserted.** The
numbers above are real output from this branch on an arm64 macOS host,
not reconstructed.
- **Three defects were found by tests rather than by reading**, and each
is called out in Changes above: the wildcard-port sharing bug (caught by
the *existing* quota test), the per-worker quota charge, and the
`localhost` registry key. The first would have shipped.
- **Two mistakes were made and corrected during the work.** A scripted
conflict resolution silently deleted #1380's
`install_run_loop_owner_context`; caught by checking the symbol still
existed, then the whole file was redone bottom-up. And the first
teardown assertion was a false positive from `TIME_WAIT`, which would
have sent a reviewer hunting a bug that does not exist.
- **The three exhaustiveness guards in this codebase did real work** —
`merge_from`'s no-`..` destructure, `merge_from_unions_every_field`'s
fully-populated literals, and `held_names_and_has_agree_when_all_held`.
Each forced a decision rather than letting the change slide through, and
the `held_names` one is recorded in the code where the next person will
hit it.

I can explain every hunk, and the two merge decisions I would most want
scrutinised are the per-worker owner context and the per-worker
activation subscribe.

## Checklist

- [x] Linked to an issue
- [x] CHANGELOG.md updated (entry under `[Unreleased]`)
- [x] I understand every change in this PR and can explain its design,
risks, and validation.
- [x] I reviewed and tested any meaningful tool-generated output
included in this PR.
- [x] Every non-bot, non-merge commit has a matching `Signed-off-by`
trailer.

---------

Signed-off-by: Jamie Steiner <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
Co-authored-by: Joshua J. Bouw <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run-loop (#[astrid::run]) capsules receive no per-principal resources: operator env overlay and home:// fs both fail

2 participants