feat(storage): mount authoritative owner filesystems on macOS, Linux, and Windows - #1535
Conversation
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Bind the graphical Home and personal renderer model to the existing resource, storage, and execution-provider architecture. Treat semantic scenes as untrusted projections, Spaces as composition rather than authority, and refresh the storage integration order around PRs #1535 and #1562. Refs #1564 Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Verification record and Rust-quality after-passThis is an unusually large change: at the current head ( What was actually measuredThe implementation was developed and reviewed as a sequence of explicit behavioral claims:
At this exact head, GitHub reports 35 passing checks, 2 intentionally skipped OCI attestation jobs, 0 pending checks, and 0 failures. This was measured from the PR check suite itself, not inferred from local output. In particular, both Runtime CLI + HTTP sweep and Published v0.10.4 Linux upgrade passed on GitHub-hosted runners. That evidence gives us meaningful confidence in the behaviors exercised. It does not prove that 80,577 changed lines are architecturally clean, free of unreachable code, exhaustively tested, or ideal Rust. Required Rust-quality after-passThe next pass should be structural and evidence-producing, not cosmetic:
This PR has been carefully exercised against real runtime, release-upgrade, restart, failure, and platform behavior. The remaining after-pass is how we ensure that the implementation is also maintainable and idiomatic—not merely behaviorally green today. |
Replace the path-included principal-home and legacy-barrier siblings with directory modules so rustc, rustfmt, and Clippy own each unit. Keep the crate-facing migration APIs and durable receipt bytes unchanged. Name the barrier host-path helper `host_fs` so it does not shadow `std::fs`. Verified: cargo test --offline -p astrid-kernel --lib principal_home_migration (8 passed); cargo test --offline -p astrid-kernel --lib legacy_migration_barrier (23 passed). AI-assisted: Codex (Grok 4.6) performed the module lift and focused tests. I reviewed the resulting module boundaries and validation. Signed-off-by: Joshua J. Bouw <[email protected]>
Turn the path-included workspace_* siblings into a directory module and move the public branch-store API out of the leftover parent file. Keep the crate re-exports and durable branch encodings unchanged. Verified: cargo test --offline -p astrid-storage --lib workspace (22 passed). AI-assisted: Codex (Grok 4.6) performed the module split and focused tests. I reviewed the resulting module boundaries and validation. Signed-off-by: Joshua J. Bouw <[email protected]>
Give durable receipt fields validated NewTypes for schema, blake3 digests, and byte/entry/page counts. Keep the JSON shape identical so existing receipts still deserialize, and reject empty, uppercase, or prefixed inventory digests at the type boundary. The crate-facing PrincipalHomeSourceIdentity snapshot remains string/u64 for compatibility. Verified: cargo test --offline -p astrid-kernel --lib principal_home_migration (15 passed). AI-assisted: Codex (Grok 4.6) added the typed receipt edge and focused rejection tests. I reviewed the durable encoding and validation. Signed-off-by: Joshua J. Bouw <[email protected]>
Move leftover path-included storage-migration files into the existing invite and pair-token directories so rustc owns those modules normally. Keep the crate-facing token APIs and durable encodings unchanged. Verified: cargo test --offline -p astrid-kernel --lib invite (28 passed); cargo test --offline -p astrid-kernel --lib pair_token (18 passed). AI-assisted: Codex (Grok 4.6) performed the module lift and focused tests. I reviewed the resulting module boundaries and validation. Signed-off-by: Joshua J. Bouw <[email protected]>
Replace the path-included hosted open/reclaim/test files with a directory module so rustc owns the hosted volume implementation normally. Keep the HostedFileVolume contract and durable encodings unchanged. Verified: cargo test --offline -p astrid-storage --lib hosted (14 passed). AI-assisted: Codex (Grok 4.6) performed the module lift and focused tests. I reviewed the resulting module boundaries and validation. Signed-off-by: Joshua J. Bouw <[email protected]>
Give ledger source snapshots a validated digest and count pair so a present source cannot carry digest `absent`, and an absent source cannot carry a real digest or a non-zero inventory. Keep the durable JSON fields unchanged. Hosted path snapshots now accumulate counts in a private builder and only construct SourceIdentity after the blake3 digest exists. Verified: cargo test --offline -p astrid-kernel --lib legacy_migration_barrier (27 passed). AI-assisted: Codex (Grok 4.6) added the typed source identity and focused rejection tests. I reviewed the durable encoding and validation. Signed-off-by: Joshua J. Bouw <[email protected]>
Turn the remaining production #[path] siblings into ordinary directory modules: durable format, durable cache, process-broker identity, and admin env handlers. Keep crate-facing APIs unchanged. Verified: cargo test --offline -p astrid-storage --lib durable (165 passed, 3 ignored); cargo test --offline -p astrid-kernel --lib storage_mount (16 passed). Five admin enforcement tests timed out under parallel load and passed serially. AI-assisted: Codex (Grok 4.6) performed the module lift and focused tests. I reviewed the resulting module boundaries and validation. Signed-off-by: Joshua J. Bouw <[email protected]>
Use schema Display and split count addition so Clippy does not treat typed receipt helpers as dead on the library build. Verified: cargo clippy --offline -p astrid-kernel -p astrid-storage --all-targets --all-features -- -D warnings. AI-assisted: Codex (Grok 4.6) fixed the dead-code surface after the typed receipt refactor. I reviewed the resulting type API. Signed-off-by: Joshua J. Bouw <[email protected]>
Keep `blake3:<hex>` source identities intact so distro-init discard proofs still bind to the inventoried digest. Host-path snapshots remain bare hex. Also move the snapshot constructor out of the barrier facade so the module stays under the source cap. Verified: cargo test --offline -p astrid-kernel --lib legacy_migration_barrier (28 passed); cargo clippy --offline -p astrid-kernel --all-targets --all-features -- -D warnings. AI-assisted: Codex (Grok 4.6) diagnosed the published v0.10.4 upgrade failure, preserved prefixed digests, and added a ledger-shape regression. I reviewed the durable encoding and validation. Signed-off-by: Joshua J. Bouw <[email protected]>
Store destination proofs as a validated transparent string so unknown prefixes and newlines cannot enter the ledger, without rewriting durable JSON. Keep blake3: prefixes intact so distro-init discard proofs still bind. Drop leftover unused workspace imports and a redundant source digest helper. Verified: cargo test --offline -p astrid-kernel --lib legacy_migration_barrier (33 passed); cargo clippy --offline -p astrid-kernel --all-targets --all-features -- -D warnings. AI-assisted: Codex (Grok 4.6) added DestinationProof, rejection tests, and the small dead-code cleanup. I reviewed the durable encoding and validation. Signed-off-by: Joshua J. Bouw <[email protected]>
Re-snapshot each capsule env and secret path against the frozen ledger identity immediately before import so a same-uid writer cannot swap bytes in the boot window. Last-close now marks a projection closing under the cache lock, refuses remount while teardown is in flight, and treats a crashed provider as stopped only after the child is reaped and the control endpoint is dead. Verified: cargo test --offline -p astrid-kernel --lib legacy_migration_barrier storage_mount process_stop (54 passed); cargo clippy --offline -p astrid-kernel --all-targets --all-features -- -D warnings. AI-assisted: Codex (Grok 4.6) implemented the two GLM AMENDs and regressions. I reviewed the fail-closed import and teardown paths. Signed-off-by: Joshua J. Bouw <[email protected]>
Apply cargo fmt to the DestinationProof helpers so CI format check matches the local rustfmt pass. Verified: cargo fmt --all -- --check. AI-assisted: Codex (Grok 4.6) applied rustfmt. I reviewed the formatting-only diff. Signed-off-by: Joshua J. Bouw <[email protected]>
## Linked Issue Closes #1561 ## Summary Rewrite of the owner-scoped KV hot cache and single-sync transaction WAL onto post-#1535 `AstridVolume`. The previous `codex/hot-kv-read-cache` commits conflicted with the owned-filesystem cut; this head is a new implementation on current `main` (`3f82d81e`) that keeps the issue's semantics: bounded conditional batches, governed hot reads, one WAL sync per accepted group, and committed state served from a pending overlay before canonical arena/root fold. WAL bytes live in the `transactions.wal` volume region rather than a host `PathBuf`. New WAL publication is opt-in (`TransactionWalPolicy`; default remains the legacy arena-then-root path). Recovery still replays an existing WAL even when new writes are disabled. The existing KV batch API is reused rather than duplicated. This replaces the conflicting pre-#1535 draft on this PR number. Head SHA: `a4e52546`. ## Changes - Owner-scoped point-read cache (`KvReadCacheConfig` / `KvReadCacheCapacity`), disabled unless an embedding opts in and reserves charge - Streaming ASTWAL2 grammar, writer, scan, replay, and pending overlay under `engine/durable/wal/` - WAL publication through `DurableEnginePolicy::with_transaction_wal`; WAL file is a volume region (`transactions.wal`) - Recovery, snapshots, usage, staging, compaction, and close observe or drain pending WAL state - Canonical object/root formats, quota accounting, one-owner rejection, and unsupported-backend fallback unchanged - Changelog entry under `[Unreleased]` ## Verification - `cargo test -p astrid-storage --lib --locked` — 746 passed (storage library suite on this head) - `cargo fmt --all -- --check` - `git diff --check` **Not re-run on this rewrite:** the original SurrealKV exact-durability comparison (~0.509x to ~0.947x of SurrealKV on the development machine). That measurement belonged to the pre-#1535 branch. This PR preserves the cache/batch/WAL semantics and ships the volume-backed rewrite; it does not claim a fresh SurrealKV number until that bench is re-run against `AstridVolume`. ## Test Plan - `cargo test -p astrid-storage --lib --locked` - `cargo clippy -p astrid-storage --all-targets --all-features --locked -- -D warnings` - Focused WAL / overlay / crash-replay / hot-cache / batch tests under `crates/astrid-storage` - CI: Test (macos/ubuntu), Clippy, MSRV 1.95.0, Wasm portability, Public API diff ## AI / Tool Assistance Assisted-by: OpenAI Codex: Grok 4.6 Codex reimplemented the cache and WAL overlay against `AstridVolume` in an isolated worktree (`/tmp/astrid-1562-volume-wal-cache`), including tests and the changelog entry. Human owns review, merge, and any re-benchmark against SurrealKV. How it was reviewed: - Semantics of #1561 were treated as frozen: one-owner batches, overlay-before-fold, explicit WAL versioning, opt-in publication. - The previous PR commits were not cherry-picked; this is a rewrite because #1535 changed the durable file model. - Performance numbers from the old branch are **not** restated as evidence for this head. ## Checklist - [x] Linked to an issue - [x] CHANGELOG.md updated (entry under `[Unreleased]`) - [x] I understand every change in this PR and can explain its design, risks, and validation. - [x] I reviewed and tested any meaningful tool-generated output included in this PR. - [x] Every non-bot, non-merge commit has a matching `Signed-off-by` trailer. --------- Signed-off-by: Joshua J. Bouw <[email protected]>
Linked Issue
Closes #1534
Related to #1391 and allocation-policy issue #1539.
Summary
Make Astrid storage the authoritative filesystem and durable state substrate for principal, fleet, and system-owner views, expose governed views as native mounts on macOS, Linux, and Windows, and fully migrate the released v0.10.4 layout before any legacy authority is retired.
Mounted files are not served from mutable alias-keyed host homes. Principal
home://resolves to that agent's immutable UID-ownedhomesubtree; Fleet processes receive only the fixedsharedsubtree; workspace processes receive an explicit durable branch/prefix view. Files remain immutable content DAGs and acknowledged namespace mutations atomically advance the selected owner root.Hosted installations realize the path-free
AstridVolumecontract as one locked, checksummedvar/astrid.volume. Mounting does not provision an owner, principal, quota, or allocation: it projects an already-authorized storage view.Changes
AstridVolumecontract and hosted single-file volume with exclusive locking, bounded recovery, checksummed records, crash-recoverable physical reclaim, and atomic metadata transactions.Released-layout migration
Kernel startup captures whether the home is fresh, released-layout, or existing-v2 before
AstridHome::ensure, acquires the singleton lock, and runs one global migration barrier before mounts or ordinary service authority become available.The barrier inventories every supported legacy source with bounded, no-follow, owner-controlled, same-device, and mount-boundary validation. Private sources retain owner-only permissions; released SurrealKV and immutable capsule-package trees admit their historical modes only when every entry is current-user-owned and not group/world writable. Pre-authority capsule installs are snapshotted once through the legacy verifier, including exact manifest, capability, and WASM pins, before their deterministic durable package digest is bound. It migrates and verifies:
state.dband the intermediate principal directory store intoastrid.volume;homesubtrees;Each component records a canonical source identity and destination proof. Destination readback must match before the global ledger becomes complete. Retirement then revalidates the exact receipted source and removes it bottom-up through no-follow, mount-safe helpers with crash-resumable semantics. Missing receipts, source mutation, alias deletion/reuse, reappeared legacy roots, malformed or oversized markers, redirects, special files, nested mounts, insufficient capacity, or unsupported non-default audit/KV/token sources fail closed before mutation or service.
AstridHome::ensureno longer performs recursive pre-barrier retirement. Existing-v2 startup also requires valid fresh-layout or released-migration provenance and resumes only receipt-bound finalization. Agent deletion cannot sweep unretired legacy authority.The published-v0.10.4 Linux fixture now seeds multiple aliases plus ordinary home data, profile, env, secrets, capsules and authority, audit, logs, distro locks, init lock, invites, pairing, revocations, temporary state, and post-cutover writes. It asserts destination proofs/readback, source absence, UID isolation, and restart persistence.
Ownership model
Platform status
/dev/fuseE2E, and published-v0.10.4 upgrade E2E.Verification
Local verification across the integrated series; changed migration, audit, and capsule-install scopes were rerun on head
6b1e2993:cargo clippy --workspace --all-features --all-targets --locked -- -D warningscargo test --workspace --locked— complete workspace tests and doctests passedcargo test -p astrid-kernel --lib --locked— 385 passedcargo test -p astrid-capsule --lib --locked— 664 passedcargo test -p astrid-storage --all-features --lib secret --locked— 28 passed; default constructor verified KV-onlycargo test -p astrid-audit --lib --locked— 45 unit tests passed, including an exact empty released-audit receipt and lock-release regressioncargo test -p astrid-capsule-install --offline— 76 unit + 13 integration passedscripts/check-wasm-portability.sh— passedcargo audit --file Cargo.lock— passed after updatingh2to the patched0.4.16Post-adversarial verification through
3bbc7724cargo test --offline -p astrid-kernel --lib --no-fail-fast -- --quiet— 404 passedcargo test --offline -p astrid-storage --lib --no-fail-fast -- --quiet— 711 passed, 7 ignoredcargo clippy --offline -p astrid-audit -p astrid-capsule-install -p astrid-gateway -p astrid-kernel -p astrid-storage --all-targets --all-features -- -D warnings— passedcargo fmt --all -- --checkandgit diff --check— passedSigned-off-bytrailers and valid GPG signaturesGitHub Actions was green on the prior integrated head
6b1e29937101a6a1954caa9bba1b0969a6fdc1fe: 35 checks passed and the two platform-inapplicable OCI attestation checks were skipped as designed. GitHub Actions was green on this after-pass headddb2cee21c074a1c01431d63ec1873e2ac89e9fe: 35 checks passed and the two platform-inapplicable OCI attestation checks were skipped as designed.AI / Tool Assistance
Assisted-by: OpenAI Codex: GPT-5.6
Codex assisted with architecture synthesis, implementation, adversarial review, tests, documentation, and validation. The human contributor reviewed the design and remains responsible for the code, evidence, and review responses.
Checklist
[Unreleased]Signed-off-bytrailer and valid GPG signatureNote
High Risk: this PR spans storage authority, released-layout migration, kernel boot ordering, native provider handoff, and multi-platform packaging. Mergeability depends on the full required CI matrix, including the real Linux upgrade/FUSE jobs and protected live FSKit certification policy.
Latest CI repair
-D warnings; Runtime E2E progressed through the previously failing registry live activation.Rust-quality after-pass through
ddb2cee2#[path]siblings into directory modules for principal-home migration, the layout barrier, workspace branches, invites/pair tokens, hosted volumes, durable format/cache, process-broker identity, and admin env handlers.blake3:<hex>source digests so distro-init discard proofs still bind.cargo clippy --offline -p astrid-kernel -p astrid-storage --all-targets --all-features -- -D warnings.Assisted-by: OpenAI Codex: Grok 4.6