perf(storage): rewrite hot cache and volume WAL overlay - #1562

Linked Issue

Closes #1561

Summary

Rewrite of the owner-scoped KV hot cache and single-sync transaction WAL onto post-#1535 AstridVolume. The previous codex/hot-kv-read-cache commits conflicted with the owned-filesystem cut; this head is a new implementation on current main (3f82d81e) that keeps the issue's semantics: bounded conditional batches, governed hot reads, one WAL sync per accepted group, and committed state served from a pending overlay before canonical arena/root fold.

WAL bytes live in the transactions.wal volume region rather than a host PathBuf. New WAL publication is opt-in (TransactionWalPolicy; default remains the legacy arena-then-root path). Recovery still replays an existing WAL even when new writes are disabled. The existing KV batch API is reused rather than duplicated.

This replaces the conflicting pre-#1535 draft on this PR number. Head SHA: a4e52546.

Changes

• Owner-scoped point-read cache (KvReadCacheConfig / KvReadCacheCapacity), disabled unless an embedding opts in and reserves charge
• Streaming ASTWAL2 grammar, writer, scan, replay, and pending overlay under engine/durable/wal/
• WAL publication through DurableEnginePolicy::with_transaction_wal; WAL file is a volume region (transactions.wal)
• Recovery, snapshots, usage, staging, compaction, and close observe or drain pending WAL state
• Canonical object/root formats, quota accounting, one-owner rejection, and unsupported-backend fallback unchanged
• Changelog entry under [Unreleased]

Verification

• cargo test -p astrid-storage --lib --locked — 746 passed (storage library suite on this head)
• cargo fmt --all -- --check
• git diff --check

Not re-run on this rewrite: the original SurrealKV exact-durability comparison (~0.509x to ~0.947x of SurrealKV on the development machine). That measurement belonged to the pre-#1535 branch. This PR preserves the cache/batch/WAL semantics and ships the volume-backed rewrite; it does not claim a fresh SurrealKV number until that bench is re-run against AstridVolume.

Test Plan

• cargo test -p astrid-storage --lib --locked
• cargo clippy -p astrid-storage --all-targets --all-features --locked -- -D warnings
• Focused WAL / overlay / crash-replay / hot-cache / batch tests under crates/astrid-storage
• CI: Test (macos/ubuntu), Clippy, MSRV 1.95.0, Wasm portability, Public API diff

AI / Tool Assistance

Assisted-by: OpenAI Codex: Grok 4.6

Codex reimplemented the cache and WAL overlay against AstridVolume in an isolated worktree (/tmp/astrid-1562-volume-wal-cache), including tests and the changelog entry. Human owns review, merge, and any re-benchmark against SurrealKV.

How it was reviewed:

• Semantics of storage: add atomic KV batches and single-sync WAL publication #1561 were treated as frozen: one-owner batches, overlay-before-fold, explicit WAL versioning, opt-in publication.
• The previous PR commits were not cherry-picked; this is a rewrite because feat(storage): mount authoritative owner filesystems on macOS, Linux, and Windows #1535 changed the durable file model.
• Performance numbers from the old branch are not restated as evidence for this head.

Checklist

• Linked to an issue
• CHANGELOG.md updated (entry under [Unreleased])
• I understand every change in this PR and can explain its design, risks, and validation.
• I reviewed and tested any meaningful tool-generated output included in this PR.
• Every non-bot, non-merge commit has a matching Signed-off-by trailer.