urbit-v1.14-rc1

Arvo 417K
Vere 1.14-rc1

This release implements demand paging for the snapshot, and improves
efficiency and error handling in snapshot updates.

Release Notes

- Demand paging is implemented with a file-backed, copy-on-write mapping of
  the home-road heap (north.bin). This reduces effective memory usage and
  improves performance under memory pressure in many situations.
  Demand paging is on by default (where supported: everywhere but Windows),
  and can be disabled with --no-demand.
- Snapshot updates involve far fewer syscalls thanks explicitly-positioned
  I/O (half as many in patch application) and coalesced memory protection
  changes (from one per page to a fixed upper bound of five).
- I/O errors and partial success are retried where possible, reducing
  unnecessary fatal errors and improving error messages.
- The guard page implementation has been simplified and corrected to prevent
  that premature bail:meme.
- The vendored libwhereami dependency has been updated, fixing the OpenBSD build.

Contributions:

Joe Bryan (46):
      u3: restages #6062 for release
      u3: misc cleanup of snapshot system
      u3: coalesces memory protection when loading a snapshot
      u3: refactors _ce_loom_blit_north(), replacing loop with a single read
      u3: coalesces memory protection when saving a snapshot
      u3: implements full demand paging for the home-road heap
      u3: adds printf ("live: mapped ...") for demand paging
      vere: adds --no-demand argument to disable demand paging
      vere: supports --no-demand for all relevant subcommands
      Merge tag 'urbit-v1.13' into jb/restage/pread-all
      Merge tag 'urbit-v1.13' into jb/restage/demand
      Merge branch 'jb/restage/pread-all' into jb/release/vere
      u3: updates memory protections and page metadata when moving guard page
      Merge branch 'jb/release/vere' into jb/restage/demand
      u3: document snapshot system invariants
      u3: cleanup comments about snapshot system limitations
      u3: adds new, batch page tracking implementations + tests
      u3: enforce snapshot invariants on segment truncation
      u3: switch page size constants to macros to avoid VLAs
      Merge pull request #6108 from urbit/philip/willoc-comment
      Merge pull request #6121 from urbit/ja/openbsd-7.2-fix
      Merge pull request #6125 from urbit/jb/ward-safely
      Merge pull request #6126 from urbit/jb/snapshot-invariants
      Merge branch 'jb/restage/demand' into jb/release/vere
      u3: protect guard page if necessary after remapping loom
      u3: protect guard page if necessary after reprotecting loom
      Merge pull request #6127 from urbit/jb/guard-always-again
      u3: updates guard-page assertion to account for variable loom sizes
      test: re-disables ames decryption test to avoid bail:evil in ci
      test: re-disables failing grq test
      test: renames nock-tests to meme-tests
      u3: removes u3e_foul()
      u3: stop using u3P.dir_c outside of events.c
      u3: move directory creation out of events.c
      u3: automatically place the guard page in u3e_live()
      u3: simplifies road stack adjustment after snapshot load
      u3: adds snapshot facade to manage.c
      u3: simplifies u3e_live() signature
      u3: fixes compile-time conditionals around guard page impl
      u3: refactors loom protect-south page-length calculation
      u3: factors out individual page protections
      u3: reimplements guard page w/out posts
      u3: adds facade for u3e_fault(), refactors error handling
      u3: refactors u3e_save() facade and watermark calculations
      u3: adds facade for u3e_ward(), refactors watermarks
      u3: adds comments to road pave implementations

James Acklin (1):
      vere: updates whereami dep to support OpenBSD

Philip Monk (1):
      u3: add comment explaining willoc logic

This PR is a forward-port / rewrite of the demand-paging implementation
from v1.14 (see urbit/urbit#6063, urbit/urbit#6126, urbit/urbit#6127,
and urbit/urbit#6152). The original scope has been decreased, and the
implementation simplified: i/o errors are not retried, the dirty page
bitmap is manipulated with much simpler code, page offsets/pointers are
calculated with macros, &c. There are additional layers of snapshot
validation for updates (controlled at compile time, always on as of this
PR); clean pages are compared to disk both before and after update.
(This validation should stay on for pre-release testing, and possibly
for initial release as well.)

This PR has been tested extensively on live ships; the corruption issues
that plagued v1.14 cannot be reproduced.

Fixes #188. Supersedes #277. (Was previously opened as #401, but a typo
in the branch name was preventing updates.)