Bump LMDB mapsize on non-macOS, use generic builder for urbit #2344
Conversation
This reverts a revert, commit 1daaf04, in order to fix the problems that led to it being reverted.
Ensures that large log limit Vere builds are labelled as such, and are thus easily identifiable.
46a438f introduced use of the stdenv generic builder instead of an explicit builder.sh script. This was reverted in 1daaf04 as it proved to cause problems when building on some Linux variants. This fixes the problem encountered on Linux by calling the standard configure script with bash in an explicit configurePhase.
Moves the secondary URBIT_VERSION assignment to a different hunk to avoid undesirable patching conflicts.
|
I don't feel like I have enough context on this. This is creating another executable that supports larger log files? We're using a patch file to have a variant which uses this larger log file? What are the implications of this? Can you run this executable on an existing log file? Can you run the old variant on a large log file? If you can't do some of these things, what happens when you do? How does this interact with King Haskell? Should this include a change to KH as well? If this is a problem worth solving now, should we instead just dig in and figure out how to grow the log as needed? |
|
How does this affect the release builds? They have a separate derivation IIRC. |
Yes and yes. The binary is only intended for use on ~zod on mainnet. The motivation here is that ~zod now regularly exceeds the 40G limit in every era. It's presently running a Vere with a 60G limit as-is -- it's just that this binary was assembled in hacky firefighting fashion (i.e. by tweaking lmdb.c and building a new binary on-the-box). This has had to be done now several times, IIRC, so if we need special-case binary with a higher log limit going forward, it's desirable to be able to produce it mechanically. (Come to think of it: I don't know why the 40G default limit was chosen instead of 60G, and if we can't just bump the default limit in Vere to 60G or even higher. @joemfb?)
Yeah, you can run this on a log that was originally produced by a binary with the 40G limit. I can't comment on what happens if you try to run a >=40G log with a <40G binary, but I presume it's something along the lines of what @ixv reported previously (via email):
Continuing:
Potentially, but in the sense of this change, only if/when we're going to run KH on ~zod. If there's no reason we can't just bump the default limit to 60G or higher, we should probably just do this across the board.
Possibly, but it's probably not too pressing. At present it's for a one-off case; I don't think any ships other than ~zod tend to bump up against the 40G limit yet. |
They're unaffected -- this just provides a new |
|
IIRC, the only issues with setting a log size > 40GB have been on MacOs. Maybe we could just #ifdef the larger limit directly in the source? /cc @eglaysher |
|
I don't see any issue with just ifdefing a larger size on OSes other than Mac. |
|
Yeah that works for me. I'll tweak this to do that, and also preserve the use of the generic builder (instead of builder.sh) as that at least makes it really easy to override the derivation in the future if we need to. |
Per discussion in #2344, there doesn't seem to be any problem in raising the default LMDB mapsize on non-mac operating systems.
Made unnecessary by 578593d45.
|
Note that this CI failure is just timeout(s) due to #2322, and can be ignored. |
* origin/jt/urbit-large-log: build: remove urbit-large-log-limit derivation vere: use 40G mapsize only on macOS, 60G otherwise build: better vere label patch build: add configurePhase for urbit derivation build: label large-log-limit vere version Revert "Revert "Merge branch 'jt/urbit-large-log' (#2211)"" Signed-off-by: Jared Tobin <[email protected]> (cherry picked from commit be2c07d)
(Update):
Per discussion in the comments, it seems fine to just
ifdefthe log limit size appropriately, so I've done that and removed the large-log-limit derivation. The use of the generic builder for the urbit derivation is retained.From #2211:
I wound up reverting this because it proved to mess up builds on Linux.
The problem turned out to be in the
./configurecalled by the implicitconfigurePhasewhen switching to the generic stdenv builder. That usesshby default, IIRC, soconfigureneeds to be called withbashinstead.