caddytls: stop mutating the live cache's options on reload - #7961
caddytls: stop mutating the live cache's options on reload#7961SillyZir wants to merge 2 commits into
Conversation
CacheOptions are immutable once a cache is created: the maintenance goroutine's tickers are created at startup, so SetOptions on a live cache both races with maintenance reading the options and silently never applies new intervals to the running tickers. Replace the cache instead of mutating it - the new cache starts empty with the new options and is repopulated by certificate loading during provisioning, while handshakes from the outgoing config generation keep reading the old cache until its maintenance loop is stopped (outside the lock, same idiom as the cleanup path).
|
I reworked this around the immutable-options approach: when capacity or either interval changes, the cache is replaced rather than mutated, and the old cache is stopped after releasing the lock. While investigating, I also found that changing the intervals through SetOptions on a live cache wouldn’t actually update the running maintenance tickers, since they’re created from the initial options. The tradeoff is that an option-changing reload rebuilds the cache and reloads the certificates, but that only happens when those options actually change. I also added regression coverage for initial creation, reuse with unchanged options, and replacement when options change. |
| if oldCache := provisionCertCache(t, cacheOpts); oldCache != nil { | ||
| // blocks until the old cache's maintenance goroutine exits; | ||
| // done outside the lock so handshakes are never stalled behind | ||
| // it (same idiom as the cleanup path below) | ||
| oldCache.Stop() |
There was a problem hiding this comment.
The replacement cache is published globally and the old cache is stopped before the rest of provisioning succeeds?
There was a problem hiding this comment.
Yes, that’s correct. I traced the reload path and reproduced the failure case. The replacement cache is published and the old cache is stopped before the remainder of Provision succeeds, so there is no rollback if a later provisioning step fails.
In that case Caddy keeps the old configuration active, and the active TLS app remains bound to the old cache, but that cache’s maintenance goroutine has already stopped. The new cache is globally assigned but is not serving the active configuration. This doesn’t cause an immediate outage, but it can leave the active configuration without renewal/OCSP maintenance until a later successful reload.
I also ran the same reproducer against the base commit (0cf03d3). The reload still fails, but there is no adverse cache state on base: the existing cache is neither replaced nor stopped, the global cache remains unchanged, and the live cache remains coherent with its own certificate and maintenance still running. So the replacement-and-stop behavior is introduced by this PR.
The reproduced case requires an actual cache capacity/interval change followed by a later provisioning failure; unchanged-option reloads are unaffected.
I think the safer direction is to keep the replacement cache app-scoped during Provision, publish the new cache from Start() once provisioning has completed, and move the irreversible Stop() of the old cache into the outgoing app’s post-commit cleanup. That gives Caddy a reversible publication step while ensuring we don’t destroy the previously committed cache before the new configuration is committed.
I also found that unset and explicitly-defaulted intervals can currently compare differently and cause an unnecessary replacement. I reproduced that on the PR head and confirmed it is absent from the base, so I think that normalization belongs in this PR as well.
Assistance Disclosure
This patch was generated by an engineering system operated by me; I reviewed and validated the change and I am accountable for it.
Fixes the data race between
SetOptionson the reused certificate cache andcertNeedsRenewalin renewal workers during provisioning (#7897).Reworked per review: cache options are now treated as immutable. On every
config reload the TLS app previously called
SetOptionson the sharedCertMagic cache, mutating options that renewal workers read concurrently.
This change never mutates a live cache:
provisionCertCachehelper (undercertCacheMu) reuses theexisting cache when the effective options (capacity, renew/OCSP
intervals) are unchanged — the common reload path touches nothing.
mutated: the new cache starts empty and is repopulated by certificate
loading during provisioning, while handshakes on the outgoing config
generation keep reading the old cache until
Stop()— called outsidethe lock, same idiom as the existing cleanup path.
GetConfigForCertdereferences a mutex-guarded current-app pointer, soreusing the cache across reloads doesn't require touching its options
just to refresh the callback.
SetOptionsis no longer called anywhere, so the racing path is goneentirely rather than narrowed.
Notes for reviewers:
(empty cache repopulated during provisioning). That's the cost of
immutability; it only occurs when those specific options change.
were previously carried by the unconditional
SetOptions). Ifcache-side logging on reload matters, happy to adjust.
certcache_test.gocovers reuse-on-identical-options andreplace-on-changed-options, restoring the package globals it touches.
Refs #7897