fix(resilience): circuit breaker correctness, stale provider inventory, fast health rechecks#480
Conversation
- Release the half-open probe slot when a request ends without a breaker verdict (local build error or caller cancellation); previously the breaker stayed half-open with the slot consumed and rejected every request until process restart. - Stop counting caller-side context cancellations as provider failures: client aborts during stream establishment could open the circuit for a healthy provider. Client deadlines still count. - Export per-provider breaker state as the gomodel_circuit_breaker_state gauge (0=closed, 1=half-open, 2=open) via ResponseInfo.CircuitState. - Simplifications: remove dead Allow() and duplicate getBaseURL(), delegate NewWithHTTPClient to New, use errors.As in extractStatusCode. Co-Authored-By: Claude Fable 5 <[email protected]>
…echeck - A provider whose model refresh fails keeps its previous inventory, marked stale, instead of losing it: direct requests reach the provider and fail with an honest 502/503 (failover rules can fire) instead of "model not found". New Catalog.ModelAvailable() lets virtual-model load balancing keep skipping stale targets, and bare model IDs served by several providers prefer a healthy one - the same routing the old inventory wipe produced, without the data loss. - Fetch the refresh sweep concurrently: one slow upstream no longer starves later providers out of the shared context budget. - Re-probe only the providers whose latest refresh failed on PROVIDER_RECHECK_INTERVAL (default 60s, 0 disables), so outage recovery and the dashboard "Last checked" no longer wait for the next full refresh (CACHE_REFRESH_INTERVAL, default hourly). Co-Authored-By: Claude Fable 5 <[email protected]>
The provider status card preferred last_model_fetch_at even when an availability probe ran more recently. Co-Authored-By: Claude Fable 5 <[email protected]>
…r gauge - What counts as a retryable failure vs a breaker failure (429/4xx/ cancellation exemptions), per-path retry behavior, and how to disable each layer. - New section separating the circuit breaker from dashboard provider health, including what happens while a provider is down and the CACHE_REFRESH_INTERVAL / PROVIDER_RECHECK_INTERVAL knobs (the former was previously undocumented outside .env.template). - Document the gomodel_circuit_breaker_state gauge. Co-Authored-By: Claude Fable 5 <[email protected]>
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughAdds a provider recheck interval, stale-inventory tracking with availability-aware routing, circuit-breaker state reporting and probe release handling, a dashboard timestamp fix, and matching documentation updates. ChangesRecheck interval, stale inventory, and observability
Estimated code review effort: 4 (Complex) | ~75 minutes Sequence Diagram(s)sequenceDiagram
participant Registry
participant Provider
participant VirtualModelService
participant Catalog
Registry->>Provider: RefreshProviderModels()
Provider-->>Registry: fetch error
Registry->>Registry: markProviderInventoryStale(provider)
Registry->>Registry: rebuild model map (fresh-first order)
VirtualModelService->>Catalog: ModelAvailable(qualifiedTarget)
Catalog-->>VirtualModelService: false (stale)
VirtualModelService->>Catalog: ModelAvailable(nextTarget)
Catalog-->>VirtualModelService: true
VirtualModelService-->>VirtualModelService: route to healthy target
sequenceDiagram
participant Client
participant CircuitBreaker
participant PrometheusHooks
Client->>Client: buildRequest fails locally
Client->>CircuitBreaker: finishRequestWithoutBreaker()
CircuitBreaker->>CircuitBreaker: releaseProbe() sets halfOpenAllowed=true
Client->>PrometheusHooks: OnRequestEnd(CircuitState)
PrometheusHooks->>PrometheusHooks: set CircuitBreakerState gauge
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
@coderabbitai continue |
|
✅ Action performedReview finished.
|
Review follow-up: a sweep in which every provider fails deliberately keeps the previous inventory routable (marking everything stale would only turn provider-level 502/503s into alias 404s, and would break aliased traffic when only the control plane is down) - that branch is now documented and pinned by a test. The real gap it hid: after a total outage, a provider recovering via the recheck loop left its still-down peers unmarked until the next full sweep, so load-balanced redirects kept routing to them for up to an hour. Failed per-provider probes (recheck loop, request-time refresh, availability checks) now mark the provider's carried inventory stale as soon as at least one healthy alternative exists, retiring it from load balancing within one recheck interval. Co-Authored-By: Claude Fable 5 <[email protected]>
|
Re: P1 "Carry-forward is skipped" (registry_init.go early return) — addressed in 9b663e7, though not by marking stale on the total-failure sweep, which would make things worse:
The real gap behind the finding was the recovery window: after a total outage, a provider recovering via the 60s recheck left its still-down peers unmarked (fresh) until the next full sweep — LB kept routing to them for up to an hour. Fixed: failed per-provider probes (recheck loop, request-time refresh, availability checks) now mark the provider's carried inventory stale as soon as at least one healthy alternative exists, retiring it from load balancing within ~one recheck interval. Covered by |
|
@coderabbitai continue |
|
Continuing the review now. ✏️ Learnings added
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/dev/prometheus-metrics.md (1)
44-67: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winHook API snippet is stale — missing the new
CircuitStatefield.This PR adds
CircuitState stringtoResponseInfo(used by the newgomodel_circuit_breaker_stategauge documented just below), but theResponseInfostruct shown here still omits it. Readers following this doc to understand the hook payload will miss the field that powers the new metric.📝 Proposed fix
type ResponseInfo struct { Provider string Model string Endpoint string StatusCode int // 0 if network error Duration time.Duration Stream bool Error error // nil on success + // CircuitState is "closed"/"half-open"/"open"; empty when the breaker + // is disabled. + CircuitState string }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/dev/prometheus-metrics.md` around lines 44 - 67, Update the hook API example so the ResponseInfo struct includes the new CircuitState string field alongside StatusCode, Duration, Stream, and Error. Use the ResponseInfo and Hooks symbols in the snippet to locate the stale doc block, and keep the example aligned with the new gomodel_circuit_breaker_state metric by showing the full current payload shape.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/providers/registry_init.go`:
- Around line 295-316: The empty-list refresh is being treated as authoritative
inventory because the presence check in the registry merge logic only tests for
a key, not whether the fetched provider model map is empty. Update this merge
path around fetched.modelsByProvider, r.modelsByProvider, and r.providerRuntime
so that providers with an empty fetched model list are carried forward from the
previous snapshot and marked inventoryStale instead of replacing the old models.
Keep rebuildGlobalModelMap using the carried-forward inventory so Supports() and
ModelAvailable() continue to reflect provider-level failure handling.
In `@internal/providers/registry_provider_refresh.go`:
- Around line 74-103: The refresh path in ModelRegistry is treating empty-list
results as successful inventory updates, which clears prior models and resets
staleness incorrectly. Update markFailedRefreshProvidersStale and
applyFetchedProviderInventory so providers with a non-empty lastModelFetchError
but no returned models are treated like failed refreshes: keep the existing
modelsByProvider entry, do not overwrite it with an empty map, and do not clear
inventoryStale for that provider. Use the existing symbols
fetched.modelsByProvider, fetched.runtimeUpdates, and
markProviderInventoryStale/applyFetchedProviderInventory to preserve stale
carry-forward behavior.
In `@internal/providers/registry.go`:
- Around line 936-1000: `markProviderInventoryStale` and `FailedProviderNames`
only look at `lastModelFetchError`, so providers with availability probe
failures can still be treated as healthy and skipped by the fast recheck loop.
Update the health checks in `ModelRegistry.markProviderInventoryStale` and
`ModelRegistry.FailedProviderNames` to also consider `lastAvailabilityError`
alongside `lastModelFetchError`, using the existing `providerRuntime` state for
each provider. Ensure the healthy-alternative gate and the returned
failed-provider list both include providers with either kind of failure.
---
Outside diff comments:
In `@docs/dev/prometheus-metrics.md`:
- Around line 44-67: Update the hook API example so the ResponseInfo struct
includes the new CircuitState string field alongside StatusCode, Duration,
Stream, and Error. Use the ResponseInfo and Hooks symbols in the snippet to
locate the stale doc block, and keep the example aligned with the new
gomodel_circuit_breaker_state metric by showing the full current payload shape.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: fec833b8-6ff2-44ce-a3ca-ff9d35a462bd
📒 Files selected for processing (27)
.env.templateCLAUDE.mdconfig/cache.goconfig/config.godocs/advanced/configuration.mdxdocs/advanced/resilience.mdxdocs/dev/prometheus-metrics.mdinternal/admin/dashboard/static/js/modules/providers.jsinternal/admin/dashboard/static/js/modules/providers.test.cjsinternal/admin/handler_virtualmodels_test.gointernal/llmclient/circuit_breaker.gointernal/llmclient/client.gointernal/llmclient/client_test.gointernal/observability/metrics.gointernal/observability/metrics_test.gointernal/providers/init.gointernal/providers/provider_status.gointernal/providers/registry.gointernal/providers/registry_init.gointernal/providers/registry_provider_refresh.gointernal/providers/registry_test.gointernal/server/handlers_test.gointernal/virtualmodels/balancer_test.gointernal/virtualmodels/helpers_test.gointernal/virtualmodels/service.gointernal/virtualmodels/snapshot.gointernal/virtualmodels/types.go
| stale := make(map[string]bool, len(fetched.runtimeUpdates)) | ||
| carriedForward := 0 | ||
| for name := range fetched.runtimeUpdates { | ||
| if _, ok := fetched.modelsByProvider[name]; ok { | ||
| continue // this sweep produced authoritative inventory | ||
| } | ||
| previous := r.modelsByProvider[name] | ||
| if len(previous) == 0 { | ||
| continue // nothing to carry forward | ||
| } | ||
| fetched.modelsByProvider[name] = previous | ||
| stale[name] = true | ||
| carriedForward++ | ||
| } | ||
| r.modelsByProvider = fetched.modelsByProvider | ||
| r.applyProviderRuntimeUpdatesLocked(fetched.runtimeUpdates) | ||
| for name := range fetched.runtimeUpdates { | ||
| state := r.providerRuntime[name] | ||
| state.inventoryStale = stale[name] | ||
| r.providerRuntime[name] = state | ||
| } | ||
| r.models = rebuildGlobalModelMap(r.modelsByProvider, r.freshFirstProviderOrderLocked()) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔴 Critical | 🏗️ Heavy lift
Empty-list refresh silently wipes inventory instead of carrying it forward stale.
fetchAllProviderModels's "provider returned empty model list" branch (Lines 206-220) still inserts an empty map into out.modelsByProvider[providerName], even though it records lastModelFetchError and is surfaced via FailedProviderNames(). The ok check here only tests presence, not content, so this empty-list "failure" is treated as authoritative fresh inventory: the provider's previous models are replaced by the empty map and inventoryStale is left false.
Net effect: the provider's models disappear from r.models/r.modelsByProvider entirely, so Supports()/ModelAvailable() both return false — direct requests now hit a "model not found" 404 instead of the provider-level 502/503 this PR set out to preserve.
The same bug also affects RefreshProviderModels's path in registry_provider_refresh.go (see comment there).
🐛 Proposed fix
stale := make(map[string]bool, len(fetched.runtimeUpdates))
carriedForward := 0
for name := range fetched.runtimeUpdates {
- if _, ok := fetched.modelsByProvider[name]; ok {
+ if providerModels, ok := fetched.modelsByProvider[name]; ok && len(providerModels) > 0 {
continue // this sweep produced authoritative inventory
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| stale := make(map[string]bool, len(fetched.runtimeUpdates)) | |
| carriedForward := 0 | |
| for name := range fetched.runtimeUpdates { | |
| if _, ok := fetched.modelsByProvider[name]; ok { | |
| continue // this sweep produced authoritative inventory | |
| } | |
| previous := r.modelsByProvider[name] | |
| if len(previous) == 0 { | |
| continue // nothing to carry forward | |
| } | |
| fetched.modelsByProvider[name] = previous | |
| stale[name] = true | |
| carriedForward++ | |
| } | |
| r.modelsByProvider = fetched.modelsByProvider | |
| r.applyProviderRuntimeUpdatesLocked(fetched.runtimeUpdates) | |
| for name := range fetched.runtimeUpdates { | |
| state := r.providerRuntime[name] | |
| state.inventoryStale = stale[name] | |
| r.providerRuntime[name] = state | |
| } | |
| r.models = rebuildGlobalModelMap(r.modelsByProvider, r.freshFirstProviderOrderLocked()) | |
| stale := make(map[string]bool, len(fetched.runtimeUpdates)) | |
| carriedForward := 0 | |
| for name := range fetched.runtimeUpdates { | |
| if providerModels, ok := fetched.modelsByProvider[name]; ok && len(providerModels) > 0 { | |
| continue // this sweep produced authoritative inventory | |
| } | |
| previous := r.modelsByProvider[name] | |
| if len(previous) == 0 { | |
| continue // nothing to carry forward | |
| } | |
| fetched.modelsByProvider[name] = previous | |
| stale[name] = true | |
| carriedForward++ | |
| } | |
| r.modelsByProvider = fetched.modelsByProvider | |
| r.applyProviderRuntimeUpdatesLocked(fetched.runtimeUpdates) | |
| for name := range fetched.runtimeUpdates { | |
| state := r.providerRuntime[name] | |
| state.inventoryStale = stale[name] | |
| r.providerRuntime[name] = state | |
| } | |
| r.models = rebuildGlobalModelMap(r.modelsByProvider, r.freshFirstProviderOrderLocked()) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@internal/providers/registry_init.go` around lines 295 - 316, The empty-list
refresh is being treated as authoritative inventory because the presence check
in the registry merge logic only tests for a key, not whether the fetched
provider model map is empty. Update this merge path around
fetched.modelsByProvider, r.modelsByProvider, and r.providerRuntime so that
providers with an empty fetched model list are carried forward from the previous
snapshot and marked inventoryStale instead of replacing the old models. Keep
rebuildGlobalModelMap using the carried-forward inventory so Supports() and
ModelAvailable() continue to reflect provider-level failure handling.
| r.markFailedRefreshProvidersStale(fetched) | ||
| return 0, core.NewProviderError(providerSelector, http.StatusServiceUnavailable, "failed to refresh provider models", fetchedProviderRefreshError(fetched)) | ||
| } | ||
| r.applyFetchedProviderInventory(providerTypes, fetched) | ||
| r.markFailedRefreshProvidersStale(fetched) | ||
| return 0, core.NewProviderError(providerSelector, http.StatusServiceUnavailable, "provider returned no models", nil) | ||
| } | ||
|
|
||
| r.applyFetchedProviderInventory(providerTypes, fetched) | ||
| r.markFailedRefreshProvidersStale(fetched) | ||
| return fetched.totalModels, nil | ||
| } | ||
|
|
||
| // markFailedRefreshProvidersStale marks the carried inventory of providers | ||
| // whose live probe failed this refresh, so the recheck loop retires a down | ||
| // provider from load balancing as soon as a healthy alternative exists — | ||
| // instead of waiting for the next full sweep. Providers that produced (or | ||
| // fell back to) an authoritative inventory this refresh are left alone. | ||
| func (r *ModelRegistry) markFailedRefreshProvidersStale(fetched fetchedInventory) { | ||
| for providerName, state := range fetched.runtimeUpdates { | ||
| if strings.TrimSpace(state.lastModelFetchError) == "" { | ||
| continue | ||
| } | ||
| if _, ok := fetched.modelsByProvider[providerName]; ok { | ||
| continue | ||
| } | ||
| r.markProviderInventoryStale(providerName) | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🔴 Critical | 🏗️ Heavy lift
Same empty-list carry-forward bug as registry_init.go's applyFetchedInventory, but more aggressive here.
markFailedRefreshProvidersStale (Line 97) uses the same presence-only ok check, so a provider whose refresh returned an empty list (a failure per lastModelFetchError) is skipped and never marked stale.
Worse, applyFetchedProviderInventory (Lines 196-202) unconditionally overwrites r.modelsByProvider[providerName] with the (possibly empty) providerModels map and resets inventoryStale = false for every provider present in fetched.modelsByProvider — unlike genuine error/nil-response failures, which never populate that map key and thus implicitly preserve the prior inventory. An empty-list "failure" therefore actively wipes previously known models and marks the provider fresh, defeating the "carry forward stale inventory" contract this PR establishes.
🐛 Proposed fix
func (r *ModelRegistry) applyFetchedProviderInventory(providerTypes map[core.Provider]string, fetched fetchedInventory) {
metadataStats := r.enrichFetchedProviderModelMaps(providerTypes, fetched.modelsByProvider)
r.mu.Lock()
for providerName, providerModels := range fetched.modelsByProvider {
+ if len(providerModels) == 0 {
+ continue // no authoritative inventory; leave prior data + let markFailedRefreshProvidersStale flag it
+ }
r.modelsByProvider[providerName] = providerModels
// A refresh that produced inventory is authoritative again.
state := r.providerRuntime[providerName]
state.inventoryStale = false
r.providerRuntime[providerName] = state
}
r.applyProviderRuntimeUpdatesLocked(fetched.runtimeUpdates)
r.models = rebuildGlobalModelMap(r.modelsByProvider, r.freshFirstProviderOrderLocked()) func (r *ModelRegistry) markFailedRefreshProvidersStale(fetched fetchedInventory) {
for providerName, state := range fetched.runtimeUpdates {
if strings.TrimSpace(state.lastModelFetchError) == "" {
continue
}
- if _, ok := fetched.modelsByProvider[providerName]; ok {
+ if providerModels, ok := fetched.modelsByProvider[providerName]; ok && len(providerModels) > 0 {
continue
}
r.markProviderInventoryStale(providerName)
}
}Also applies to: 192-206
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@internal/providers/registry_provider_refresh.go` around lines 74 - 103, The
refresh path in ModelRegistry is treating empty-list results as successful
inventory updates, which clears prior models and resets staleness incorrectly.
Update markFailedRefreshProvidersStale and applyFetchedProviderInventory so
providers with a non-empty lastModelFetchError but no returned models are
treated like failed refreshes: keep the existing modelsByProvider entry, do not
overwrite it with an empty map, and do not clear inventoryStale for that
provider. Use the existing symbols fetched.modelsByProvider,
fetched.runtimeUpdates, and
markProviderInventoryStale/applyFetchedProviderInventory to preserve stale
carry-forward behavior.
…echecks Review follow-up. A request-time refresh can fail the availability gate (which marks the inventory stale) without ever attempting a model fetch, leaving lastModelFetchError empty. FailedProviderNames skipped such providers, so the recheck loop never re-probed them and they stayed retired from load balancing until the next full sweep - even after recovering. The healthy-alternative gate had the same blind spot and could count an availability-failing provider as the alternative that justifies retiring a peer. Both checks now consider lastAvailabilityError alongside lastModelFetchError, and the dashboard classifier reports Degraded (instead of Healthy) for a provider whose inventory is stale from an availability-only failure. Also documents the ResponseInfo.CircuitState field in the hook API snippet of the prometheus-metrics doc. Co-Authored-By: Claude Fable 5 <[email protected]>
….yaml PR #480 added the fast provider recheck loop and documented PROVIDER_RECHECK_INTERVAL in .env.template and the docs, but the YAML spelling was missing from config.example.yaml — the file users copy to configure via YAML. Add it next to refresh_interval, and mention the YAML key alongside the env var in the resilience docs. Refs #462 Co-Authored-By: Claude Fable 5 <[email protected]>
Summary
Review of the circuit breaker feature and the provider-health machinery around it, prompted by a user report of a down provider showing "Last checked 20m ago" with no way to control the cadence. Two confirmed breaker bugs fixed, the registry's failure handling reworked, and the previously invisible/undocumented pieces surfaced.
Circuit breaker fixes (
internal/llmclient)503 circuit breaker is openuntil process restart. The probe slot is now released when a request ends without a provider verdict.RecordFailure, so a handful of impatient clients could open the circuit for a healthy provider. Caller cancellations are now neutral; client deadlines still count.Allow(), duplicategetBaseURL()),NewWithHTTPClientdelegates toNew,extractStatusCodeuseserrors.As.Provider health rework (
internal/providers,internal/virtualmodels)Catalog.ModelAvailable()keeps virtual-model load balancing skipping stale targets, and bare model IDs served by several providers prefer a healthy one — the same routing the wipe produced, without the data loss. The dashboard's previously unreachable "Degraded — previous inventory is still available" state is now what a down provider shows.PROVIDER_RECHECK_INTERVAL(default 60s,0disables) re-probes only the providers whose last refresh failed, so outage recovery and the dashboard "Last checked" no longer wait for the next full refresh (CACHE_REFRESH_INTERVAL, default hourly).Observability & dashboard
gomodel_circuit_breaker_state{provider}gauge (0=closed, 1=half-open, 2=open) — previously breaker state was invisible everywhere.Docs
resilience.mdxnow documents what counts as a retryable vs breaker failure (429/4xx/cancellation exemptions), per-path retry behavior (streaming never retries; passthrough only when replay-safe), how to disable each layer, and a "circuit breaker vs dashboard provider health" section.CACHE_REFRESH_INTERVALenters the user-facing configuration reference (it was only an.env.templatecomment).User-visible impact
PROVIDER_RECHECK_INTERVAL(60); new metric:gomodel_circuit_breaker_state.Testing
go test ./...green,-raceclean on all touched packages, all 400 dashboard JS tests pass.🤖 Generated with Claude Code
Summary by CodeRabbit
PROVIDER_RECHECK_INTERVALcache setting (default 60s) to re-probe only failed providers.gomodel_circuit_breaker_stategauge for circuit-breaker visibility.