diff --git a/devlog/_plan/260914_cost_guard_stabilization/000_unit.md b/devlog/_plan/260914_cost_guard_stabilization/000_unit.md index 63d800c88b..87c6cffe36 100644 --- a/devlog/_plan/260914_cost_guard_stabilization/000_unit.md +++ b/devlog/_plan/260914_cost_guard_stabilization/000_unit.md @@ -83,8 +83,10 @@ what protects the operator who explicitly opts back out. ## Write scope Permitted: `src/codex/routing.ts`, `src/types/config.ts`, `src/config.ts`, the -account-pool and session-affinity code, their tests under -`tests/codex-integration/`, `docs-site/` configuration reference and its locales, +account-pool and session-affinity code, `src/routing/` for the identity, quota and +cache-domain layers `090_remaining_stack.md` plans (wpc's classifier, wpe's reservation +ledger, wpf's probe lease), their tests under `tests/codex-integration/` and +`tests/routing/`, `docs-site/` configuration reference and its locales, `structure/` docs that own the affected invariants, and this unit. Excluded, owned by concurrent lanes: `src/providers/devin*`, diff --git a/devlog/_plan/260914_cost_guard_stabilization/070_delivery.md b/devlog/_plan/260914_cost_guard_stabilization/070_delivery.md index 63617e8708..8fd02d5ca2 100644 --- a/devlog/_plan/260914_cost_guard_stabilization/070_delivery.md +++ b/devlog/_plan/260914_cost_guard_stabilization/070_delivery.md @@ -30,3 +30,36 @@ The configuration reference and every locale translation change in the same pull request as the behaviour, because a default documented in eight languages is wrong in eight languages the moment the code lands. `structure/` ownership docs for the affected invariants change with them. + +## 2.55.0 release record + +| field | value | +| --- | --- | +| product snapshot | `62f02223a0` on `dev` | +| preview SHA | `7bdd1b29b5` (`2.55.0-preview.20260914`) | +| stable SHA | `1cc89cf88c` (`2.55.0`) | +| dev next | `2.56.0` (#4618) | +| preview push CI | run 34833399886, success | +| preview service lifecycle | run 34833399853, success | +| preview dry-run / publish | 34834321705 / 34834502951, both success | +| main push CI | run 34835022788, success | +| main service lifecycle | run 34835022762, success | +| main dry-run / publish | 34836327017 / 34836498588, both success | +| registry: preview | verified, `registry.npmjs.org/@bitkyc08%2Fopencodex/2.55.0-preview.20260914` returns 200 | +| registry: stable | **pending** -- the version endpoint still returns 404 | + +The preview and stable trees are byte-identical apart from `package.json.version`; +`git diff origin/preview origin/main -- . ':!package.json'` is empty. + +**The stable registry line is the honest part.** The publish job reported success, its +post-publish registry smoke passed on the runner, and the `v2.55.0` tag and GitHub Release point +at `1cc89cf88c`. Fifteen minutes later the registry version endpoint still answers 404 and +`dist-tags.latest` still reads `2.54.0`, while the preview published minutes earlier answers 200. +So the receipt exists and availability is unconfirmed, which is `registryVerification: pending` -- +not a missing package. Do not re-run the publish: a second dispatch against the same version is +the failure mode the bounded-read path exists to prevent. Recover the observation, then announce. + +What this release does not claim: the PRD's RG2 set is not complete. The durable cross-restart +reservation ledger, V2 child first placement, the minimum quota/cache domain contract, the +transient half-open probe lease, combo hops on the shared budget, Cursor's inner retries and the +sends-per-logical-request surfacing all remain open, so #4546 stays open too. diff --git a/devlog/_plan/260914_cost_guard_stabilization/090_remaining_stack.md b/devlog/_plan/260914_cost_guard_stabilization/090_remaining_stack.md new file mode 100644 index 0000000000..74308842a0 --- /dev/null +++ b/devlog/_plan/260914_cost_guard_stabilization/090_remaining_stack.md @@ -0,0 +1,93 @@ +# 090 — what is left after 2.55.0, as a seven-layer stack + +## Why this doc exists + +`070_delivery.md` closed the 2.55.0 release record with a list of things the +release deliberately does not claim: the durable cross-restart reservation +ledger, V2 child first placement, the minimum quota/cache domain contract, the +transient half-open probe lease, combo hops on the shared budget, Cursor's inner +retries, and sends-per-logical-request surfacing. That list is accurate and it is +also unordered, which is the problem. Each item touches a different layer of the +same request path, and three of them change the same two files. + +This doc fixes the order and the write scopes so the remaining work can ship as a +stack of independently revertible pull requests rather than one unreviewable diff. + +## What is already true + +Stating this once, because repeating the original problem statement as if nothing +landed is the failure mode this unit keeps hitting. On `dev@4f788f91`: a request +carries a guarded four-send profile with a three-send base allowance and one shared +final-recovery reserve (#4609); a zero budget no longer floors to one (#4613); the +workflow guard caps physical sends, distinct children and concurrency and reserves +an interactive slot (#4614); a healthy detour is promoted rather than discarded when +a hold expires, and `Retry-After` is honoured on the transient path (#4616). + +So the remaining work is not "add a budget". It is: make the budget reach the +paths it still cannot see, make it correct under concurrency, and stop it from +being reset by a restart or side-stepped by a fresh identity. + +## The stack + +Listed in the order the branches are stacked, each one based on the branch above it. + +| # | Layer | Branch | What it closes | +| --- | --- | --- | --- | +| 1 | wpc | `codex/4546-wpc-quota-cache-domains` | Auth identity, quota domain and cache domain as three separate values, plus conversational-state portability as its own check | +| 2 | wpe | `codex/4546-wpe-durable-reservation` | Token-and-output reservation at three scopes, unresolved spend, and a ledger that survives restart | +| 3 | wpf | `codex/4546-wpf-probe-lease-backpressure` | The half-open probe lease, `Retry-After` preserved past the local maximum, and pool-wide retry backpressure | +| 4 | wpd | `codex/4546-wpd-v2-lineage-placement` | V2 root/parent/thread lineage and child first placement onto the parent's current serving account | +| 5 | wpa | `codex/4546-wpa-dispatch-coverage` | The reset-retry counting seam, compact's routed fallback, the generic OAuth and Anthropic hops, the gated-400 ladder's relation to the shared cap, and permit atomicity | +| 6 | wpb | `codex/4546-wpb-combo-adapter-retries` | Combo's real hop and target transition under a per-target policy, and Cursor's and Kiro's inner retries | +| 7 | wpg | `codex/4546-wpg-spend-instrumentation` | Sends per logical request, reserved/settled/unresolved spend, cache provenance, and a no-account failure that explains itself | + +Only two of those adjacencies are real dependencies. wpb needs wpa's permit +contract to be atomic before a second dispatcher may be wired to it, and wpg +reports what every earlier layer produces, so it is last by construction rather +than by importance. The rest are contract layers that introduce a module and its +tests without rewiring a call site, which is what makes them stackable in +readiness order and revertible one at a time. + +That independence is deliberate and it is also the honest limitation of the first +three layers: wpc's classifier, wpe's ledger and wpf's lease are each landed +tested and, for now, partly unreferenced. Each one names in its own pull request +which later layer is obliged to call it. A module that nobody calls does not +protect anything, so the stack is not finished until the wiring layers land on +top of it. + +## The three corrections this stack is built on + +**"Passes the holder" and "limits every send" are different completion +conditions.** #4608 gave a combo child the budget object; #4609 gave the request a +policy. Neither makes a second combo target draw the remainder, because the +adapter's initial send still reads its own policy allowance. A layer that receives +the counter and does not consult it as a limit reintroduces the multiplier +silently. + +**The permit is not yet atomic.** `reserveDispatch()` evaluates the remainder and +`permit.use()` charges it. Two legs that reserve concurrently against one +remaining send both receive a permit. The fix is to make the reservation the +charge and add an explicit release for an abandoned reservation, which is why wpa +has to land before anything else wires a new caller. + +**A memory Map is not a budget.** The ledger lives in process memory, and cleanup +only protects roots with active requests, so an exhausted-but-idle root can be +deleted and recreated fresh under the same id. Until reservations are durable and +cleanup is exhaustion-aware, "this root is out of budget" means "out of budget +until something restarts". + +## Verification posture + +Unchanged from `070_delivery.md` and restated because it governs every layer here: +the local suite, typecheck, install and build are **not run**, by explicit +instruction. Pushes use `--no-verify`. The only proof is hosted CI at the exact +final head SHA of each branch, and a green run against an earlier commit is not +evidence for the head that merges. Each pull request states that posture in its +Verification section rather than implying a local green. + +## What would make this fail + +Landing wpe's refusal path with a default limit low enough to refuse an +unconfigured install. The count caps from #4614 are already live and permissive; +token accounting must start observational and only enforce behind explicit +operator configuration, or the first upgrade turns a cost guard into an outage. diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index 7eb8a12524..9a0b822f52 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -54,6 +54,7 @@ separate. Full request URLs such as `/api/v1/responses` are not provider base UR | `autoSwitchThreshold?` | `number` | `80` | Usage threshold for placing new/unbound work. `quota` can re-evaluate unbound tasks on their next request once usage crosses this threshold. Bound tasks keep their account past the threshold by default (`pool.cacheAffinity`); they leave only when that account is exhausted or otherwise cannot serve, and then only for an account with genuine quota headroom and strictly lower usage. Set `pool.cacheAffinity: false` to re-evaluate bound tasks at this threshold, still only onto such a destination. `fill-first` uses it only as the drain point for unbound assignment; normal `round-robin` selection does not use it. The score uses the hottest known 5h, weekly, or 30d quota window. `0` disables usage-based proactive switching only, not unbound assignment or failure recovery. | | `accountPoolStrategy?` | `"quota" \| "round-robin" \| "fill-first" \| "reset-first"` | `"quota"` | Assignment strategy for new/unbound Codex requests. A request is unbound when it has no live (parent thread id, quota scope) affinity; a visible existing task can become unbound after proxy restart or affinity reset. `quota` picks the lowest-usage eligible account when no active account exists, keeps an eligible active account below `autoSwitchThreshold`, and after the threshold may move an unbound request. Bound tasks follow `pool.cacheAffinity` (on by default): they stay until the account is exhausted (known usage at 100%) or otherwise cannot serve, and then may rebind only to an account with genuine quota headroom and strictly lower usage. Set the flag `false` to proactively rebind a bound task at the threshold, still only onto such a destination. `round-robin` distributes unbound requests evenly; `fill-first` keeps assigning unbound requests to the active account until cooldown, unavailability, or the configured drain threshold. `reset-first`: Prefer the nearest future 5-hour or weekly reset among accounts below the usage threshold. Bound tasks follow the configured affinity policy. Independent model quotas use quota ordering. Monthly resets do not determine this ordering. | | `pool.cacheAffinity?` | `boolean` | `true` | Cache-affinity ordering for bound Codex threads, independent of `pool.kernel`. On by default; omitting the key or setting `true` keeps a bound task on its account until that account genuinely cannot serve. Only an explicit `false` restores threshold-based rebinding of bound tasks. A live binding outranks quota headroom: `quota` does not move the thread merely because usage crossed `autoSwitchThreshold`. The thread still leaves if that account cannot serve — paused, unusable, or genuinely exhausted (known usage at 100%) — and then only to an account with genuine quota headroom and strictly lower usage. Under either setting, an account with unknown usage is never chosen as a destination for a bound task, so when every account sits above the threshold the task stays put. Affinity is a reordering, not a pin. | +| `pool.credentialGroups?` | `Array<{ id: string; credentials: string[]; note?: string }>` | `[]` | Accepted and validated, but not yet consumed by routing: declaring a group changes no routing decision until a consuming layer lands. Operator-declared quota domains: groups of credentials that demonstrably share one upstream usage limit. Members of one group count once toward available capacity, and a quota refusal inside a group is not answered by rotating to another member — the limit is the same, so the move would pay a cold prefix for zero new capacity. Declared groups speak only to quota; sharing a limit says nothing about prompt-cache compatibility, which is classified separately. Each member is written provider-qualified as `":"`, because a credential id means something only inside its provider; the provider segment accepts the usual aliases (`chatgpt:` and `codex:` both mean OpenAI). Group ids must be unique, `credentials` must be non-empty, and a credential may appear in at most one group — an ambiguous declaration is rejected on write and dropped with a warning on load rather than resolved by whichever group is listed first, since that would merge two unrelated quota domains. A malformed list costs only the grouping: `pool.kernel` and `pool.cacheAffinity` are preserved. Absent or empty means no declared grouping, so an unconfigured install behaves exactly as before. | | `accountPoolStickyLimit?` | `number` | `1` | New/unbound task assignments retained on one round-robin selection before advancing; the counter advances when a task is bound, not after an upstream success. Range 1–100. | | `upstreamFailoverThreshold?` | `number` | `3` | Consecutive transient failures before future new sessions fail over. Set `0` to disable. For regular Responses and native compact sends, proven pre-connection DNS/TCP reachability failures are tracked at the provider-host level: they never affect account health, account cooldowns, thread/session affinity, active-account selection, or Pool routing, and never count toward this threshold. | | `upstreamHostCircuitThreshold?` | `number` | `0` | Opt-in circuit threshold for proven pre-connection DNS/TCP failures on native OpenAI forward Responses and compact sends. `0` disables it; `1`–`20` opens a 30-second provider-origin cooldown after that many terminal logical requests. While open, requests receive `503` with `Retry-After` before account selection or upstream send; after cooldown, one half-open request is admitted. Timeouts and HTTP responses never count, and any HTTP response closes the circuit. Applies only to Codex Pool routing with no pinned account; it is inert for `codexAccountMode: "direct"` and account-qualified selectors. | diff --git a/package.json b/package.json index 9e9f74ae75..cc0c690747 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@bitkyc08/opencodex", - "version": "2.55.0", + "version": "2.56.0", "description": "Universal provider proxy for OpenAI Codex & Claude Code — use any LLM with Codex CLI/App/SDK and Claude Code", "type": "module", "main": "./bin/package-main.mjs", diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 81a50f69a2..a731a90c41 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -1041,6 +1041,7 @@ "prime-client.test.ts": "clients", "privacy-mask-account.test.ts": "lib", "privacy-scan-meta-key.test.ts": "ci-workflows", + "probe-lease.test.ts": "routing", "process-control-graceful.test.ts": "lib", "process-control.test.ts": "lib", "process-state.test.ts": "service", @@ -1208,6 +1209,7 @@ "routing-compatibility-boundaries.test.ts": "routing", "routing-compatibility-model-matching.test.ts": "routing", "routing-compatibility.test.ts": "routing", + "routing-identity-domains.test.ts": "routing", "routing-intelligence-ui.test.ts": "gui", "routing-policy-fallback.test.ts": "routing", "routing-policy-pool-quota.test.ts": "routing", @@ -1270,6 +1272,8 @@ "skill-ocx.test.ts": "ci-workflows", "slug-codec.test.ts": "codex-integration", "sponsor-presets.test.ts": "providers", + "spend-ledger-file-journal.test.ts": "lib", + "spend-reservation-ledger.test.ts": "lib", "sse-client-frame-bounds.test.ts": "responses", "sse-decoder.test.ts": "responses", "sse-failed-tail.test.ts": "responses", @@ -1413,6 +1417,7 @@ "windows-user-principal.test.ts": "windows", "winsw-stop-hardening.test.ts": "windows", "winsw.test.ts": "service", + "workflow-budget.test.ts": "lib", "ws-endpoint.test.ts": "responses", "ws-failure-stage.test.ts": "responses", "ws-upstream-reuse.test.ts": "responses", diff --git a/src/config.ts b/src/config.ts index 4f851e4ffb..935fce734f 100644 --- a/src/config.ts +++ b/src/config.ts @@ -60,6 +60,7 @@ import { import { parseAccountPriority } from "./codex/pool-rotation"; import { COMBO_NAMESPACE, comboConfigIssues } from "./combos/types"; import { routingProfileIssues } from "./routing/profile"; +import { credentialGroupIssues } from "./routing/identity-domains"; import { POLICY_NAMESPACE } from "./routing/profile-namespace"; import { forgetEphemeralSecretPath, @@ -1201,6 +1202,43 @@ const codexPoolSchema = z.object({ excludedPlans: z.array(z.string().trim().min(1)).optional(), }).strict(); +/** + * Shape guard for the cross-element checks below. Zod runs an array-level check even + * when an element failed its own validation, and a failed element is not the shape the + * checker expects — reading `credentials.length` off it would throw out of `safeParse` + * and take the whole config load with it. Those elements already carry their own issues. + */ +function isCredentialGroupShape(value: unknown): value is { id: string; credentials: string[] } { + if (!value || typeof value !== "object" || Array.isArray(value)) return false; + const group = value as { id?: unknown; credentials?: unknown }; + return typeof group.id === "string" + && Array.isArray(group.credentials) + && group.credentials.every(member => typeof member === "string"); +} + +/** + * Operator-declared quota domains (`pool.credentialGroups`). + * + * Loose enough to hand-write, strict enough that it cannot mean two things: unique group + * ids, a non-empty member list, provider-qualified members, and each credential in at + * most one group. Those are not tidiness rules. `classifyCredential` keys a declared + * domain by group id, so a duplicate id or a credential listed twice merges two quota + * domains the operator never said were one -- after which the pool counts real capacity + * once and declines to rotate into it. A bare credential id is ambiguous for the same + * reason ids are provider-scoped in the auth store, so members carry their provider. + * {@link credentialGroupIssues} is the single definition, shared with the classifier. + */ +const credentialGroupsSchema = z.array(z.object({ + id: z.string().trim().min(1), + credentials: z.array(z.string().trim().min(1)).min(1), + note: z.string().optional(), +})).superRefine((groups, ctx) => { + if (!Array.isArray(groups) || !groups.every(isCredentialGroupShape)) return; + for (const message of credentialGroupIssues(groups)) { + ctx.addIssue({ code: "custom", message }); + } +}); + /** * Quota-reset notification section. * @@ -1392,6 +1430,12 @@ const configSchema = z.object({ pool: z.object({ kernel: z.boolean().optional(), cacheAffinity: z.boolean().optional(), + // The catch belongs on the list, not on `pool`. Left to the outer catch below, one + // malformed group failed this nested object and dropped the whole `pool` -- taking + // `kernel` and `cacheAffinity` with it, which is a live routing change the operator + // never made. Scoped here, a malformed or ambiguous group costs only the declared + // grouping: loadConfig warns, and the write path rejects it outright. + credentialGroups: credentialGroupsSchema.optional().catch(undefined), }).optional().catch(undefined), // Model ids excluded from the Grok Build managed block (dashboard switches). grokExcludedModels: z.array(z.string()).optional(), @@ -2147,6 +2191,30 @@ function warnDegradedCodexQuotaAutoRefresh(rawParsed: unknown, validated: OcxCon if (warning) console.warn(`⚠️ config.json ${warning}`); } +/** + * Companion to the degrade warnings above, for a malformed or ambiguous declared + * grouping. The list now degrades on its own so the rest of `pool` survives, which is + * also why it needs a voice: nothing else about the config looks different afterwards, + * and silently ungrouped credentials read as capacity the pool does not have. + */ +function degradedCredentialGroupsWarning(rawParsed: unknown): string | null { + const pool = rawConfigRecord(rawConfigRecord(rawParsed)?.pool); + if (!pool || pool.credentialGroups === undefined) return null; + const parsed = credentialGroupsSchema.safeParse(pool.credentialGroups); + if (parsed.success) return null; + // Every issue message is redacted before it is joined. The custom messages embed the + // offending member through `JSON.stringify`, so a malformed credential string that + // happens to carry secret material would otherwise be printed verbatim at config load + // — a config file is exactly where a pasted token ends up in the wrong field. + const details = parsed.error.issues.map(issue => redactSecretString(issue.message)).join("; "); + return `pool.credentialGroups is invalid (${details}) — declared quota grouping is disabled; other pool settings were preserved`; +} + +function warnDegradedCredentialGroups(rawParsed: unknown): void { + const warning = degradedCredentialGroupsWarning(rawParsed); + if (warning) console.warn(`⚠️ config.json ${warning}`); +} + /** * The apiKeys schema salvages entry by entry rather than failing the parse, so a * dropped key is otherwise invisible — and it will not be re-saved by the next @@ -2616,6 +2684,7 @@ export function loadConfig(): OcxConfig { warnDegradedQuotaResetNotify(parsed); warnDegradedCatalogAutoRefresh(parsed); warnDegradedCodexPool(parsed); + warnDegradedCredentialGroups(parsed); return withRefreshedCostOverlays(normalizeClaudeSubagentEffort(normalizeNativeSubagentSync(config, parsed), parsed)); } // Schema validation failed — merge defaults into the raw object instead of @@ -2659,6 +2728,7 @@ export function loadConfig(): OcxConfig { warnDegradedQuotaResetNotify(parsed); warnDegradedCatalogAutoRefresh(parsed); warnDegradedCodexPool(parsed); + warnDegradedCredentialGroups(parsed); return withRefreshedCostOverlays(normalizeClaudeSubagentEffort(normalizeNativeSubagentSync(config, parsed), parsed)); } // Still failing, but if every complaint is about one or more named entries @@ -2687,6 +2757,7 @@ export function loadConfig(): OcxConfig { warnDegradedQuotaResetNotify(parsed); warnDegradedCatalogAutoRefresh(parsed); warnDegradedCodexPool(parsed); + warnDegradedCredentialGroups(parsed); return withRefreshedCostOverlays(normalizeClaudeSubagentEffort(normalizeNativeSubagentSync(config, parsed), parsed)); } } @@ -3050,6 +3121,26 @@ function codexAccountPrioritiesError(value: unknown): string | null { return null; } +/** + * Same reasoning as {@link codexAccountPrioritiesError}, plus one of its own. The read + * path drops an invalid grouping, so a degraded write would erase a declaration the + * operator is still editing and still report success. And an ambiguous declaration -- + * one id used twice, one credential in two groups -- has no safe silent answer at all: + * resolving it by list order would quietly merge two quota domains. A live caller is + * told which group is the problem instead. + */ +function poolCredentialGroupsError(value: unknown): string | null { + const pool = rawConfigRecord(rawConfigRecord(value)?.pool); + if (!pool || pool.credentialGroups === undefined) return null; + const parsed = credentialGroupsSchema.safeParse(pool.credentialGroups); + if (parsed.success) return null; + const details = parsed.error.issues.map(issue => { + const path = issue.path.join("."); + return path ? `${path}: ${issue.message}` : issue.message; + }).join("; "); + return `schema_invalid: pool.credentialGroups: ${details}`; +} + function codexQuotaAutoRefreshError(value: unknown): string | null { const raw = rawConfigRecord(value); if (!raw || raw.codexQuotaAutoRefresh === undefined) return null; @@ -3248,6 +3339,7 @@ export function validateConfigCandidate(value: unknown): { ok: true; config: Ocx ?? codexPoolError(value) ?? googleAntigravityStaticCatalogVersionError(value) ?? codexAccountPrioritiesError(value) + ?? poolCredentialGroupsError(value) ?? codexQuotaAutoRefreshError(value) ?? codexAccountPickerEnabledError(value) ?? emptyCompletionRetryError(value) diff --git a/src/lib/spend-reservation-ledger.ts b/src/lib/spend-reservation-ledger.ts new file mode 100644 index 0000000000..21cdd78c72 --- /dev/null +++ b/src/lib/spend-reservation-ledger.ts @@ -0,0 +1,940 @@ +/** + * Durable token spend reservation, above the send-count workflow guard (#4546). + * + * The count cap treats a 1k-token send and a 150k-token send as the same unit, and the + * in-memory ledger forgets everything on restart: an exhausted root came back with a fresh + * allowance after every relaunch, and a second process never saw the first one's spend at + * all. This ledger reserves TOKENS before dispatch and rebuilds its state from a journal + * under the opencodex home directory, so an exhausted scope is still exhausted after a + * restart. + * + * A reservation is always the request's whole input plus its ENFORCEABLE output ceiling -- + * the caller's max_output_tokens, or the model's documented cap when the caller sent none. + * Never an optimistic estimate, and never shrunk by a cache-hit expectation: a prefix that + * misses is billed in full, so the safety figure reserves as if it misses. Cache + * expectations may inform efficiency reporting; they do not move this number. + * + * Admission requires, at every scope that applies at once -- root workflow, authenticated + * identity, and account pool: + * + * settled spend + in-flight reservations + unresolved spend + this reservation <= limit + * + * Unresolved spend is the conservative residue of a send whose usage frame was lost: the + * tokens may have been billed, so the reservation is moved to unresolved rather than + * released. Minting a new root id mints no new budget because the identity and pool scopes + * still hold the spend. + * + * SUPPORTED TOPOLOGY: this guarantees a single proxy process against its own journal. The + * file is append-friendly, but nothing here serializes two live processes writing it, so a + * second proxy sharing the same OPENCODEX_HOME is explicitly outside the guarantee -- that + * needs a shared store with cross-process atomicity and is declared out of scope rather + * than implied. + * + * Five properties this file owes its callers. Each one was absent in the first draft, and a + * budget that can be bypassed is worse than no budget because it looks like protection: + * + * 1. IDENTITY OF A SEND. A send id is either KNOWN -- and then reserving it again is refused + * rather than waved through booking nothing -- or FULLY forgotten, and then it books a + * fresh reservation. There is no third state where the ledger recognises an id and + * charges nothing for it, which is what let one id authorise unlimited physical sends. + * 2. DURABILITY BEFORE ADMISSION. Under a configured limit the reserve record must be on + * disk before the request is admitted. Failing open on a disk-full or permission error + * forgets the request across a restart, which is the exact case durability exists for. + * Observe-only mode still admits, and says so through `durable: false`. + * 3. REPLAY VALIDATES. Every journal record is checked field by field before it moves a + * counter. A corrupt record in the MIDDLE of the file would silently undercount, so it + * fails accounting closed instead; only an unparseable FINAL line -- a torn tail write -- + * is dropped quietly. + * 4. BOUNDED RETENTION. Cleanup runs automatically, writes durable tombstones so replay + * cannot resurrect what it removed, and compacts the journal to a checkpoint. When + * nothing can be evicted safely, admission is refused rather than made room for by + * forgetting an exhausted scope -- forgetting one is the laundering this layer prevents. + * 5. NOTHING IDENTIFYING ON DISK. Root ids come from a client header and identity ids are + * credential ids, so the journal stores salted aliases only, under owner-only permissions + * that are re-applied to an EXISTING file rather than trusted from its creation. + */ + +import { appendFileSync, chmodSync, existsSync, mkdirSync, readFileSync, renameSync, statSync, writeFileSync } from "node:fs"; +import { createHash, randomBytes } from "node:crypto"; +import { dirname, join } from "node:path"; +// Definition-site import, not the ../config barrel -- same reasoning as +// src/quota/reset-seen-store.ts: the barrel pulls ~154 modules into a hot path. +import { getConfigDir } from "../config/paths"; +import { assertNotRealHomeUnderTest } from "./test-home-guard"; +// Windows chmod does not remove inherited ACEs; this is the repository's icacls path. +import { hardenSecretPath } from "./windows-secret-acl"; + +export const SPEND_LEDGER_JOURNAL_FILENAME = "spend-ledger.jsonl"; +/** + * Per-install alias salt, beside the journal. Losing it is exactly as bad as losing the + * journal -- both reset accounting, both live in the same 0700 directory -- so it is not a + * new weakness, and keeping it out of the journal stops a copied or attached journal from + * being reversible by dictionary attack on guessable pool and identity ids. + */ +export const SPEND_LEDGER_SALT_FILENAME = "spend-ledger.salt"; + +export type SpendScope = "root" | "identity" | "pool"; + +export interface SpendScopeLimit { + /** + * Approved token ceiling for the scope. Undefined means OBSERVE ONLY: spend is still + * accounted and reported, but nothing is refused. That is the unconfigured default -- + * an install that never opted in keeps the count caps and is not newly refused. + */ + readonly maxTokens?: number; +} + +export interface SpendReservationPolicy { + readonly root: SpendScopeLimit; + readonly identity: SpendScopeLimit; + readonly pool: SpendScopeLimit; + /** + * How long a dormant scope's accounting is retained. A scope may be dropped only when it + * is BOTH inactive (no open reservation) AND not exhausted inside this window; dropping + * an exhausted scope would hand it a fresh allowance on next use. + */ + readonly retentionMs: number; + /** + * Hard ceiling on tracked scopes. Retention alone bounds nothing: a caller minting a fresh + * root id per request fills the map long before the window elapses. At the ceiling the + * ledger evicts the oldest scope that is safe to forget -- idle, under its limit, past + * retention -- and if there is none it REFUSES the new scope. Refusing is the only answer + * left: the alternative is evicting an exhausted scope, which hands it a fresh allowance. + */ + readonly maxTrackedScopes?: number; + /** Hard ceiling on remembered send ids, with the same evict-or-refuse rule. */ + readonly maxTrackedSends?: number; + /** + * Journal records after which the file is compacted into a single checkpoint. Without + * this the file grows forever even while the in-memory maps stay bounded, and replay + * resurrects every entry cleanup removed. + */ + readonly compactAfterRecords?: number; +} + +const DEFAULT_MAX_TRACKED_SCOPES = 4_096; +const DEFAULT_MAX_TRACKED_SENDS = 16_384; +const DEFAULT_COMPACT_AFTER_RECORDS = 8_192; + +/** + * Unconfigured default: every limit undefined, so token accounting runs in observe-only + * mode and the count caps remain the only enforcement. Real numbers belong behind + * explicit operator configuration. + */ +export const DEFAULT_SPEND_RESERVATION_POLICY: SpendReservationPolicy = { + root: {}, + identity: {}, + pool: {}, + retentionMs: 7 * 24 * 60 * 60_000, +}; + +export interface SpendScopes { + readonly rootId?: string; + readonly identityId?: string; + readonly poolId?: string; +} + +export interface SpendUsage { + readonly inputTokens: number; + readonly outputTokens: number; +} + +export interface SpendReservationRequest { + /** Stable id of the physical send. Settlement is idempotent on this key. */ + readonly sendId: string; + readonly scopes: SpendScopes; + readonly inputTokens: number; + /** Enforceable output ceiling -- max_output_tokens or the model's documented cap. */ + readonly outputCeilingTokens: number; + readonly at?: number; +} + +/** + * Why a reservation was refused. Every member refuses a DISPATCH: none of them is an + * "already fine, carry on" answer, because that is precisely how a duplicate send id used + * to buy an unlimited number of physical sends while the scope totals never moved. + */ +export type SpendDenial = + | { + readonly reason: "spend-limit-exceeded"; + readonly scope: SpendScope; + readonly scopeId: string; + readonly limit: number; + readonly projected: number; + } + /** This send id is already known -- open, settled, lost or abandoned. */ + | { readonly reason: "duplicate-send-id"; readonly sendId: string } + /** The reserve record could not be written, and a configured limit needs it to survive. */ + | { readonly reason: "reserve-not-durable"; readonly sendId: string } + /** Replay rejected records mid-file, so no scope total can be proven complete. */ + | { readonly reason: "journal-corrupt"; readonly corruptRecords: number } + /** Tracking is full and nothing may be forgotten safely. */ + | { readonly reason: "tracking-capacity-exhausted"; readonly scope?: SpendScope }; + +export type SpendReservationDecision = + | { + readonly reserved: true; + readonly sendId: string; + readonly tokens: number; + /** + * False only in observe-only mode, where the reservation was admitted although its + * journal record did not reach disk. A restart will not remember this spend; the flag + * is how a caller learns that instead of discovering it after the fact. + */ + readonly durable: boolean; + } + | { readonly reserved: false; readonly denial: SpendDenial }; + +interface ScopeState { + settled: number; + reserved: number; + unresolved: number; + lastSeenAt: number; +} + +/** + * `open` means admitted but not yet handed to a transport: it may still be abandoned for + * free. `dispatched` means bytes left for upstream, so from there a missing usage frame is + * unresolved SPEND rather than a release -- it may have been billed. Only a dispatched send + * can become `lost`; only an undispatched one can become `abandoned`. + */ +type ReservationStatus = "open" | "dispatched" | "settled" | "lost" | "abandoned"; + +interface ScopeRef { + readonly scope: SpendScope; + readonly alias: string; +} + +interface Reservation { + readonly targets: readonly ScopeRef[]; + readonly tokens: number; + status: ReservationStatus; + readonly at: number; + /** When the status last changed; drives eviction of resolved entries. */ + resolvedAt: number; +} + +/** + * Journal shape. Every id on disk is a salted alias, never a root header value, credential + * id or pool name. `forget` and `drop` are the tombstones that make bounded cleanup + * durable -- without them replay rebuilds exactly what cleanup removed -- and `checkpoint` + * is a whole-state snapshot that lets the file be compacted instead of growing forever. + */ +type JournalRecord = + | { v: 1; kind: "reserve"; send: string; targets: ScopeRef[]; tokens: number; at: number } + | { v: 1; kind: "dispatch"; send: string; at: number } + | { v: 1; kind: "settle"; send: string; tokens: number; at: number } + | { v: 1; kind: "lost"; send: string; at: number } + | { v: 1; kind: "abandon"; send: string; at: number } + | { v: 1; kind: "forget"; send: string; at: number } + | { v: 1; kind: "drop"; scope: SpendScope; alias: string; at: number } + | { + v: 1; + kind: "checkpoint"; + at: number; + scopes: { scope: SpendScope; alias: string; settled: number; unresolved: number; seenAt: number }[]; + sends: { send: string; status: ReservationStatus; targets: ScopeRef[]; tokens: number; at: number; resolvedAt: number }[]; + }; + +const isCountable = (value: unknown): value is number => + typeof value === "number" && Number.isFinite(value) && value >= 0; + +const isAlias = (value: unknown): value is string => + typeof value === "string" && value.length > 0 && value.length <= 256; + +const isScopeName = (value: unknown): value is SpendScope => + value === "root" || value === "identity" || value === "pool"; + +const isStatus = (value: unknown): value is ReservationStatus => + value === "open" || value === "dispatched" || value === "settled" + || value === "lost" || value === "abandoned"; + +const parseTargets = (value: unknown): ScopeRef[] | undefined => { + if (!Array.isArray(value) || value.length > 3) return undefined; + const targets: ScopeRef[] = []; + for (const entry of value) { + if (typeof entry !== "object" || entry === null) return undefined; + const { scope, alias } = entry as { scope?: unknown; alias?: unknown }; + if (!isScopeName(scope) || !isAlias(alias)) return undefined; + targets.push({ scope, alias }); + } + return targets; +}; + +/** + * Validate one journal line into a record, or reject it. + * + * Exported because this is the boundary where a hostile or damaged file meets the accounting: + * `JSON.parse(line) as JournalRecord` type-asserts a lie, and a bare `null` line or a + * `{"v":1,"kind":"reserve"}` with no fields crashed the rebuild rather than being rejected. + * Every field is checked, including that numbers are finite and non-negative. + */ +export function parseSpendJournalRecord(line: string): JournalRecord | undefined { + let raw: unknown; + try { + raw = JSON.parse(line); + } catch { + return undefined; + } + if (typeof raw !== "object" || raw === null || Array.isArray(raw)) return undefined; + const record = raw as Record; + if (record.v !== 1) return undefined; + if (!isCountable(record.at)) return undefined; + const at = record.at; + switch (record.kind) { + case "reserve": { + const targets = parseTargets(record.targets); + if (!isAlias(record.send) || targets === undefined || !isCountable(record.tokens)) return undefined; + return { v: 1, kind: "reserve", send: record.send, targets, tokens: record.tokens, at }; + } + case "settle": + if (!isAlias(record.send) || !isCountable(record.tokens)) return undefined; + return { v: 1, kind: "settle", send: record.send, tokens: record.tokens, at }; + case "dispatch": + if (!isAlias(record.send)) return undefined; + return { v: 1, kind: "dispatch", send: record.send, at }; + case "lost": + if (!isAlias(record.send)) return undefined; + return { v: 1, kind: "lost", send: record.send, at }; + case "abandon": + if (!isAlias(record.send)) return undefined; + return { v: 1, kind: "abandon", send: record.send, at }; + case "forget": + if (!isAlias(record.send)) return undefined; + return { v: 1, kind: "forget", send: record.send, at }; + case "drop": + if (!isScopeName(record.scope) || !isAlias(record.alias)) return undefined; + return { v: 1, kind: "drop", scope: record.scope, alias: record.alias, at }; + case "checkpoint": { + if (!Array.isArray(record.scopes) || !Array.isArray(record.sends)) return undefined; + const scopes: { scope: SpendScope; alias: string; settled: number; unresolved: number; seenAt: number }[] = []; + for (const entry of record.scopes) { + if (typeof entry !== "object" || entry === null) return undefined; + const e = entry as Record; + if (!isScopeName(e.scope) || !isAlias(e.alias)) return undefined; + if (!isCountable(e.settled) || !isCountable(e.unresolved) || !isCountable(e.seenAt)) return undefined; + scopes.push({ scope: e.scope, alias: e.alias, settled: e.settled, unresolved: e.unresolved, seenAt: e.seenAt }); + } + const sends: { send: string; status: ReservationStatus; targets: ScopeRef[]; tokens: number; at: number; resolvedAt: number }[] = []; + for (const entry of record.sends) { + if (typeof entry !== "object" || entry === null) return undefined; + const e = entry as Record; + const targets = parseTargets(e.targets); + if (!isAlias(e.send) || !isStatus(e.status) || targets === undefined) return undefined; + if (!isCountable(e.tokens) || !isCountable(e.at) || !isCountable(e.resolvedAt)) return undefined; + sends.push({ send: e.send, status: e.status, targets, tokens: e.tokens, at: e.at, resolvedAt: e.resolvedAt }); + } + return { v: 1, kind: "checkpoint", at, scopes, sends }; + } + default: + return undefined; + } +} + +/** + * Append-mostly persistence. `read` returns raw lines so replay can tell a torn TAIL write + * from corruption earlier in the file; only the former is safe to drop quietly. `append` + * THROWS when the record did not reach storage -- that signal is what lets admission refuse + * rather than admit a request a restart would forget. `rewrite` is optional: a store that + * cannot replace its contents atomically simply never compacts. + */ +export interface SpendJournal { + read(): string[]; + append(line: string): void; + rewrite?(lines: string[]): void; +} + +/** + * Re-apply owner-only permissions to a file that already exists. + * + * `mode` in a write option is honoured only when the file is CREATED, so a journal that was + * created loose -- by an older build, a restored backup, or a lax umask -- would keep its + * mode forever. Best-effort by design: a non-owner cannot chmod, and failing every append + * over it would be worse than the loose mode it is fixing. + * + * `force` marks the points where the WINDOWS ACL can actually be wrong: creation, compaction, + * and each process's replay. Windows chmod cannot drop inherited ACEs, so icacls is the real + * boundary there, and its memo keys on the file's ctime -- which every append changes. Running + * it per reservation would therefore spawn a process per send while protecting nothing an + * append can alter. On POSIX the mode is checked on every write and repaired the moment it + * drifts, which costs one stat. + */ +function hardenLedgerFile(path: string, options: { readonly force?: boolean } = {}): void { + if (process.platform === "win32") { + if (options.force) hardenSecretPath(path, { required: false }); + return; + } + try { + if ((statSync(path).mode & 0o777) === 0o600) return; + chmodSync(path, 0o600); + } catch { /* best-effort: a non-owner cannot chmod */ } +} + +export function createFileSpendJournal(path: string): SpendJournal { + const ensureDir = (): string => { + const dir = dirname(path); + // The guard runs before any mutation so a rejected write leaves nothing behind. + assertNotRealHomeUnderTest(dir); + mkdirSync(dir, { recursive: true, mode: 0o700 }); + return dir; + }; + return { + read(): string[] { + if (!existsSync(path)) return []; + // Replay is once per process and is the moment a journal inherited from an older build + // or a restored backup first passes through here. + hardenLedgerFile(path, { force: true }); + return readFileSync(path, "utf8").split("\n").filter((line) => line.length > 0); + }, + append(line: string): void { + ensureDir(); + const created = !existsSync(path); + appendFileSync(path, line + "\n", { encoding: "utf8", mode: 0o600 }); + hardenLedgerFile(path, { force: created }); + }, + rewrite(lines: string[]): void { + ensureDir(); + // Same directory, so the rename is atomic on the same filesystem: a crash mid-compaction + // leaves either the old journal or the new one, never a half-written ledger. + const temp = `${path}.compact-${process.pid}`; + writeFileSync(temp, lines.map((line) => line + "\n").join(""), { encoding: "utf8", mode: 0o600 }); + hardenLedgerFile(temp, { force: true }); + renameSync(temp, path); + hardenLedgerFile(path, { force: true }); + }, + }; +} + +/** + * Load the per-install alias salt, minting it on first use. + * + * The salt must be STABLE across restarts or replay cannot match a live request to its own + * recorded spend, which would hand every scope a fresh allowance -- so it is a file, not a + * per-process value. + */ +export function loadOrCreateSpendLedgerSalt(path: string): string { + if (existsSync(path)) { + hardenLedgerFile(path, { force: true }); + const existing = readFileSync(path, "utf8").trim(); + if (/^[0-9a-f]{32,}$/.test(existing)) return existing; + } + const dir = dirname(path); + assertNotRealHomeUnderTest(dir); + mkdirSync(dir, { recursive: true, mode: 0o700 }); + const salt = randomBytes(32).toString("hex"); + writeFileSync(path, salt + "\n", { encoding: "utf8", mode: 0o600 }); + hardenLedgerFile(path, { force: true }); + return salt; +} + +export interface ScopeSpendSnapshot { + readonly settled: number; + readonly reserved: number; + readonly unresolved: number; + readonly exhausted: boolean; +} + +export interface SpendReservationLedger { + reserve(request: SpendReservationRequest): SpendReservationDecision; + /** + * The send left for upstream. Until this is called the reservation may be abandoned for + * free; after it, a missing usage frame becomes unresolved spend. Returns false when the + * send is unknown or no longer open. + */ + markDispatched(sendId: string): boolean; + /** + * The send never happened -- local validation, routing, or a refusal before any byte left + * this process. The reservation is RELEASED and books nothing, because inventing debt the + * account never incurred is its own way of breaking the budget. Refused once the send is + * dispatched: from there only settle or markLost is honest. + */ + abandon(sendId: string): boolean; + /** + * Settle with real usage. Returns false when the send is unknown or already resolved -- + * double settlement is as wrong as none, so a repeat call changes nothing. + */ + settle(sendId: string, usage: SpendUsage): boolean; + /** + * Usage never arrived. The reservation moves to unresolved spend -- it may have been + * billed -- rather than being released. Idempotent on the same key as settle. + */ + markLost(sendId: string): boolean; + snapshot(scope: SpendScope, scopeId: string): ScopeSpendSnapshot | undefined; + exhausted(scope: SpendScope, scopeId: string): boolean; + /** + * Drop dormant scopes per the retention rule in SpendReservationPolicy. Cleanup also runs + * automatically on every reservation, so nothing depends on a caller remembering this. + */ + prune(now?: number): void; + /** Whether this send id is already known, and therefore refused. */ + knows(sendId: string): boolean; + /** Journal writes that failed; a nonzero count means durability is degraded. */ + readonly persistFailures: number; + /** + * Records replay rejected in the MIDDLE of the journal. Nonzero means no scope total can + * be proven complete, so configured limits refuse rather than undercount. + */ + readonly corruptRecords: number; + /** True when durability is degraded in either direction: failed writes or a corrupt file. */ + readonly degraded: boolean; +} + +const scopeKey = (scope: SpendScope, alias: string): string => scope + "\0" + alias; + +const sanitizeTokens = (value: number): number => + Number.isFinite(value) ? Math.max(0, Math.trunc(value)) : 0; + +export function createSpendReservationLedger(options: { + readonly journal?: SpendJournal; + readonly policy?: SpendReservationPolicy; + readonly now?: () => number; + /** + * Per-install alias salt. Production passes the file-backed value from + * `loadOrCreateSpendLedgerSalt`; an empty default is for in-memory journals, which have + * no file anyone could correlate. + */ + readonly salt?: string; +} = {}): SpendReservationLedger { + const policy = options.policy ?? DEFAULT_SPEND_RESERVATION_POLICY; + const journal = options.journal; + const now = options.now ?? (() => Date.now()); + const salt = options.salt ?? ""; + const maxTrackedScopes = policy.maxTrackedScopes ?? DEFAULT_MAX_TRACKED_SCOPES; + const maxTrackedSends = policy.maxTrackedSends ?? DEFAULT_MAX_TRACKED_SENDS; + const compactAfterRecords = policy.compactAfterRecords ?? DEFAULT_COMPACT_AFTER_RECORDS; + const scopes = new Map(); + const reservations = new Map(); + let persistFailures = 0; + let corruptRecords = 0; + let recordsOnDisk = 0; + + /** + * Salted alias for one identifier. The raw value -- a client-supplied root header, a + * credential id, a pool name -- never leaves this function, so nothing identifying is + * written to disk or held in a map key. + */ + const aliasFor = (kind: SpendScope | "send", id: string): string => + createHash("sha256").update(salt).update("\u0000").update(kind).update("\u0000").update(id) + .digest("hex").slice(0, 32); + + const scopeState = (scope: SpendScope, alias: string): ScopeState => { + const key = scopeKey(scope, alias); + let state = scopes.get(key); + if (!state) { + state = { settled: 0, reserved: 0, unresolved: 0, lastSeenAt: 0 }; + scopes.set(key, state); + } + return state; + }; + + const limitFor = (scope: SpendScope): number | undefined => policy[scope].maxTokens; + + const isExhausted = (scope: SpendScope, state: ScopeState): boolean => { + const limit = limitFor(scope); + return limit !== undefined && state.settled + state.reserved + state.unresolved >= limit; + }; + + /** The scopes a request touches, as aliases. Creates no state: a refusal must leave none. */ + const refsFor = (targets: SpendScopes): ScopeRef[] => { + const refs: ScopeRef[] = []; + if (targets.rootId !== undefined) refs.push({ scope: "root", alias: aliasFor("root", targets.rootId) }); + if (targets.identityId !== undefined) refs.push({ scope: "identity", alias: aliasFor("identity", targets.identityId) }); + if (targets.poolId !== undefined) refs.push({ scope: "pool", alias: aliasFor("pool", targets.poolId) }); + return refs; + }; + + /** + * Returns whether the record reached storage. With no journal there is nothing to fail, + * and the caller's durability question is vacuously satisfied. + */ + const append = (record: JournalRecord): boolean => { + if (!journal) return true; + try { + journal.append(JSON.stringify(record)); + recordsOnDisk += 1; + return true; + } catch { + // In-memory state still bounds this process; the counter is how a caller learns the + // restart guarantee degraded instead of discovering it after the fact. + persistFailures += 1; + return false; + } + }; + + const applyReserve = (send: string, targets: readonly ScopeRef[], tokens: number, at: number): void => { + if (reservations.has(send)) return; + reservations.set(send, { targets, tokens, status: "open", at, resolvedAt: at }); + for (const ref of targets) { + const state = scopeState(ref.scope, ref.alias); + state.reserved += tokens; + state.lastSeenAt = Math.max(state.lastSeenAt, at); + } + }; + + const isLive = (status: ReservationStatus): boolean => status === "open" || status === "dispatched"; + + /** + * Resolve a live reservation. `settled` books the real figure, `lost` keeps the whole + * reservation as unresolved spend because it may have been billed, and `abandoned` + * releases it because no byte ever left this process. + */ + const applyResolve = (send: string, outcome: "settled" | "lost" | "abandoned", tokens: number, at: number): void => { + const reservation = reservations.get(send); + if (!reservation || !isLive(reservation.status)) return; + reservation.status = outcome; + reservation.resolvedAt = at; + for (const ref of reservation.targets) { + const state = scopeState(ref.scope, ref.alias); + state.reserved = Math.max(0, state.reserved - reservation.tokens); + if (outcome === "lost") state.unresolved += reservation.tokens; + else if (outcome === "settled") state.settled += tokens; + state.lastSeenAt = Math.max(state.lastSeenAt, at); + } + }; + + const applyDispatch = (send: string, at: number): void => { + const reservation = reservations.get(send); + if (!reservation || reservation.status !== "open") return; + reservation.status = "dispatched"; + reservation.resolvedAt = at; + }; + + /** Tombstone replay: the entry is gone, so a later reuse of the id books a fresh charge. */ + const applyForget = (send: string): void => { + const reservation = reservations.get(send); + if (!reservation || isLive(reservation.status)) return; + reservations.delete(send); + }; + + const applyDrop = (scope: SpendScope, alias: string): void => { + const state = scopes.get(scopeKey(scope, alias)); + if (!state || state.reserved > 0) return; + scopes.delete(scopeKey(scope, alias)); + }; + + const applyCheckpoint = (record: Extract): void => { + scopes.clear(); + reservations.clear(); + for (const entry of record.scopes) { + scopes.set(scopeKey(entry.scope, entry.alias), { + settled: entry.settled, + reserved: 0, + unresolved: entry.unresolved, + lastSeenAt: entry.seenAt, + }); + } + for (const entry of record.sends) { + // `reserved` is rebuilt from the live entries rather than trusted from the snapshot, + // so the two can never disagree about the same tokens. + if (isLive(entry.status)) { + applyReserve(entry.send, entry.targets, entry.tokens, entry.at); + if (entry.status === "dispatched") applyDispatch(entry.send, entry.resolvedAt); + continue; + } + reservations.set(entry.send, { + targets: entry.targets, + tokens: entry.tokens, + status: entry.status, + at: entry.at, + resolvedAt: entry.resolvedAt, + }); + } + }; + + // Rebuild from the journal before serving: an exhausted scope must still be exhausted + // after a restart, which is the whole reason this store exists. + if (journal) { + const lines = journal.read(); + recordsOnDisk = lines.length; + for (let index = 0; index < lines.length; index += 1) { + const line = lines[index] as string; + const record = parseSpendJournalRecord(line); + if (!record) { + // A rejected FINAL line is a torn tail write -- the process died between the write + // and its newline -- and is dropped quietly, because that record never completed and + // therefore never authorised anything. A rejected line ANYWHERE ELSE is different: + // the records after it did complete, so skipping it silently undercounts a scope and + // hands back budget. It is counted, and a configured limit refuses on it below. + if (index < lines.length - 1) corruptRecords += 1; + continue; + } + switch (record.kind) { + case "reserve": applyReserve(record.send, record.targets, sanitizeTokens(record.tokens), record.at); break; + case "dispatch": applyDispatch(record.send, record.at); break; + case "settle": applyResolve(record.send, "settled", sanitizeTokens(record.tokens), record.at); break; + case "lost": applyResolve(record.send, "lost", 0, record.at); break; + case "abandon": applyResolve(record.send, "abandoned", 0, record.at); break; + case "forget": applyForget(record.send); break; + case "drop": applyDrop(record.scope, record.alias); break; + case "checkpoint": applyCheckpoint(record); break; + } + } + } + + /** + * Bounded cleanup. It runs before every admission, so nothing depends on a caller + * remembering `prune()` -- the first draft exported one and no production path called it. + * Every removal writes a tombstone: without one, replay rebuilds precisely what cleanup + * removed and the file keeps growing while the maps look bounded. + * + * `force` is the at-capacity pass. It ignores the retention window but never the safety + * rule: an ACTIVE or EXHAUSTED scope is not a candidate at any pressure, because dropping + * one hands it a fresh allowance under the same id. When that leaves nothing to remove, + * the caller refuses admission rather than making room by forgetting a spent scope. + */ + const evictScopes = (at: number, force: boolean): number => { + const cutoff = at - policy.retentionMs; + const candidates: { key: string; scope: SpendScope; alias: string; seenAt: number }[] = []; + for (const [key, state] of scopes) { + const separator = key.indexOf("\0"); + const scope = key.slice(0, separator) as SpendScope; + if (state.reserved > 0) continue; + if (isExhausted(scope, state)) continue; + if (!force && state.lastSeenAt >= cutoff) continue; + candidates.push({ key, scope, alias: key.slice(separator + 1), seenAt: state.lastSeenAt }); + } + if (force) { + candidates.sort((a, b) => a.seenAt - b.seenAt); + candidates.length = Math.min(candidates.length, 1); + } + for (const candidate of candidates) { + scopes.delete(candidate.key); + append({ v: 1, kind: "drop", scope: candidate.scope, alias: candidate.alias, at }); + } + return candidates.length; + }; + + /** + * Forget resolved send ids. A forgotten id is forgotten COMPLETELY: reusing it later books + * a fresh reservation against every scope, which is conservative. The state this must never + * produce is the middle one -- an id the ledger recognises but charges nothing for. + */ + const evictSends = (at: number, force: boolean): number => { + const cutoff = at - policy.retentionMs; + const candidates: { send: string; resolvedAt: number }[] = []; + for (const [send, reservation] of reservations) { + if (isLive(reservation.status)) continue; + if (!force && reservation.resolvedAt >= cutoff) continue; + candidates.push({ send, resolvedAt: reservation.resolvedAt }); + } + if (force) { + candidates.sort((a, b) => a.resolvedAt - b.resolvedAt); + candidates.length = Math.min(candidates.length, 1); + } + for (const candidate of candidates) { + reservations.delete(candidate.send); + append({ v: 1, kind: "forget", send: candidate.send, at }); + } + return candidates.length; + }; + + /** + * Replace the journal with a single checkpoint once it has grown past its record budget. + * Bounded maps are not enough on their own: the file behind them is what replay reads, and + * an uncompacted file grows forever on unique root and send ids. + */ + const compact = (at: number): void => { + const rewrite = journal?.rewrite; + if (!journal || !rewrite || recordsOnDisk < compactAfterRecords) return; + const checkpoint: JournalRecord = { + v: 1, + kind: "checkpoint", + at, + scopes: [...scopes].map(([key, state]) => { + const separator = key.indexOf("\0"); + return { + scope: key.slice(0, separator) as SpendScope, + alias: key.slice(separator + 1), + settled: state.settled, + unresolved: state.unresolved, + seenAt: state.lastSeenAt, + }; + }), + sends: [...reservations].map(([send, reservation]) => ({ + send, + status: reservation.status, + targets: [...reservation.targets], + tokens: reservation.tokens, + at: reservation.at, + resolvedAt: reservation.resolvedAt, + })), + }; + try { + rewrite.call(journal, [JSON.stringify(checkpoint)]); + recordsOnDisk = 1; + } catch { + // Compaction is maintenance, not accounting: a failed rewrite leaves the previous + // journal intact and every figure in it still replayable. + persistFailures += 1; + } + }; + + /** The denial when tracking cannot fit this request, or undefined when it can. */ + const makeRoom = (refs: readonly ScopeRef[], at: number): SpendDenial | undefined => { + evictSends(at, false); + evictScopes(at, false); + while (reservations.size >= maxTrackedSends) { + if (evictSends(at, true) === 0) return { reason: "tracking-capacity-exhausted" }; + } + let fresh = 0; + for (const ref of refs) if (!scopes.has(scopeKey(ref.scope, ref.alias))) fresh += 1; + while (scopes.size + fresh > maxTrackedScopes) { + if (evictScopes(at, true) === 0) { + return { reason: "tracking-capacity-exhausted", scope: refs[0]?.scope }; + } + } + return undefined; + }; + + return { + get persistFailures() { return persistFailures; }, + get corruptRecords() { return corruptRecords; }, + get degraded() { return persistFailures > 0 || corruptRecords > 0; }, + + reserve(request: SpendReservationRequest): SpendReservationDecision { + const tokens = sanitizeTokens(request.inputTokens) + sanitizeTokens(request.outputCeilingTokens); + const at = request.at ?? now(); + const send = aliasFor("send", request.sendId); + const refs = refsFor(request.scopes); + const enforced = refs.some((ref) => limitFor(ref.scope) !== undefined); + + // A send id this ledger already knows is REFUSED. Returning success while booking + // nothing -- the old behaviour -- let one id authorise an unlimited number of physical + // sends with the scope totals never moving. + if (reservations.has(send)) { + return { reserved: false, denial: { reason: "duplicate-send-id", sendId: request.sendId } }; + } + // Replay could not prove these totals are complete, so a configured ceiling cannot be + // enforced on them. Observe-only accounting continues and reports the degradation. + if (enforced && corruptRecords > 0) { + return { reserved: false, denial: { reason: "journal-corrupt", corruptRecords } }; + } + const capacity = makeRoom(refs, at); + if (capacity) return { reserved: false, denial: capacity }; + + // Check every scope before mutating any: a refusal must not leave a partial + // reservation booked on the scopes that would have passed. Reading state without + // creating it matters here -- a denied request must not leave a tracked scope behind. + for (const ref of refs) { + const limit = limitFor(ref.scope); + if (limit === undefined) continue; + const state = scopes.get(scopeKey(ref.scope, ref.alias)); + const projected = (state ? state.settled + state.reserved + state.unresolved : 0) + tokens; + if (projected > limit) { + const scopeId = ref.scope === "root" + ? request.scopes.rootId + : ref.scope === "identity" ? request.scopes.identityId : request.scopes.poolId; + return { + reserved: false, + denial: { reason: "spend-limit-exceeded", scope: ref.scope, scopeId: scopeId ?? "", limit, projected }, + }; + } + } + + // Durability BEFORE admission. The record goes to disk first, and under a configured + // limit a failed write refuses the request rather than admitting one that a restart + // would forget -- which is exactly the disk-full and permission case durability is for. + const durable = append({ v: 1, kind: "reserve", send, targets: refs, tokens, at }); + if (!durable && enforced) { + return { reserved: false, denial: { reason: "reserve-not-durable", sendId: request.sendId } }; + } + applyReserve(send, refs, tokens, at); + compact(at); + return { reserved: true, sendId: request.sendId, tokens, durable }; + }, + + markDispatched(sendId: string): boolean { + const send = aliasFor("send", sendId); + const reservation = reservations.get(send); + if (!reservation || reservation.status !== "open") return false; + const at = now(); + applyDispatch(send, at); + append({ v: 1, kind: "dispatch", send, at }); + return true; + }, + + abandon(sendId: string): boolean { + const send = aliasFor("send", sendId); + const reservation = reservations.get(send); + // Only an UNDISPATCHED reservation may be released for free. Once bytes have left for + // upstream the tokens may already be billed, so the caller owes settle or markLost. + if (!reservation || reservation.status !== "open") return false; + const at = now(); + applyResolve(send, "abandoned", 0, at); + append({ v: 1, kind: "abandon", send, at }); + return true; + }, + + settle(sendId: string, usage: SpendUsage): boolean { + const send = aliasFor("send", sendId); + const reservation = reservations.get(send); + if (!reservation || !isLive(reservation.status)) return false; + const tokens = sanitizeTokens(usage.inputTokens) + sanitizeTokens(usage.outputTokens); + const at = now(); + applyResolve(send, "settled", tokens, at); + append({ v: 1, kind: "settle", send, tokens, at }); + return true; + }, + + markLost(sendId: string): boolean { + const send = aliasFor("send", sendId); + const reservation = reservations.get(send); + if (!reservation || !isLive(reservation.status)) return false; + const at = now(); + applyResolve(send, "lost", 0, at); + append({ v: 1, kind: "lost", send, at }); + return true; + }, + + knows(sendId: string): boolean { + return reservations.has(aliasFor("send", sendId)); + }, + + snapshot(scope: SpendScope, scopeId: string): ScopeSpendSnapshot | undefined { + const state = scopes.get(scopeKey(scope, aliasFor(scope, scopeId))); + if (!state) return undefined; + return { + settled: state.settled, + reserved: state.reserved, + unresolved: state.unresolved, + exhausted: isExhausted(scope, state), + }; + }, + + exhausted(scope: SpendScope, scopeId: string): boolean { + const state = scopes.get(scopeKey(scope, aliasFor(scope, scopeId))); + return state !== undefined && isExhausted(scope, state); + }, + + prune(at: number = now()): void { + // Removal requires BOTH inactive and not exhausted inside the window. An + // exhausted-but-idle scope that was dropped would be recreated fresh under the + // same id -- the exact laundering the ceiling exists to stop. + evictSends(at, false); + evictScopes(at, false); + }, + }; +} + +let sharedLedger: SpendReservationLedger | undefined; + +/** + * Process-wide ledger backed by the journal under OPENCODEX_HOME. Created lazily so + * importing the module -- or running a request path that never reserves -- touches no + * disk. + */ +export function sharedSpendLedger(): SpendReservationLedger { + if (!sharedLedger) { + const home = getConfigDir(); + sharedLedger = createSpendReservationLedger({ + journal: createFileSpendJournal(join(home, SPEND_LEDGER_JOURNAL_FILENAME)), + salt: loadOrCreateSpendLedgerSalt(join(home, SPEND_LEDGER_SALT_FILENAME)), + }); + } + return sharedLedger; +} + +/** Test seam. Production never discards the ledger: that would reset a spent budget. */ +export function resetSharedSpendLedgerForTest(): void { + sharedLedger = undefined; +} diff --git a/src/lib/upstream-retry.ts b/src/lib/upstream-retry.ts index 9cd26731cb..aba90a96b2 100644 --- a/src/lib/upstream-retry.ts +++ b/src/lib/upstream-retry.ts @@ -139,7 +139,11 @@ export interface RetryBackoffOptions { * first instead of silently lengthening every adapter's backoff. */ retryAfterIsLowerBound?: boolean; - /** Hard ceiling for an honoured `Retry-After`, so an hour-long wait cannot park a request. */ + /** + * The wait deadline a caller applies to an honoured `Retry-After`. The delay itself is + * never shortened: an instruction longer than the deadline is a reason to END with the + * upstream answer, not to send early. Kept for callers that still pass it. + */ retryAfterCeilingMs?: number; } @@ -305,10 +309,10 @@ export function retryBackoffDelayMs(attempt: number, opts: RetryBackoffOptions): // A provider that names a wait is stating when it will serve again; sending earlier is a // request we already know will be refused, and refusing it twice is the retry storm the // header exists to prevent. The local maximum bounds our OWN exponential backoff and has no - // business shortening someone else's instruction. The ceiling is separate: it stops an - // hour-long Retry-After from parking a request forever. - const ceiling = opts.retryAfterCeilingMs ?? RETRY_AFTER_CEILING_MS; - return Math.min(Math.max(retryAfter, jittered), ceiling); + // business shortening someone else's instruction, so the instruction is returned in full. + // Whether the request can afford to wait that long is the caller's deadline decision -- + // fetchWithTransientRetry ends with the upstream answer rather than retrying early. + return Math.max(retryAfter, jittered); } export function cancelResponseBodyBestEffort(res: Response): void { @@ -364,6 +368,15 @@ export interface TransientRetryOptions extends ResetRetryOptions { * keep them on ONE budget instead of handing each leg a fresh one. */ onSendsConsumed?: (sends: number) => void; + /** + * How long this caller can wait on an honoured `Retry-After`, defaulting to + * {@link RETRY_AFTER_CEILING_MS}. It is a deadline, never a clamp: an instruction inside it + * is slept in full, and an instruction past it ends the call with the upstream answer and + * its `Retry-After` intact rather than sending early at a provider that already said it + * would refuse. A caller with a shorter budget than a minute says so and is not parked past + * it; a caller that can genuinely wait longer says so and is not cut short. + */ + retryAfterCeilingMs?: number; } export type UpstreamSendRecovery = "connection-reset" | "transient-5xx"; @@ -519,6 +532,19 @@ export async function fetchWithTransientRetry( // a response whose body we just cancelled. if (opts.abortSignal?.aborted) return res; if (Date.now() - attemptStart > slowAttemptMs) return res; + const instructedDelay = retryAfterDelayMs(res.headers); + // The deadline is the CALLER'S, not this module's default. Reading the constant directly + // broke it in both directions: a caller with a 30s budget slept the full 45s an upstream + // asked for, and a caller that could genuinely wait 120s was handed the error back for a + // 90s instruction it was willing to honour. + const waitDeadlineMs = opts.retryAfterCeilingMs ?? RETRY_AFTER_CEILING_MS; + if (instructedDelay !== undefined && instructedDelay > waitDeadlineMs) { + // Honouring the stated wait would park this request past the deadline it can commit + // to, and sleeping only up to the deadline is a send the provider already said it will + // refuse. End here instead: the caller receives the upstream answer with its + // Retry-After intact and applies its own policy, exactly as on the direct path. + return res; + } console.warn( `[upstream-retry] transient ${res.status}${opts.label ? ` (${opts.label})` : ""} — retrying (${sent + 1}/${budget})`, ); diff --git a/src/lib/workflow-budget.ts b/src/lib/workflow-budget.ts index 5cbe8e665f..8ca435352c 100644 --- a/src/lib/workflow-budget.ts +++ b/src/lib/workflow-budget.ts @@ -10,11 +10,22 @@ * header when the client supplies one. A retry is not a new user task and gets no new * allowance; a genuinely new top-level request does. * - * This ledger is process-local and in-memory. It bounds a single proxy process honestly and - * says nothing about a second process sharing the same account pool; that needs a shared - * durable store and is declared out of scope rather than implied. + * Two caps intersect here. The COUNT caps (concurrency, distinct children, physical sends) + * are process-local and in-memory. The TOKEN cap is the durable spend-reservation ledger in + * spend-reservation-ledger.ts: when the caller supplies a spend request, admission also + * reserves input + enforceable output ceiling against the root, identity and pool scopes, + * and that accounting survives a restart. The count caps alone remain the guarantee for a + * second process sharing the pool; the durable ledger's single-process topology is stated + * in that module's header and applies here unchanged. */ +import { + sharedSpendLedger, + type SpendReservationLedger, + type SpendScope, + type SpendUsage, +} from "./spend-reservation-ledger"; + export interface WorkflowBudgetPolicy { /** Children admitted concurrently under one root. */ readonly maxConcurrentChildren: number; @@ -27,7 +38,12 @@ export interface WorkflowBudgetPolicy { * root still gets admitted; without this a worker burst starves the conversation it serves. */ readonly interactiveReserve: number; - /** Roots tracked at once. Bounded so a caller minting new ids cannot grow this forever. */ + /** + * Roots tracked at once, as a hard bound rather than a hint. At the ceiling one idle, + * under-limit root is evicted to make room; when no root may be forgotten safely the new + * root is REFUSED with `workflow-tracking-exhausted`. Admitting it anyway is what made a + * caller minting new ids able to grow this map past the number written here. + */ readonly maxTrackedRoots: number; } @@ -42,18 +58,58 @@ export const DEFAULT_WORKFLOW_BUDGET_POLICY: WorkflowBudgetPolicy = { export type WorkflowDenial = | "workflow-concurrency-exhausted" | "workflow-sends-exhausted" - | "workflow-children-exhausted"; + | "workflow-children-exhausted" + | "workflow-spend-exhausted" + /** + * The root table is full and every entry is active or exhausted, so admitting this root + * would mean evicting one whose ceiling has already fired. Refusing is the honest answer: + * `maxTrackedRoots` is a bound, and inserting anyway made it a suggestion. + */ + | "workflow-tracking-exhausted" + /** This send id was already reserved once; a repeat buys no second dispatch. */ + | "workflow-send-replayed" + /** The reservation could not be made durable, and a configured ceiling requires it. */ + | "workflow-spend-undurable"; export type WorkflowLane = "interactive" | "worker"; export interface WorkflowAdmission { readonly rootId: string; + /** + * The request is about to leave for upstream. Call this at the dispatch boundary: until it + * runs, releasing the lease costs nothing, and after it a missing usage frame is booked as + * unresolved spend. + */ + markDispatched(): void; release(): void; } export type WorkflowDecision = | { admitted: true; lease: WorkflowAdmission } - | { admitted: false; reason: WorkflowDenial; rootId: string }; + | { + admitted: false; + reason: WorkflowDenial; + rootId: string; + /** Which spend scope refused, when the denial came from the token ledger. */ + spendScope?: SpendScope; + }; + +/** + * Token reservation attached to an admission. `outputCeilingTokens` is the ENFORCEABLE + * ceiling -- the caller's max_output_tokens or the model's documented cap, never an + * optimistic estimate and never shrunk by a cache-hit expectation. Omitting `spend` + * entirely keeps the historical count-only admission, which is also what an unconfigured + * install gets: token accounting is observed by default and refuses nothing until an + * operator sets real limits. + */ +export interface WorkflowSpendRequest { + /** Stable id of the physical send; settlement is idempotent on this key. */ + readonly sendId: string; + readonly identityId?: string; + readonly poolId?: string; + readonly inputTokens: number; + readonly outputCeilingTokens: number; +} interface WorkflowState { active: number; @@ -64,16 +120,29 @@ interface WorkflowState { const roots = new Map(); -function pruneOldestRoot(): void { +/** + * Evict the oldest root that is safe to forget, and report whether one was found. + * + * The return value is the point. An earlier version returned void and the caller inserted + * the new root regardless, so `maxTrackedRoots` bounded nothing whenever every candidate + * was active or exhausted -- which is precisely the fan-out this file exists to bound. + */ +function evictOneRoot(policy: WorkflowBudgetPolicy, spendLedger?: SpendReservationLedger): boolean { let oldestKey: string | undefined; let oldestAt = Number.POSITIVE_INFINITY; for (const [key, state] of roots) { // An active root is never evicted: dropping it would hand its fan-out a fresh allowance, - // which is the exact laundering this ledger exists to prevent. + // which is the exact laundering this ledger exists to prevent. The same holds for an + // EXHAUSTED-but-idle root -- count-exhausted or spend-exhausted -- because recreating it + // fresh under the same id resets the very ceiling that already fired. if (state.active > 0) continue; + if (state.sends >= policy.maxPhysicalSends) continue; + if (spendLedger?.exhausted("root", key) === true) continue; if (state.lastSeenMs < oldestAt) { oldestAt = state.lastSeenMs; oldestKey = key; } } - if (oldestKey !== undefined) roots.delete(oldestKey); + if (oldestKey === undefined) return false; + roots.delete(oldestKey); + return true; } /** @@ -81,6 +150,14 @@ function pruneOldestRoot(): void { * * `childId` distinguishes the members of a fan-out; omit it for the root's own turns. * An interactive lane may use the reserved slots a worker lane may not. + * + * When `spend` is given, admission also reserves its tokens on the spend ledger -- at the + * root, identity and pool scopes at once -- before a concurrency slot is taken. A turn + * released without settlement is resolved by whether it was ever DISPATCHED: an undispatched + * turn gives its tokens back, and a dispatched one keeps them as unresolved spend, because a + * send whose usage never arrived may still have been billed. Call `lease.markDispatched()` + * at the point the request leaves for upstream; without it, admission followed by a local + * validation or routing failure would book spend that never happened. */ export function admitWorkflowTurn( rootId: string | undefined, @@ -88,11 +165,21 @@ export function admitWorkflowTurn( policy: WorkflowBudgetPolicy = DEFAULT_WORKFLOW_BUDGET_POLICY, childId?: string, now: number = Date.now(), + spend?: WorkflowSpendRequest, + spendLedger?: SpendReservationLedger, ): WorkflowDecision | undefined { if (!rootId) return undefined; + // An explicit ledger is consulted even without a spend request, so root eviction can + // still see spend-exhausted entries. With neither, no token tracking is in play. + const ledger = spendLedger ?? (spend ? sharedSpendLedger() : undefined); let state = roots.get(rootId); if (!state) { - if (roots.size >= policy.maxTrackedRoots) pruneOldestRoot(); + if (roots.size >= policy.maxTrackedRoots && !evictOneRoot(policy, ledger)) { + // Nothing may be forgotten, so the new root is refused instead of admitted over the + // bound. The alternative -- evicting an exhausted root -- resets the ceiling that + // already fired, and a caller minting fresh ids would get unlimited budget from it. + return { admitted: false, reason: "workflow-tracking-exhausted", rootId }; + } state = { active: 0, sends: 0, children: new Set(), lastSeenMs: now }; roots.set(rootId, state); } @@ -112,6 +199,35 @@ export function admitWorkflowTurn( return { admitted: false, reason: "workflow-concurrency-exhausted", rootId }; } + if (spend && ledger) { + const decision = ledger.reserve({ + sendId: spend.sendId, + scopes: { rootId, identityId: spend.identityId, poolId: spend.poolId }, + inputTokens: spend.inputTokens, + outputCeilingTokens: spend.outputCeilingTokens, + at: now, + }); + if (!decision.reserved) { + const denial = decision.denial; + // Every ledger refusal denies a DISPATCH. A duplicate send id and an undurable + // reservation are reported as themselves rather than folded into "exhausted", because + // an operator reading a 429 needs to know which of the three happened. + const reason: WorkflowDenial = denial.reason === "duplicate-send-id" + ? "workflow-send-replayed" + : denial.reason === "reserve-not-durable" || denial.reason === "journal-corrupt" + ? "workflow-spend-undurable" + : denial.reason === "tracking-capacity-exhausted" + ? "workflow-tracking-exhausted" + : "workflow-spend-exhausted"; + return { + admitted: false, + reason, + rootId, + spendScope: denial.reason === "spend-limit-exceeded" ? denial.scope : undefined, + }; + } + } + state.active += 1; if (childId !== undefined) state.children.add(childId); let released = false; @@ -119,13 +235,25 @@ export function admitWorkflowTurn( admitted: true, lease: { rootId, + markDispatched(): void { + if (spend && ledger) ledger.markDispatched(spend.sendId); + }, release(): void { if (released) return; released = true; const current = roots.get(rootId); - if (!current) return; - current.active = Math.max(0, current.active - 1); - current.lastSeenMs = Date.now(); + if (current) { + current.active = Math.max(0, current.active - 1); + current.lastSeenMs = Date.now(); + } + // Which of the two applies depends on whether the send ever left this process. + // `abandon` succeeds only while the reservation is undispatched -- a turn refused by + // local validation or routing releases its tokens and books nothing, because + // inventing debt the account never incurred breaks the budget in the other + // direction. Once dispatched, abandon refuses and markLost keeps the cost as + // unresolved spend, since a send whose usage frame never arrived may still have been + // billed. Both are no-ops once settleWorkflowSpend already ran. + if (spend && ledger && !ledger.abandon(spend.sendId)) ledger.markLost(spend.sendId); }, }, }; @@ -143,6 +271,41 @@ export function chargeWorkflowSends(rootId: string | undefined, sends: number): state.lastSeenMs = Date.now(); } +/** + * Settle a send's reservation with the usage the response actually reported. Idempotent + * per send id -- a second call returns false and books nothing. When the usage frame was + * lost, call this never and let the lease's release move the reservation to unresolved + * spend, or call the ledger's markLost directly. + */ +export function settleWorkflowSpend( + sendId: string, + usage: SpendUsage, + spendLedger?: SpendReservationLedger, +): boolean { + return (spendLedger ?? sharedSpendLedger()).settle(sendId, usage); +} + +/** + * Record that the send left for upstream. + * + * This is the line between "may be released for free" and "may have been billed". Admission + * alone is not dispatch: a turn can be admitted and then fail request validation, provider + * routing, or a local guard without a single byte reaching a model. Booking those as spend + * invents debt the account never incurred, so the reservation only becomes unresolvable + * after this call. + */ +export function dispatchWorkflowSpend(sendId: string, spendLedger?: SpendReservationLedger): boolean { + return (spendLedger ?? sharedSpendLedger()).markDispatched(sendId); +} + +/** + * Give a reservation back because the send never happened. Refused once dispatched, where + * settle or markLost is the only honest outcome. + */ +export function abandonWorkflowSpend(sendId: string, spendLedger?: SpendReservationLedger): boolean { + return (spendLedger ?? sharedSpendLedger()).abandon(sendId); +} + /** * Whether this root has already spent its whole physical-send ceiling. * diff --git a/src/routing/identity-domains.ts b/src/routing/identity-domains.ts new file mode 100644 index 0000000000..8d7431b180 --- /dev/null +++ b/src/routing/identity-domains.ts @@ -0,0 +1,449 @@ +/** + * Authentication identity, quota domain, and cache domain are three different + * questions (#4546, wp6). + * + * A credential pool is stored as a flat list, which smuggles in two assumptions that are + * each wrong in the opposite direction: two API keys are treated as two independent pools + * of capacity, and two accounts on one provider are treated as not sharing a cache. The + * first overcounts available capacity -- OpenAI rate limits are per organization and + * project, so failing over from key A to key B inside the same limit buys nothing while + * still paying a cold prefix. The second discards warm prefixes the provider would have + * served, or worse, assumes a hit the provider never promised. + * + * This module is a conservative CLASSIFIER, not a claim about where a provider stores + * anything. Every answer carries provenance: "operator-declared" comes from configured + * credential groups, "provider-documented" comes from the small built-in table below for + * the cases the PRD names, and "unknown" is a first-class result. "unknown" is never + * silently read as "no sharing" and never as "shared" -- relations report it explicitly + * so the caller applies its own conservative rule. + * + * Provenance is only half of it. A documented rule can prove that two credentials are in + * DIFFERENT domains without proving that two others are in the SAME one, so every domain + * also carries which of those two facts its key supports ({@link DomainEvidence}). That + * is why two OpenAI keys in one organization and region relate "unknown" for cache: the + * documentation separates, then declines to promise the hit. + * + * Conversational-state portability is a separate question from cache compatibility and + * is deliberately not folded into the domain keys: a request carrying + * previous_response_id, a provider-side conversation id, uploaded file ids, or encrypted + * reasoning cannot be replayed onto another credential at all, no matter how the domains + * relate. `canPortConversationState` is that separate check. + */ + +/** Where a domain answer comes from. Order of trust: operator > provider docs > nothing. */ +export type IdentityDomainProvenance = "operator-declared" | "provider-documented" | "unknown"; + +/** + * What a domain key is evidence FOR, which is two facts rather than one. + * + * Proven SEPARATION and proven SHARING are different claims, and a provider routinely + * gives the first without the second. OpenAI documents that prompt caches are not shared + * across organizations or processing regions, and in the same breath documents that + * changing keys inside one organization does not guarantee a hit. So a different + * org-or-region key proves two domains, while an identical one proves nothing: a + * positive cache inference needs the provider to actually promise the hit, and here the + * provider declines to. Inferring "shared" from an equal key would be the same guess + * this module exists to refuse, only pointed the other way. + * + * "separates" therefore means two different keys are two different domains while two + * identical keys stay "unknown". "separates-and-shares" means the same source also + * promised that one key is one domain. + */ +export type DomainEvidence = "separates" | "separates-and-shares"; + +/** + * An opaque, comparable domain. `key` is only meaningful for equality when both sides + * are known; two "unknown" domains never compare shared because each carries a key + * derived from its own credential id. `evidence` decides whether an equal key is even + * allowed to mean "shared". + */ +export interface IdentityDomain { + readonly key: string; + readonly provenance: IdentityDomainProvenance; + readonly evidence: DomainEvidence; +} + +/** + * What the classifier knows about one credential. Every field beyond `credentialId` is + * optional evidence; a documented rule that needs a field this ref does not have yields + * "unknown", never a guess. + */ +export interface CredentialDomainRef { + readonly credentialId: string; + readonly provider?: string; + readonly organizationId?: string; + readonly projectId?: string; + readonly workspaceId?: string; + readonly deploymentId?: string; + readonly region?: string; +} + +export interface CredentialIdentity { + /** The credential the request is sent as. Never grouped, never shared. */ + readonly authIdentity: string; + /** The set of credentials that demonstrably share one usage limit. */ + readonly quotaDomain: IdentityDomain; + /** The conservative prompt-cache compatibility class. */ + readonly cacheDomain: IdentityDomain; + /** + * Group ids that claim this credential when the declaration is ambiguous: the same + * group id declared twice, or the credential listed in more than one group. An + * ambiguous declaration is never resolved by list order -- the quota domain falls back + * to the provider-documented or unknown answer and the conflict is reported here. + * `pool.credentialGroups` rejects such a declaration on write and drops it on load, so + * this covers a caller that assembled groups some other way. + */ + readonly declaredGroupConflict?: readonly string[]; +} + +/** + * How two domains relate. "unknown" is returned rather than collapsed into either + * answer, because treating it as "distinct" rotates within a shared limit (paying a + * cold prefix for zero capacity) and treating it as "shared" strands capacity that may + * be independent. An equal key whose evidence only proves separation also relates + * "unknown", which is how a documented non-sharing rule stays a non-sharing rule. + */ +export type DomainRelation = "shared" | "distinct" | "unknown"; + +/** + * Operator-declared grouping from `pool.credentialGroups`. + * + * `credentials` holds PROVIDER-QUALIFIED ids, `":"`. A bare id + * is ambiguous: credential ids are provider-scoped everywhere else -- `src/oauth/store.ts` + * keys an account by provider and id -- so `"acct-1"` names one credential per provider, + * and a bare declaration would silently merge unrelated quota domains. The provider + * segment normalizes through the same alias table as a classified ref, so + * `"chatgpt:acct-1"` and `"codex:acct-1"` name the same credential. + * + * Group ids must be unique, `credentials` must be non-empty, and a credential may appear + * in at most one group. {@link credentialGroupIssues} is the shared checker. + */ +export interface DeclaredCredentialGroup { + readonly id: string; + readonly credentials: readonly string[]; + readonly note?: string; +} + +/** + * The provider-documented cases the PRD names, and only those. A rule returns + * undefined when the ref lacks the evidence the documentation requires; the caller + * then classifies "unknown" rather than extrapolating. + * + * - OpenAI: rate limits are per organization and project, with model groups sharing a + * limit; prompt caches are not shared across organizations or processing regions. + * - Anthropic: prompt cache is isolated per workspace even inside one organization. + * (Cache-read tokens are also excluded from input TPM there, which is quota + * accounting, not domain shape, so it does not appear here.) + * - Azure: limits and cache breakpoints are per deployment. + * + * Each rule also carries what its documented sentence proves ({@link DomainEvidence}), + * because two of these are separation rules and the rest promise sharing as well. + */ +interface DocumentedDomainRule { + key(ref: CredentialDomainRef): string | undefined; + readonly evidence: DomainEvidence; +} + +const PROVIDER_DOCUMENTED_DOMAINS: Record = { + openai: { + quota: { + // Positive on both halves: the limit is defined per organization and project, and + // model groups share one limit, so two keys in one org and project are one limit. + key: (ref) => ref.organizationId !== undefined && ref.projectId !== undefined + ? `openai:org:${ref.organizationId}:project:${ref.projectId}` + : undefined, + evidence: "separates-and-shares", + }, + cache: { + // Separation only. The documentation says caches are not shared across + // organizations or processing regions, and says in the same place that changing + // keys inside one organization does not guarantee a hit. So a different org or + // region is proven distinct, while same org and region is "unknown" -- claiming + // "shared" there would assert a warm prefix the provider explicitly refuses to + // promise, and the caller would pay for it by replaying a long prompt that misses. + key: (ref) => ref.organizationId !== undefined && ref.region !== undefined + ? `openai:org:${ref.organizationId}:region:${ref.region}` + : undefined, + evidence: "separates", + }, + }, + anthropic: { + cache: { + // The cache is scoped to the workspace as a resource: isolated from other + // workspaces inside one organization, and reused within it. Both halves come from + // the same documented scoping, so an equal key may mean shared. + key: (ref) => ref.workspaceId !== undefined + ? `anthropic:workspace:${ref.workspaceId}` + : undefined, + evidence: "separates-and-shares", + }, + }, + azure: { + quota: { + // Quota and cache are both properties of the deployment resource itself. + key: (ref) => ref.deploymentId !== undefined + ? `azure:deployment:${ref.deploymentId}` + : undefined, + evidence: "separates-and-shares", + }, + cache: { + key: (ref) => ref.deploymentId !== undefined + ? `azure:deployment:${ref.deploymentId}` + : undefined, + evidence: "separates-and-shares", + }, + }, +}; + +const PROVIDER_ALIASES: Record = { + "azure-openai": "azure", + "chatgpt": "openai", + "codex": "openai", +}; + +function normalizedProvider(provider: string | undefined): string | undefined { + if (provider === undefined) return undefined; + const lowered = provider.trim().toLowerCase(); + return PROVIDER_ALIASES[lowered] ?? lowered; +} + +function unknownDomain(kind: "quota" | "cache", credentialId: string): IdentityDomain { + // The credential id in the key keeps two unknown domains from ever comparing equal: + // uniqueness is what makes "unknown" impossible to misread as "shared". + return { key: `unknown:${kind}:${credentialId}`, provenance: "unknown", evidence: "separates" }; +} + +function documentedDomain( + rule: DocumentedDomainRule | undefined, + ref: CredentialDomainRef, +): IdentityDomain | undefined { + const key = rule?.key(ref); + if (rule === undefined || key === undefined) return undefined; + return { key, provenance: "provider-documented", evidence: rule.evidence }; +} + +/** `":"`, the only accepted spelling of a declared member. */ +export const CREDENTIAL_GROUP_MEMBER_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._-]*:\S+$/; + +function splitMember(member: string): { provider: string; credentialId: string } | undefined { + if (!CREDENTIAL_GROUP_MEMBER_PATTERN.test(member)) return undefined; + const separator = member.indexOf(":"); + const provider = normalizedProvider(member.slice(0, separator)); + if (provider === undefined || provider === "") return undefined; + return { provider, credentialId: member.slice(separator + 1) }; +} + +function canonicalMember(member: string): string { + const parsed = splitMember(member); + return parsed === undefined ? `unqualified:${member}` : `${parsed.provider}:${parsed.credentialId}`; +} + +function memberMatches(member: string, ref: CredentialDomainRef): boolean { + const parsed = splitMember(member); + if (parsed === undefined) return false; + if (parsed.credentialId !== ref.credentialId) return false; + const refProvider = normalizedProvider(ref.provider); + // A ref without a provider cannot be matched to a provider-scoped declaration, so it + // keeps the documented or unknown answer instead of borrowing someone else's group. + return refProvider !== undefined && refProvider === parsed.provider; +} + +/** + * Every way a declared grouping can be ambiguous, as operator-readable messages. The + * config write path rejects on any of these and the load path drops the list, so an + * ambiguous declaration is reported rather than resolved by whichever group came first. + */ +export function credentialGroupIssues(groups: readonly DeclaredCredentialGroup[]): string[] { + const issues: string[] = []; + const seenIds = new Set(); + const owner = new Map(); + for (const group of groups) { + // A duplicate id is not cosmetic: both groups key to `declared:`, so the second + // group's members join the first group's quota domain without anyone saying so. + if (seenIds.has(group.id)) issues.push(`duplicate group id ${JSON.stringify(group.id)}`); + seenIds.add(group.id); + if (group.credentials.length === 0) { + issues.push(`group ${JSON.stringify(group.id)} lists no credentials`); + } + for (const member of group.credentials) { + if (splitMember(member) === undefined) { + issues.push( + `group ${JSON.stringify(group.id)} member ${JSON.stringify(member)} must be provider-qualified as ":"`, + ); + continue; + } + const existing = owner.get(canonicalMember(member)); + if (existing === group.id) { + issues.push(`credential ${JSON.stringify(member)} is listed twice in group ${JSON.stringify(group.id)}`); + } else if (existing !== undefined) { + issues.push( + `credential ${JSON.stringify(member)} is declared in more than one group (${existing}, ${group.id})`, + ); + } else { + owner.set(canonicalMember(member), group.id); + } + } + } + return issues; +} + +function resolveDeclaredGroup( + ref: CredentialDomainRef, + groups: readonly DeclaredCredentialGroup[], +): { group?: DeclaredCredentialGroup; conflict?: readonly string[] } { + const matches = groups.filter((group) => group.credentials.some((member) => memberMatches(member, ref))); + if (matches.length === 0) return {}; + const conflicting = new Set(); + for (const match of matches) { + if (matches.length > 1) conflicting.add(match.id); + if (groups.filter((group) => group.id === match.id).length > 1) conflicting.add(match.id); + } + if (conflicting.size > 0) return { conflict: [...conflicting] }; + return { group: matches[0] }; +} + +/** + * Classify one credential. `declaredGroups` is `pool.credentialGroups`; an operator + * declaration wins over the provider table because the operator can observe account + * topology the table cannot. Declared groups speak only to quota: sharing a usage + * limit says nothing about cache compatibility, so the cache domain never reads them. + * + * An ambiguous declaration -- a duplicated group id, or a credential claimed by two + * groups -- is not resolved by taking the first match. It is reported on + * `declaredGroupConflict` and the quota domain falls back to the documented or unknown + * answer, so a config that slipped past validation cannot silently merge two unrelated + * quota domains. + */ +export function classifyCredential( + ref: CredentialDomainRef, + declaredGroups: readonly DeclaredCredentialGroup[] = [], +): CredentialIdentity { + const { group: declared, conflict } = resolveDeclaredGroup(ref, declaredGroups); + const documented = PROVIDER_DOCUMENTED_DOMAINS[normalizedProvider(ref.provider) ?? ""] ?? {}; + + const quotaDomain: IdentityDomain = declared !== undefined + ? { key: `declared:${declared.id}`, provenance: "operator-declared", evidence: "separates-and-shares" } + : documentedDomain(documented.quota, ref) ?? unknownDomain("quota", ref.credentialId); + + const cacheDomain: IdentityDomain = documentedDomain(documented.cache, ref) + ?? unknownDomain("cache", ref.credentialId); + + return conflict === undefined + ? { authIdentity: ref.credentialId, quotaDomain, cacheDomain } + : { authIdentity: ref.credentialId, quotaDomain, cacheDomain, declaredGroupConflict: conflict }; +} + +function relateDomains(a: IdentityDomain, b: IdentityDomain): DomainRelation { + if (a.provenance === "unknown" || b.provenance === "unknown") return "unknown"; + if (a.key !== b.key) return "distinct"; + // Equal keys are proof of sharing only when both sides' evidence includes the sharing + // half. A separation-only rule (OpenAI's cache) stops here at "unknown". + return a.evidence === "separates-and-shares" && b.evidence === "separates-and-shares" + ? "shared" + : "unknown"; +} + +export function relateQuotaDomain(a: CredentialIdentity, b: CredentialIdentity): DomainRelation { + return relateDomains(a.quotaDomain, b.quotaDomain); +} + +export function relateCacheDomain(a: CredentialIdentity, b: CredentialIdentity): DomainRelation { + return relateDomains(a.cacheDomain, b.cacheDomain); +} + +/** + * What a quota refusal on `from` means for rotating to `to`. A refusal inside a known + * shared domain must not be answered by rotating within it -- the limit is the same, + * so the move pays a cold prefix for zero new capacity. "unknown" hands the decision + * back to the caller, which applies its own conservative rule. + */ +export type QuotaRotationVerdict = "same-domain" | "distinct-domain" | "unknown"; + +export function assessQuotaRotation( + from: CredentialIdentity, + to: CredentialIdentity, +): QuotaRotationVerdict { + const relation = relateQuotaDomain(from, to); + if (relation === "shared") return "same-domain"; + if (relation === "distinct") return "distinct-domain"; + return "unknown"; +} + +/** + * Available capacity across a credential set. Credentials in one known quota domain + * count ONCE. Unknown-domain credentials are reported separately rather than merged + * into either count, so the caller decides whether each is its own pool or not. + */ +export function countQuotaCapacity(identities: readonly CredentialIdentity[]): { + readonly known: number; + readonly unknown: number; +} { + const knownKeys = new Set(); + let unknown = 0; + for (const identity of identities) { + if (identity.quotaDomain.provenance === "unknown") { + unknown += 1; + } else { + knownKeys.add(identity.quotaDomain.key); + } + } + return { known: knownKeys.size, unknown }; +} + +/** Why a conversation cannot be replayed onto a different credential. */ +export type PortabilityDenial = + | "previous-response-id" + | "provider-conversation-id" + | "uploaded-file-ids" + | "encrypted-reasoning"; + +/** + * The parts of a request that bind it to the credential that produced them. Presence + * is what matters; the values stay opaque so nothing here logs or inspects ids. + */ +export interface ConversationStateCarriers { + readonly previousResponseId?: string | null; + readonly providerConversationId?: string | null; + readonly fileIds?: readonly string[]; + readonly encryptedReasoning?: unknown; +} + +export type PortabilityVerdict = + | { readonly portable: true } + | { readonly portable: false; readonly reason: PortabilityDenial }; + +function present(value: unknown): boolean { + if (value === undefined || value === null) return false; + if (typeof value === "string" || Array.isArray(value)) return value.length > 0; + return true; +} + +/** + * Whether a request's conversational state can move credentials at all. This is NOT + * cache compatibility: a shared cacheDomain means a replayed prefix might hit, while a + * refusal here means replaying is wrong regardless of warmth -- a previous_response_id + * or provider conversation id names server-side state another credential cannot see, + * and an uploaded file id or encrypted reasoning payload is bound to the account that + * issued it. A same-cacheDomain answer must never be read as portability, and a + * portable request gains no cache promise. + */ +export function canPortConversationState( + state: ConversationStateCarriers, +): PortabilityVerdict { + if (present(state.previousResponseId)) { + return { portable: false, reason: "previous-response-id" }; + } + if (present(state.providerConversationId)) { + return { portable: false, reason: "provider-conversation-id" }; + } + if (present(state.fileIds)) { + return { portable: false, reason: "uploaded-file-ids" }; + } + if (present(state.encryptedReasoning)) { + return { portable: false, reason: "encrypted-reasoning" }; + } + return { portable: true }; +} diff --git a/src/routing/probe-lease.ts b/src/routing/probe-lease.ts new file mode 100644 index 0000000000..fb63061e01 --- /dev/null +++ b/src/routing/probe-lease.ts @@ -0,0 +1,511 @@ +/** + * Half-open recovery for a held account, and the pool-wide retry/probe budget + * that sits above it (#4546, wp3 follow-up). + * + * The transient hold (#4616) keeps a thread's binding while its account serves a + * 5xx streak, and detours requests to a healthy sibling. What it cannot answer is + * whether the held account is actually back: a soft-avoided account receives no + * traffic, so the two-success clearing rule can only fire through the "held" + * fallback, which hands the failing account back to every pinned thread at once. + * This module is the bounded trial that closes that gap -- a single-holder probe + * lease keyed on the health domain (the quota-cooldown domain already has its own + * lease in src/codex/routing.ts and is a different thing). + * + * Three rules the lease enforces: + * + * - While an account is held, exactly one in-flight probe may test it. Every + * other request keeps the remembered detour, so a failed probe costs the + * caller nothing -- the detour's identity is never dropped to run the trial. + * - The lease has a deadline and is released on success, failure, or expiry. A + * response that arrives after its lease was lost is STALE: it must not + * overwrite a newer binding or a newer failure state, so every lease carries + * the generation it was issued under and a settle that fails the fence + * mutates nothing. + * - When every candidate is held the caller gets a typed "binding remembered, + * dispatch withheld" outcome -- not a send to an account already known to be + * failing. + * + * The pool-wide limiter exists because per-request send budgets do not prevent + * a retry storm: thousands of requests each staying inside their own allowance + * still compose into an unbounded rate against an already-failing upstream. + * Recovery dispatches (retries and probes, never the initial send of a new + * request) are admitted only while they stay under a ratio of observed initial + * sends in a sliding window -- the standard overload-guidance shape. + */ + +/** How long a granted probe may be in flight before its lease is forfeit. */ +export const TRANSIENT_PROBE_LEASE_MS = 30_000; +/** + * Minimum spacing between probes of the same held account. Without it every + * request that follows a settled probe becomes the next probe, which is the + * same storm the single-holder rule exists to bound -- just serialized. + */ +export const TRANSIENT_PROBE_INTERVAL_MS = 15_000; + +/** + * Grace kept on top of an entry's pacing and lease deadlines before it may be forgotten. + * Inside it a late settle can still answer "expired" rather than "stale", which is the + * distinction the settle contract exists to report. + */ +const PROBE_STATE_RETENTION_MS = 60_000; +/** + * Hard ceiling on remembered accounts. Pacing state is per account id, and account ids churn + * with configuration: without a ceiling a long-lived proxy accumulates one entry per id it + * ever probed. Above the ceiling the entries whose pacing lapses soonest are dropped, which + * at worst lets one dormant account be probed earlier than its interval; an entry holding a + * LIVE lease is never dropped, because that would hand out a second concurrent probe and + * break the single-holder rule the lease exists to enforce. + */ +export const MAX_TRANSIENT_PROBE_STATES = 1_024; +/** Below this the map is too small to be worth scanning on a grant. */ +const PROBE_STATE_SWEEP_THRESHOLD = 64; +/** + * Eviction target once the ceiling is reached. Clearing a block at a time keeps the ordering + * pass off the common grant path: it runs once per block of new accounts instead of once per + * grant forever after the first time the ceiling is touched. + */ +const PROBE_STATE_EVICTION_LOW_WATER = Math.floor(MAX_TRANSIENT_PROBE_STATES * 0.9); + +export interface TransientProbeLease { + readonly accountId: string; + readonly leaseId: string; + /** Epoch the lease was issued under; a settle must match the CURRENT epoch. */ + readonly generation: number; + readonly expiresAt: number; +} + +export type TransientProbeOutcome = "recovered" | "failed"; + +/** + * What a settle did to the lease. + * + * - `applied`: the probe still held the lease inside its deadline; the caller + * may act on the outcome (clear the hold, or record the fresh failure). + * - `stale`: the lease was already lost -- expired and re-issued, or invalidated + * by newer authoritative state. The result is dropped; nothing is overwritten. + * - `expired`: the probe finished after its own deadline. The lease is dead + * either way; this answer exists so the caller can tell "lost a race" from + * "ran long". + */ +export type TransientProbeSettle = "applied" | "stale" | "expired"; + +interface AccountProbeState { + /** Bumped on every lease grant and every external invalidation. */ + generation: number; + leaseId?: string; + leaseExpiresAt?: number; + lastProbeAt?: number; + /** + * Moment this account's pacing interval lapses, recorded at grant time from the interval + * that grant actually used. Kept alongside `lastProbeAt` so cleanup honours a caller's + * longer interval instead of assuming the default. + */ + pacedUntil?: number; + lastOutcome?: TransientProbeOutcome; +} + +const probeStates = new Map(); +let probeLeaseSeq = 0; + +function probeStateFor(accountId: string): AccountProbeState { + let state = probeStates.get(accountId); + if (!state) { + state = { generation: 0 }; + probeStates.set(accountId, state); + } + return state; +} + +function liveLease(state: AccountProbeState, now: number): boolean { + return state.leaseId !== undefined && state.leaseExpiresAt !== undefined && state.leaseExpiresAt > now; +} + +/** + * Moment an entry stops carrying anything a future decision can read: its pacing interval and + * any unsettled lease deadline, plus the grace above. + */ +function probeStateRetiresAt(state: AccountProbeState): number { + return Math.max(state.pacedUntil ?? 0, state.leaseExpiresAt ?? 0) + PROBE_STATE_RETENTION_MS; +} + +/** + * Bound the remembered accounts. Called on the one path that can grow the map -- a grant is + * the only insertion -- so the ceiling holds without a timer. + * + * The first pass drops only entries that can no longer change an answer: no live lease, the + * pacing interval lapsed, and the grace elapsed. Re-creating such an entry later yields the + * same decisions it would have produced, and a late settle against it still cannot be applied + * because lease ids are issued from a monotonic counter and never repeat. + */ +function sweepProbeStates(now: number): void { + if (probeStates.size <= PROBE_STATE_SWEEP_THRESHOLD) return; + for (const [accountId, state] of probeStates) { + if (liveLease(state, now)) continue; + if (now >= probeStateRetiresAt(state)) probeStates.delete(accountId); + } + if (probeStates.size <= MAX_TRANSIENT_PROBE_STATES) return; + // Still over the ceiling with nothing retired: churn is faster than the retention window. + // Evict in retirement order so the entries closest to meaningless go first, and never one + // holding a live lease. + const evictable = Array.from(probeStates) + .filter(([, state]) => !liveLease(state, now)) + .sort((a, b) => probeStateRetiresAt(a[1]) - probeStateRetiresAt(b[1])); + let excess = probeStates.size - PROBE_STATE_EVICTION_LOW_WATER; + for (const [accountId] of evictable) { + if (excess <= 0) break; + probeStates.delete(accountId); + excess -= 1; + } +} + +/** + * Grant the single in-flight probe for a held account, or null when another + * probe is already out or the pacing interval has not elapsed. The grant bumps + * the epoch, so a result from any earlier lease is stale the moment it lands. + */ +export function tryAcquireTransientProbe( + accountId: string, + now = Date.now(), + options?: { leaseMs?: number; minIntervalMs?: number }, +): TransientProbeLease | null { + const state = probeStateFor(accountId); + if (liveLease(state, now)) return null; + const interval = options?.minIntervalMs ?? TRANSIENT_PROBE_INTERVAL_MS; + if (state.lastProbeAt !== undefined && now - state.lastProbeAt < interval) return null; + const leaseMs = options?.leaseMs ?? TRANSIENT_PROBE_LEASE_MS; + const leaseId = `tprobe-${(probeLeaseSeq += 1).toString(36)}`; + const expiresAt = now + Math.max(1, leaseMs); + state.generation += 1; + state.leaseId = leaseId; + state.leaseExpiresAt = expiresAt; + state.lastProbeAt = now; + state.pacedUntil = now + Math.max(0, interval); + // After the grant, not before it: the entry this call just wrote holds a live lease and is + // therefore the one entry the sweep may never touch, so the ceiling is a real ceiling + // rather than "the ceiling plus whatever was inserted after the scan". + sweepProbeStates(now); + return { + accountId, + leaseId, + generation: state.generation, + expiresAt, + }; +} + +/** Side-effect-free mirror of {@link tryAcquireTransientProbe} eligibility. */ +export function canAcquireTransientProbe( + accountId: string, + now = Date.now(), + options?: { minIntervalMs?: number }, +): boolean { + const state = probeStates.get(accountId); + if (!state) return true; + if (liveLease(state, now)) return false; + const interval = options?.minIntervalMs ?? TRANSIENT_PROBE_INTERVAL_MS; + return state.lastProbeAt === undefined || now - state.lastProbeAt >= interval; +} + +/** + * Report a probe's outcome. Only the current lease holder inside its deadline + * applies: anything else is a late answer from a probe that already lost, and + * dropping it is what keeps it from overwriting a newer binding or a newer + * failure state. An applied settle clears the lease so the next probe is paced + * by the interval, not by the expiry. + */ +export function settleTransientProbe( + lease: TransientProbeLease, + outcome: TransientProbeOutcome, + now = Date.now(), +): TransientProbeSettle { + const state = probeStates.get(lease.accountId); + if (!state || state.leaseId !== lease.leaseId || state.generation !== lease.generation) { + return "stale"; + } + // `>=`, matching liveLease: at exactly the deadline the lease is already gone, so applying + // the outcome there would let a probe act on a lease the grant path would refuse to + // recognise -- two answers to the same instant. + if (now >= lease.expiresAt) return "expired"; + state.leaseId = undefined; + state.leaseExpiresAt = undefined; + state.lastOutcome = outcome; + return "applied"; +} + +/** + * Hand a lease back with no outcome -- the probe never reached upstream, so + * there is nothing to record. Only the holder may release; a stale lease is + * already dead and needs no cleanup. + */ +export function releaseTransientProbe(lease: TransientProbeLease): void { + const state = probeStates.get(lease.accountId); + if (!state || state.leaseId !== lease.leaseId || state.generation !== lease.generation) return; + state.leaseId = undefined; + state.leaseExpiresAt = undefined; +} + +/** + * Fence the epoch against newer authoritative state. A fresh failure recorded + * through the ordinary outcome path, or a binding that moved on, must not be + * overwritten by a probe result that was issued before it -- bumping the epoch + * makes every outstanding lease stale without waiting for its deadline. + */ +export function invalidateTransientProbe(accountId: string): void { + const state = probeStates.get(accountId); + if (!state) return; + state.generation += 1; + state.leaseId = undefined; + state.leaseExpiresAt = undefined; +} + +export interface TransientProbeDiagnostics { + readonly held: boolean; + readonly generation: number; + readonly leaseId?: string; + readonly leaseExpiresAt?: number; + readonly lastProbeAt?: number; + readonly lastOutcome?: TransientProbeOutcome; +} + +/** Current lease state for one account, for diagnostics. Never mutates. */ +export function transientProbeDiagnostics(accountId: string, now = Date.now()): TransientProbeDiagnostics { + const state = probeStates.get(accountId); + if (!state) return { held: false, generation: 0 }; + return { + held: liveLease(state, now), + generation: state.generation, + ...(state.leaseId !== undefined ? { leaseId: state.leaseId, leaseExpiresAt: state.leaseExpiresAt } : {}), + ...(state.lastProbeAt !== undefined ? { lastProbeAt: state.lastProbeAt } : {}), + ...(state.lastOutcome !== undefined ? { lastOutcome: state.lastOutcome } : {}), + }; +} + +/** Test seam: lease state is module-global and must not leak between cases. */ +export function clearTransientProbeLeasesForTests(): void { + probeStates.clear(); +} + +/** + * How many accounts currently carry probe state. Diagnostic, and the assertion surface for + * the {@link MAX_TRANSIENT_PROBE_STATES} bound. + */ +export function transientProbeStateCount(): number { + return probeStates.size; +} + +/** + * What a request may do while its bound account is held. + * + * - `probe`: this caller holds the lease and may send ONE trial to the held + * account. + * - `detour`: a probe is already out (or was refused); keep the remembered + * detour. The detour's identity survives the whole probing window -- a failed + * trial must not cost the caller its working route. + * - `withheld`: every candidate is held. The binding is remembered and dispatch + * is refused; `retryAt` is the earliest moment a probe could next go out. + * Sending anyway here is exactly the "must not send, sends anyway" defect the + * hold was added to close. + */ +export type HeldAccountDispatch = + | { kind: "probe"; lease: TransientProbeLease } + | { kind: "detour"; accountId: string } + | { kind: "withheld"; boundAccountId: string; detourAccountId?: string; retryAt: number }; + +/** + * Decide what a request bound to a held account may do this turn. The probe is + * tried first -- somebody has to find out whether the account is back, and the + * lease guarantees it is exactly one somebody. Everyone else keeps the detour, + * and a caller with no detour left is told to wait rather than sent at an + * account already known to be failing. + */ +export function resolveHeldAccountDispatch(input: { + boundAccountId: string; + detourAccountId?: string; + now?: number; + leaseMs?: number; + minProbeIntervalMs?: number; + backpressure?: PoolBackpressureLimiter; +}): HeldAccountDispatch { + const now = input.now ?? Date.now(); + const limiter = input.backpressure ?? sharedPoolBackpressure(); + // The lease check runs before the budget charge: a probe another holder already has out is + // not a dispatch, and charging the pool for it would shrink the recovery budget by phantom + // sends. Between the check and the grant there is no await, so eligibility cannot change. + if ( + canAcquireTransientProbe(input.boundAccountId, now, { + ...(input.minProbeIntervalMs !== undefined ? { minIntervalMs: input.minProbeIntervalMs } : {}), + }) + && limiter.tryPermitProbeDispatch(now) + ) { + const lease = tryAcquireTransientProbe(input.boundAccountId, now, { + ...(input.leaseMs !== undefined ? { leaseMs: input.leaseMs } : {}), + ...(input.minProbeIntervalMs !== undefined ? { minIntervalMs: input.minProbeIntervalMs } : {}), + }); + if (lease) return { kind: "probe", lease }; + } + if (input.detourAccountId !== undefined && input.detourAccountId !== input.boundAccountId) { + return { kind: "detour", accountId: input.detourAccountId }; + } + return { + kind: "withheld", + boundAccountId: input.boundAccountId, + ...(input.detourAccountId !== undefined ? { detourAccountId: input.detourAccountId } : {}), + retryAt: nextProbeAt(input.boundAccountId, now, input.minProbeIntervalMs), + }; +} + +/** Earliest moment a probe of this account could next be granted. */ +function nextProbeAt(accountId: string, now: number, minIntervalMs?: number): number { + const state = probeStates.get(accountId); + if (!state) return now; + const interval = minIntervalMs ?? TRANSIENT_PROBE_INTERVAL_MS; + const paced = state.lastProbeAt !== undefined ? state.lastProbeAt + interval : now; + const leased = liveLease(state, now) ? state.leaseExpiresAt! : now; + return Math.max(paced, leased); +} + +/* ------------------------------------------------------------------ */ +/* Pool-wide recovery backpressure */ +/* ------------------------------------------------------------------ */ + +export interface PoolBackpressurePolicy { + /** Sliding window the ratio is measured over. */ + readonly windowMs: number; + /** + * Recovery dispatches (retries + probes) admitted per observed initial send. + * 0.2 is the standard overload-guidance budget: at most one recovery send for + * every five new requests. + */ + readonly maxRetryRatio: number; + /** + * Floor under the ratio so a quiet pool can still recover: with almost no + * traffic a strict ratio admits nothing, which would wedge every held + * account behind a probe that can never run. + */ + readonly minRecoveryAllowance: number; +} + +export const DEFAULT_POOL_BACKPRESSURE_POLICY: PoolBackpressurePolicy = { + windowMs: 10_000, + maxRetryRatio: 0.2, + minRecoveryAllowance: 3, +}; + +export interface PoolBackpressureState { + readonly windowMs: number; + readonly initialSends: number; + readonly recoveryDispatches: number; + /** Dispatches admitted under the current window's allowance. */ + readonly allowance: number; + /** Lifetime refusals, including windows already rotated out. */ + readonly refusedTotal: number; + readonly ratioLimit: number; +} + +export interface PoolBackpressureLimiter { + /** A new request's FIRST send. Always recorded, never refused. */ + recordInitialSend(now?: number): void; + /** Admit one retry dispatch, or refuse when the window's ratio is spent. */ + tryPermitRetryDispatch(now?: number): boolean; + /** Admit one probe dispatch under the same shared recovery budget. */ + tryPermitProbeDispatch(now?: number): boolean; + state(now?: number): PoolBackpressureState; +} + +const BACKPRESSURE_BUCKETS = 10; + +/** + * Ratio limiter over a bucketed sliding window. Buckets give a sliding answer + * without keeping per-event state: the window is the sum of the buckets whose + * span falls inside it, and grant/refuse decisions read that sum. + */ +export function createPoolBackpressureLimiter( + policy: PoolBackpressurePolicy = DEFAULT_POOL_BACKPRESSURE_POLICY, +): PoolBackpressureLimiter { + const bucketMs = Math.max(1, Math.floor(policy.windowMs / BACKPRESSURE_BUCKETS)); + const buckets: Array<{ start: number; initials: number; recoveries: number }> = []; + let refusedTotal = 0; + + function bucketFor(now: number): { start: number; initials: number; recoveries: number } { + const start = Math.floor(now / bucketMs) * bucketMs; + const last = buckets[buckets.length - 1]; + if (last && last.start === start) return last; + while (buckets.length > 0 && buckets[0]!.start <= start - policy.windowMs) buckets.shift(); + const bucket = { start, initials: 0, recoveries: 0 }; + buckets.push(bucket); + return bucket; + } + + function totals(now: number): { initials: number; recoveries: number } { + let initials = 0; + let recoveries = 0; + for (const bucket of buckets) { + if (bucket.start <= now - policy.windowMs) continue; + initials += bucket.initials; + recoveries += bucket.recoveries; + } + return { initials, recoveries }; + } + + function allowanceFor(initials: number): number { + return Math.max(policy.minRecoveryAllowance, Math.floor(initials * policy.maxRetryRatio)); + } + + function tryPermit(now: number): boolean { + const bucket = bucketFor(now); + const { initials, recoveries } = totals(now); + if (recoveries + 1 > allowanceFor(initials)) { + refusedTotal += 1; + return false; + } + bucket.recoveries += 1; + return true; + } + + return { + recordInitialSend(now = Date.now()): void { + bucketFor(now).initials += 1; + }, + tryPermitRetryDispatch(now = Date.now()): boolean { + return tryPermit(now); + }, + tryPermitProbeDispatch(now = Date.now()): boolean { + return tryPermit(now); + }, + state(now = Date.now()): PoolBackpressureState { + const { initials, recoveries } = totals(now); + return { + windowMs: policy.windowMs, + initialSends: initials, + recoveryDispatches: recoveries, + allowance: allowanceFor(initials), + refusedTotal, + ratioLimit: policy.maxRetryRatio, + }; + }, + }; +} + +let sharedLimiter: PoolBackpressureLimiter | undefined; + +/** + * The process-wide limiter every recovery dispatch shares. A per-request + * limiter cannot see the storm, which is the entire reason this layer exists. + */ +export function sharedPoolBackpressure(): PoolBackpressureLimiter { + sharedLimiter ??= createPoolBackpressureLimiter(); + return sharedLimiter; +} + +/** + * Point the shared limiter at a different policy. The ceiling is deliberately + * configurable here and not yet plumbed into OcxConfig -- the wiring lane owns + * that seam; this is the knob it turns. + */ +export function configureSharedPoolBackpressure(policy: PoolBackpressurePolicy): void { + sharedLimiter = createPoolBackpressureLimiter(policy); +} + +/** Test seam: the shared limiter is module-global. */ +export function resetSharedPoolBackpressureForTests(): void { + sharedLimiter = undefined; +} diff --git a/src/types/config.ts b/src/types/config.ts index 4d985d64f5..bcc21c825d 100644 --- a/src/types/config.ts +++ b/src/types/config.ts @@ -882,6 +882,36 @@ export interface OcxConfig { * binding under either setting -- neither is a cache-affinity preference. */ cacheAffinity?: boolean; + /** + * Operator-declared quota domains: groups of credential ids that demonstrably share + * one upstream usage limit (#4546, wp6). Members of one group count once toward + * available capacity, and a quota refusal inside a group is never answered by + * rotating to another member -- the limit is the same, so the move would pay a cold + * prefix for zero new capacity. + * + * Declared groups speak only to quota. Sharing a usage limit says nothing about + * prompt-cache compatibility, which keeps its own provider-documented domain. + * Absent or empty means no declared grouping, so an unconfigured install behaves + * exactly as before. + * + * A declaration has to mean exactly one thing, so the config rejects the spellings + * that could mean two. Credential ids are provider-scoped elsewhere (the auth store + * keys an account by provider and id), so each member is written + * `":"` -- a bare `"acct-1"` names one credential per + * provider and would merge unrelated domains. The provider segment is matched + * case-insensitively through the usual aliases, so `chatgpt:` and `codex:` both mean + * OpenAI. Group ids must be unique, `credentials` must be non-empty, and a credential + * may belong to at most one group; a declaration that breaks any of those is rejected + * on write and dropped with a warning on load, never resolved by list order. + */ + credentialGroups?: Array<{ + /** Operator-chosen group identifier; only equality matters. */ + id: string; + /** Provider-qualified credential ids (`":"`), non-empty. */ + credentials: string[]; + /** Free-text provenance note for the operator's own records. */ + note?: string; + }>; }; /** Active pool account id for next session. undefined = main (passthrough as-is). */ activeCodexAccountId?: string; diff --git a/structure/catalog.md b/structure/catalog.md index 47a827426b..fc7e65a9d5 100644 --- a/structure/catalog.md +++ b/structure/catalog.md @@ -230,10 +230,48 @@ Pool mode routes across main plus added Codex credentials. Key rules: an omitted flag preserves the established behavior of a nonempty hand-written selector map. - **Rotation is sticky.** A conversation stays on its selected account while that account is usable; failure moves it, success does not (`src/codex/pool-rotation.ts`). +- **A transient hold is probed half-open, never opened all at once.** While a bound account is + held for a 5xx streak, one in-flight probe may test it and every other request keeps the + remembered detour; the lease carries a deadline and a generation so a late answer from a + probe that already lost cannot overwrite a newer binding or failure state. When every + candidate is held the caller gets a typed withheld outcome, not a send. Recovery dispatches + (retries and probes, never a new request's initial send) sit under a pool-wide ratio ceiling + measured over a sliding window (`src/routing/probe-lease.ts`). - **The credential store is generation-guarded.** A refresh takes a lock and persists only if the generation it started from still holds; a lost race raises a generation-conflict error rather than overwriting the newer credential (`src/codex/account-store.ts`). Callers handle that error; they do not assume a silent retry. +- **Authentication identity, quota domain, and cache domain are tracked separately** + (`src/routing/identity-domains.ts`). `classifyCredential` returns all three with provenance: + `pool.credentialGroups` supplies operator-declared quota domains, a small built-in table + supplies the provider-documented cases (OpenAI limits per organization and project and caches + per organization and region, Anthropic cache per workspace, Azure per deployment), and every + other answer is `unknown`. `unknown` is a first-class relation result, never silently read as + shared or as distinct: `assessQuotaRotation` reports `same-domain` so a quota refusal is not + answered by rotating inside the limit that refused, `countQuotaCapacity` counts one known + domain once and reports unknown-domain credentials separately, and + `canPortConversationState` keeps conversational-state portability a separate question from + cache compatibility by refusing any request that carries `previous_response_id`, a + provider-side conversation id, uploaded file ids, or encrypted reasoning. The classifier is groundwork that no routing boundary calls yet: it lands with its tests + so the consuming layers can be reviewed one at a time. Until one of them wires it, declaring + `pool.credentialGroups` changes no routing decision, and the rules above state the contract + those consumers must honour rather than behaviour an operator can rely on today. +- **Proven separation and proven sharing are separate facts** (`src/routing/identity-domains.ts`). + Every domain carries `evidence` alongside its provenance: a rule that documents only that two + credentials are in different domains never lets an equal key mean "shared". OpenAI's cache rule + is the case that forces it — caches are documented as not shared across organizations or + processing regions, while changing keys inside one organization is documented as not + guaranteeing a hit, so a different org or region relates `distinct` and the same org and region + relates `unknown`. OpenAI quota, Anthropic workspace cache, and Azure deployment domains carry + the sharing half as well and still relate `shared`. +- **A declared credential group cannot mean two things** (`src/routing/identity-domains.ts`, + `src/config.ts`). `credentialGroupIssues` is the one definition of a valid grouping: unique + group ids, a non-empty member list, and each credential in at most one group, with members + written `":"` because ids are provider-scoped in the auth store. The + config write path rejects a declaration that breaks any of those and the load path drops the + list with a warning, keeping `pool.kernel` and `pool.cacheAffinity`; `classifyCredential` + reports an ambiguous claim on `declaredGroupConflict` and falls back to the documented or + unknown answer rather than taking the first matching group. Warmup issues a bounded request with a fallback model so a cold account reports usability before a real turn depends on it (`src/codex/warmup.ts`). diff --git a/tests/config/config-load-degrade.test.ts b/tests/config/config-load-degrade.test.ts index 9e98c709ea..62dd4b24fe 100644 --- a/tests/config/config-load-degrade.test.ts +++ b/tests/config/config-load-degrade.test.ts @@ -339,3 +339,63 @@ test("an invalid desktopProfile is dropped without resetting providers (#4430)", expect(error.mock.calls.join("\n")).not.toContain("Using default config"); } finally { error.mockRestore(); } }); + +function writePoolConfig(credentialGroups: unknown): string { + const bytes = JSON.stringify({ + ...candidate(undefined), + pool: { kernel: true, cacheAffinity: false, credentialGroups }, + }); + writeFileSync(getConfigPath(), bytes); + return bytes; +} + +test("a malformed credentialGroups entry costs the list, not the rest of pool (#4546)", () => { + const bytes = writePoolConfig([ + { id: "team", credentials: ["openai:key-a"] }, + { id: "", credentials: [] }, + ]); + const warn = spyOn(console, "warn").mockImplementation(() => {}); + try { + const loaded = loadConfig(); + expect(loaded.pool?.credentialGroups).toBeUndefined(); + // The two siblings are live routing settings: an outer catch used to take them both + // because one group failed the nested object. + expect(loaded.pool?.kernel).toBe(true); + expect(loaded.pool?.cacheAffinity).toBe(false); + expect(loaded.providers.xai.note).toBe("keep me"); + expect(warn.mock.calls.flat().join("\n")).toContain("pool.credentialGroups"); + expect(readFileSync(getConfigPath(), "utf8")).toBe(bytes); + } finally { warn.mockRestore(); } +}); + +test("an ambiguous credentialGroups declaration is rejected on write, never ordered away (#4546)", () => { + const base = candidate(undefined); + const withGroups = (credentialGroups: unknown) => ({ ...base, pool: { kernel: true, credentialGroups } }); + + const twoGroups = validateConfigCandidate(withGroups([ + { id: "left", credentials: ["openai:key-a"] }, + { id: "right", credentials: ["openai:key-a"] }, + ])); + expect(twoGroups.ok).toBe(false); + expect(twoGroups.ok === false && twoGroups.error).toContain("pool.credentialGroups"); + + const duplicateId = validateConfigCandidate(withGroups([ + { id: "team", credentials: ["openai:key-a"] }, + { id: "team", credentials: ["openai:key-b"] }, + ])); + expect(duplicateId.ok).toBe(false); + expect(duplicateId.ok === false && duplicateId.error).toContain("duplicate group id"); + + const bareId = validateConfigCandidate(withGroups([{ id: "team", credentials: ["key-a"] }])); + expect(bareId.ok).toBe(false); + expect(bareId.ok === false && bareId.error).toContain("provider-qualified"); + + const empty = validateConfigCandidate(withGroups([{ id: "team", credentials: [] }])); + expect(empty.ok).toBe(false); + + const valid = validateConfigCandidate(withGroups([ + { id: "team", credentials: ["openai:key-a", "azure:key-a"], note: "one billed org" }, + ])); + expect(valid.ok).toBe(true); + expect(valid.ok === true && valid.config.pool?.credentialGroups).toHaveLength(1); +}); diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 0888b0825f..8500c4fc78 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -869,6 +869,7 @@ "prime-client.test.ts": "clients", "privacy-mask-account.test.ts": "lib", "privacy-scan-meta-key.test.ts": "ci-workflows", + "probe-lease.test.ts": "routing", "process-control-graceful.test.ts": "lib", "process-control.test.ts": "lib", "process-state.test.ts": "service", @@ -1036,6 +1037,7 @@ "routing-compatibility-boundaries.test.ts": "routing", "routing-compatibility-model-matching.test.ts": "routing", "routing-compatibility.test.ts": "routing", + "routing-identity-domains.test.ts": "routing", "routing-intelligence-ui.test.ts": "gui", "routing-policy-fallback.test.ts": "routing", "routing-policy-pool-quota.test.ts": "routing", @@ -1098,6 +1100,8 @@ "skill-ocx.test.ts": "ci-workflows", "slug-codec.test.ts": "codex-integration", "sponsor-presets.test.ts": "providers", + "spend-ledger-file-journal.test.ts": "lib", + "spend-reservation-ledger.test.ts": "lib", "sse-client-frame-bounds.test.ts": "responses", "sse-decoder.test.ts": "responses", "sse-failed-tail.test.ts": "responses", @@ -1241,6 +1245,7 @@ "windows-user-principal.test.ts": "windows", "winsw-stop-hardening.test.ts": "windows", "winsw.test.ts": "service", + "workflow-budget.test.ts": "lib", "ws-endpoint.test.ts": "responses", "ws-failure-stage.test.ts": "responses", "ws-upstream-reuse.test.ts": "responses", diff --git a/tests/lib/spend-ledger-file-journal.test.ts b/tests/lib/spend-ledger-file-journal.test.ts new file mode 100644 index 0000000000..f2c10952fd --- /dev/null +++ b/tests/lib/spend-ledger-file-journal.test.ts @@ -0,0 +1,65 @@ +import { describe, expect, test } from "bun:test"; +import { chmodSync, mkdtempSync, readFileSync, readdirSync, statSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + createFileSpendJournal, + loadOrCreateSpendLedgerSalt, +} from "../../src/lib/spend-reservation-ledger"; + +/** POSIX mode bits do not describe a Windows ACL, where hardenSecretPath does the work. */ +const posixModes = process.platform !== "win32"; +const modeOf = (path: string): number => statSync(path).mode & 0o777; +const line = (send: string): string => JSON.stringify({ v: 1, kind: "lost", send, at: 1 }); + +describe("spend ledger file journal", () => { + test.skipIf(!posixModes)("a journal that already exists is re-hardened, not trusted", () => { + const dir = mkdtempSync(join(tmpdir(), "ocx-spend-journal-")); + const path = join(dir, "spend-ledger.jsonl"); + const journal = createFileSpendJournal(path); + + journal.append(line("alias-one")); + expect(modeOf(path)).toBe(0o600); + + // `mode` in a write option applies only when the file is CREATED. A journal left + // group-readable by an older build, a restored backup or a lax umask would keep that mode + // for its whole life, which is the gap this closes. + chmodSync(path, 0o644); + journal.append(line("alias-two")); + expect(modeOf(path)).toBe(0o600); + + chmodSync(path, 0o644); + expect(journal.read()).toHaveLength(2); + expect(modeOf(path)).toBe(0o600); + }); + + test("compaction replaces the journal atomically and leaves no temp behind", () => { + const dir = mkdtempSync(join(tmpdir(), "ocx-spend-compact-")); + const path = join(dir, "spend-ledger.jsonl"); + const journal = createFileSpendJournal(path); + journal.append(line("alias-one")); + journal.append(line("alias-two")); + + const rewrite = journal.rewrite; + expect(rewrite).toBeDefined(); + rewrite?.call(journal, [line("checkpoint-stand-in")]); + + expect(readFileSync(path, "utf8")).toBe(line("checkpoint-stand-in") + "\n"); + expect(journal.read()).toHaveLength(1); + // The temp file is renamed over the journal, never left in the home directory. + expect(readdirSync(dir)).toEqual(["spend-ledger.jsonl"]); + if (posixModes) expect(modeOf(path)).toBe(0o600); + }); + + test("the alias salt is minted once and reused, so replay still matches live requests", () => { + const dir = mkdtempSync(join(tmpdir(), "ocx-spend-salt-")); + const path = join(dir, "spend-ledger.salt"); + + const minted = loadOrCreateSpendLedgerSalt(path); + expect(minted).toMatch(/^[0-9a-f]{64}$/); + // Stability is the whole contract: a salt that changed per process would alias the same + // root id differently after a restart and hand every scope a fresh allowance. + expect(loadOrCreateSpendLedgerSalt(path)).toBe(minted); + if (posixModes) expect(modeOf(path)).toBe(0o600); + }); +}); diff --git a/tests/lib/spend-reservation-ledger.test.ts b/tests/lib/spend-reservation-ledger.test.ts new file mode 100644 index 0000000000..9440504bea --- /dev/null +++ b/tests/lib/spend-reservation-ledger.test.ts @@ -0,0 +1,397 @@ +import { describe, expect, test } from "bun:test"; +import { + createSpendReservationLedger, + parseSpendJournalRecord, + type SpendJournal, + type SpendReservationPolicy, +} from "../../src/lib/spend-reservation-ledger"; + +/** In-memory journal: same replay and compaction contract as the file store, without disk. */ +const memoryJournal = (): SpendJournal & { lines: string[] } => { + const lines: string[] = []; + return { + lines, + read: () => [...lines], + append: (line) => { lines.push(line); }, + rewrite: (next) => { lines.length = 0; lines.push(...next); }, + }; +}; + +/** A journal that cannot persist: the disk-full and permission case durability exists for. */ +const unwritableJournal = (): SpendJournal => ({ + read: () => [], + append: () => { throw new Error("ENOSPC: no space left on device"); }, +}); + +const policy = (maxTokens: number | undefined, retentionMs = 60_000): SpendReservationPolicy => ({ + root: { maxTokens }, + identity: { maxTokens }, + pool: { maxTokens }, + retentionMs, +}); + +describe("spend reservation ledger", () => { + test("reserves input plus the enforceable output ceiling and refuses at the boundary", () => { + const ledger = createSpendReservationLedger({ policy: policy(100), now: () => 1_000 }); + // 60 input + 40 ceiling = 100 exactly: the boundary admits. + expect(ledger.reserve({ + sendId: "s1", + scopes: { rootId: "r1" }, + inputTokens: 60, + outputCeilingTokens: 40, + }).reserved).toBe(true); + // One more token projects past the limit and is refused, naming the scope. + const denied = ledger.reserve({ + sendId: "s2", + scopes: { rootId: "r1" }, + inputTokens: 1, + outputCeilingTokens: 0, + }); + expect(denied.reserved).toBe(false); + if (!denied.reserved) { + expect(denied.denial.scope).toBe("root"); + expect(denied.denial.limit).toBe(100); + expect(denied.denial.projected).toBe(101); + } + // The refused reservation booked nothing: settling its send id is a no-op. + expect(ledger.settle("s2", { inputTokens: 1, outputTokens: 0 })).toBe(false); + }); + + test("enforces root, identity and pool scopes at once, so a fresh root id mints no budget", () => { + const ledger = createSpendReservationLedger({ policy: policy(100), now: () => 1_000 }); + const req = (sendId: string, rootId: string) => ({ + sendId, + scopes: { rootId, identityId: "user-1", poolId: "pool-1" }, + inputTokens: 60, + outputCeilingTokens: 40, + }); + expect(ledger.reserve(req("s1", "root-a")).reserved).toBe(true); + // A brand-new root still carries the identity and pool spend: all three scopes are + // checked, so laundering through a fresh root id fails on the identity scope. + const denied = ledger.reserve(req("s2", "root-b")); + expect(denied.reserved).toBe(false); + if (!denied.reserved) expect(denied.denial.scope).toBe("identity"); + // A different identity under the same pool is still stopped at the pool scope. + const poolDenied = ledger.reserve({ + sendId: "s3", + scopes: { rootId: "root-c", identityId: "user-2", poolId: "pool-1" }, + inputTokens: 60, + outputCeilingTokens: 40, + }); + expect(poolDenied.reserved).toBe(false); + if (!poolDenied.reserved) expect(poolDenied.denial.scope).toBe("pool"); + }); + + test("settlement is idempotent per send id", () => { + const ledger = createSpendReservationLedger({ policy: policy(1_000), now: () => 1_000 }); + ledger.reserve({ sendId: "s1", scopes: { rootId: "r1" }, inputTokens: 100, outputCeilingTokens: 100 }); + expect(ledger.settle("s1", { inputTokens: 90, outputTokens: 10 })).toBe(true); + // The double settlement books nothing: reserved stays released exactly once. + expect(ledger.settle("s1", { inputTokens: 90, outputTokens: 10 })).toBe(false); + const snap = ledger.snapshot("root", "r1"); + expect(snap?.settled).toBe(100); + expect(snap?.reserved).toBe(0); + // markLost after a settlement is likewise a no-op. + expect(ledger.markLost("s1")).toBe(false); + }); + + test("lost usage becomes unresolved spend instead of being released", () => { + const ledger = createSpendReservationLedger({ policy: policy(150), now: () => 1_000 }); + ledger.reserve({ sendId: "s1", scopes: { rootId: "r1" }, inputTokens: 100, outputCeilingTokens: 50 }); + expect(ledger.markLost("s1")).toBe(true); + const snap = ledger.snapshot("root", "r1"); + expect(snap?.reserved).toBe(0); + expect(snap?.unresolved).toBe(150); + // Unresolved spend still counts: the full reservation may have been billed. + expect(ledger.exhausted("root", "r1")).toBe(true); + expect(ledger.reserve({ + sendId: "s2", scopes: { rootId: "r1" }, inputTokens: 1, outputCeilingTokens: 0, + }).reserved).toBe(false); + }); + + test("an exhausted root stays exhausted across a simulated restart", () => { + const journal = memoryJournal(); + const first = createSpendReservationLedger({ journal, policy: policy(100), now: () => 1_000 }); + first.reserve({ sendId: "s1", scopes: { rootId: "r1" }, inputTokens: 60, outputCeilingTokens: 40 }); + first.settle("s1", { inputTokens: 60, outputTokens: 40 }); + expect(first.exhausted("root", "r1")).toBe(true); + + // Restart: a new ledger replays the same journal and refuses the same root. + const second = createSpendReservationLedger({ journal, policy: policy(100), now: () => 2_000 }); + expect(second.exhausted("root", "r1")).toBe(true); + expect(second.reserve({ + sendId: "s2", scopes: { rootId: "r1" }, inputTokens: 1, outputCeilingTokens: 0, + }).reserved).toBe(false); + // And the replayed settlement is still idempotent after the rebuild. + expect(second.settle("s1", { inputTokens: 60, outputTokens: 40 })).toBe(false); + }); + + test("the unconfigured default observes spend but refuses nothing", () => { + const ledger = createSpendReservationLedger({ now: () => 1_000 }); + for (let i = 0; i < 10; i += 1) { + expect(ledger.reserve({ + sendId: `s${i}`, scopes: { rootId: "r1" }, inputTokens: 1_000_000, outputCeilingTokens: 1_000_000, + }).reserved).toBe(true); + } + const snap = ledger.snapshot("root", "r1"); + expect(snap?.reserved).toBe(20_000_000); + expect(snap?.exhausted).toBe(false); + }); + + test("prune removes a dormant under-limit scope but never an exhausted one", () => { + const journal = memoryJournal(); + const ledger = createSpendReservationLedger({ journal, policy: policy(100, 1_000), now: () => 0 }); + ledger.reserve({ sendId: "s1", scopes: { rootId: "spent" }, inputTokens: 60, outputCeilingTokens: 40 }); + ledger.settle("s1", { inputTokens: 60, outputTokens: 40 }); + ledger.reserve({ sendId: "s2", scopes: { rootId: "light" }, inputTokens: 10, outputCeilingTokens: 0 }); + ledger.settle("s2", { inputTokens: 10, outputTokens: 0 }); + + ledger.prune(10_000); + // Both are idle and past the retention window, but only the under-limit one may go. + expect(ledger.snapshot("root", "light")).toBeUndefined(); + const spent = ledger.snapshot("root", "spent"); + expect(spent?.exhausted).toBe(true); + expect(ledger.reserve({ + sendId: "s3", scopes: { rootId: "spent" }, inputTokens: 1, outputCeilingTokens: 0, + }).reserved).toBe(false); + }); + + test("a torn tail line in the journal is skipped on replay", () => { + const journal = memoryJournal(); + const first = createSpendReservationLedger({ journal, policy: policy(100), now: () => 1_000 }); + first.reserve({ sendId: "s1", scopes: { rootId: "r1" }, inputTokens: 60, outputCeilingTokens: 40 }); + journal.lines.push("{not-json"); + const second = createSpendReservationLedger({ journal, policy: policy(100), now: () => 2_000 }); + expect(second.exhausted("root", "r1")).toBe(true); + // Quietly: the final record is the one that never finished being written, so nothing + // after it is missing and no total is understated. + expect(second.corruptRecords).toBe(0); + expect(second.degraded).toBe(false); + }); +}); + +describe("spend reservation ledger, send identity", () => { + test("a duplicate send id is refused instead of authorising a free dispatch", () => { + const journal = memoryJournal(); + const ledger = createSpendReservationLedger({ journal, policy: policy(1_000), now: () => 1_000 }); + const request = { sendId: "s1", scopes: { rootId: "r1" }, inputTokens: 10, outputCeilingTokens: 10 }; + expect(ledger.reserve(request).reserved).toBe(true); + + // The old behaviour returned success here while booking nothing, so one id bought an + // unlimited number of physical sends with the scope totals frozen. + const repeat = ledger.reserve(request); + expect(repeat.reserved).toBe(false); + if (!repeat.reserved) expect(repeat.denial.reason).toBe("duplicate-send-id"); + expect(ledger.snapshot("root", "r1")?.reserved).toBe(20); + + // Still refused once the original send resolves... + expect(ledger.settle("s1", { inputTokens: 10, outputTokens: 10 })).toBe(true); + expect(ledger.reserve(request).reserved).toBe(false); + expect(ledger.snapshot("root", "r1")?.settled).toBe(20); + + // ...and after a restart rebuilds the ledger from the journal. + const restarted = createSpendReservationLedger({ journal, policy: policy(1_000), now: () => 2_000 }); + expect(restarted.knows("s1")).toBe(true); + expect(restarted.reserve(request).reserved).toBe(false); + }); + + test("an undispatched reservation is released and only a dispatched one becomes unresolved", () => { + const ledger = createSpendReservationLedger({ policy: policy(1_000), now: () => 1_000 }); + ledger.reserve({ sendId: "never-sent", scopes: { rootId: "r1" }, inputTokens: 40, outputCeilingTokens: 10 }); + expect(ledger.abandon("never-sent")).toBe(true); + const released = ledger.snapshot("root", "r1"); + expect(released?.reserved).toBe(0); + expect(released?.unresolved).toBe(0); + expect(released?.settled).toBe(0); + // Abandoning is terminal, and the id stays known so it cannot be replayed. + expect(ledger.markLost("never-sent")).toBe(false); + expect(ledger.knows("never-sent")).toBe(true); + + ledger.reserve({ sendId: "sent", scopes: { rootId: "r1" }, inputTokens: 40, outputCeilingTokens: 10 }); + expect(ledger.markDispatched("sent")).toBe(true); + // Bytes left for upstream, so the tokens may already be billed and cannot be handed back. + expect(ledger.abandon("sent")).toBe(false); + expect(ledger.markLost("sent")).toBe(true); + expect(ledger.snapshot("root", "r1")?.unresolved).toBe(50); + }); +}); + +describe("spend reservation ledger, durability", () => { + test("under a configured limit a reservation that cannot be persisted is refused", () => { + const ledger = createSpendReservationLedger({ + journal: unwritableJournal(), policy: policy(1_000), now: () => 1_000, + }); + const denied = ledger.reserve({ + sendId: "s1", scopes: { rootId: "r1" }, inputTokens: 10, outputCeilingTokens: 10, + }); + // Admitting here would keep the request but forget it across a restart, which defeats the + // durable ceiling in exactly the disk-full and permission cases durability exists for. + expect(denied.reserved).toBe(false); + if (!denied.reserved) expect(denied.denial.reason).toBe("reserve-not-durable"); + // And it booked nothing: no scope was created and the id was not remembered. + expect(ledger.snapshot("root", "r1")).toBeUndefined(); + expect(ledger.knows("s1")).toBe(false); + expect(ledger.persistFailures).toBe(1); + expect(ledger.degraded).toBe(true); + }); + + test("observe-only mode still admits, and says the reservation is not durable", () => { + const ledger = createSpendReservationLedger({ + journal: unwritableJournal(), policy: policy(undefined), now: () => 1_000, + }); + const decision = ledger.reserve({ + sendId: "s1", scopes: { rootId: "r1" }, inputTokens: 10, outputCeilingTokens: 10, + }); + expect(decision.reserved).toBe(true); + if (decision.reserved) expect(decision.durable).toBe(false); + expect(ledger.degraded).toBe(true); + // An unconfigured install refuses nothing, so the accounting continues in memory. + expect(ledger.snapshot("root", "r1")?.reserved).toBe(20); + }); +}); + +describe("spend reservation ledger, journal validation", () => { + test("every malformed record shape is rejected rather than asserted into the replay", () => { + // Each of these used to be type-asserted straight into the rebuild: `null` crashed at + // record.v and the field-less reserve crashed inside applyReserve. + expect(parseSpendJournalRecord("null")).toBeUndefined(); + expect(parseSpendJournalRecord("[]")).toBeUndefined(); + expect(parseSpendJournalRecord("{not-json")).toBeUndefined(); + expect(parseSpendJournalRecord(JSON.stringify({ v: 1, kind: "reserve" }))).toBeUndefined(); + expect(parseSpendJournalRecord(JSON.stringify({ v: 2, kind: "lost", send: "a", at: 1 }))).toBeUndefined(); + expect(parseSpendJournalRecord(JSON.stringify({ v: 1, kind: "nope", send: "a", at: 1 }))).toBeUndefined(); + expect(parseSpendJournalRecord(JSON.stringify({ v: 1, kind: "lost", send: "a", at: -1 }))).toBeUndefined(); + expect(parseSpendJournalRecord(JSON.stringify({ v: 1, kind: "lost", send: "", at: 1 }))).toBeUndefined(); + expect(parseSpendJournalRecord(JSON.stringify({ + v: 1, kind: "reserve", send: "a", targets: [{ scope: "elsewhere", alias: "b" }], tokens: 1, at: 1, + }))).toBeUndefined(); + expect(parseSpendJournalRecord(JSON.stringify({ + v: 1, kind: "reserve", send: "a", targets: [{ scope: "root", alias: "b" }], tokens: Number.NaN, at: 1, + }))).toBeUndefined(); + expect(parseSpendJournalRecord(JSON.stringify({ v: 1, kind: "settle", send: "a", tokens: -5, at: 1 }))).toBeUndefined(); + expect(parseSpendJournalRecord(JSON.stringify({ v: 1, kind: "drop", scope: "root", at: 1 }))).toBeUndefined(); + // The one well-formed shape survives. + expect(parseSpendJournalRecord(JSON.stringify({ + v: 1, kind: "reserve", send: "a", targets: [{ scope: "root", alias: "b" }], tokens: 5, at: 7, + }))).toBeDefined(); + }); + + test("corruption in the middle of the journal fails accounting closed", () => { + const journal = memoryJournal(); + const first = createSpendReservationLedger({ journal, policy: policy(1_000), now: () => 1_000 }); + first.reserve({ sendId: "s1", scopes: { rootId: "r1" }, inputTokens: 10, outputCeilingTokens: 10 }); + first.settle("s1", { inputTokens: 10, outputTokens: 10 }); + // Records AFTER this one completed, so dropping it quietly would understate the root and + // hand back budget. Only a torn tail may be dropped. + journal.lines.splice(1, 0, "null"); + + const second = createSpendReservationLedger({ journal, policy: policy(1_000), now: () => 2_000 }); + expect(second.corruptRecords).toBe(1); + expect(second.degraded).toBe(true); + const denied = second.reserve({ + sendId: "s2", scopes: { rootId: "r1" }, inputTokens: 1, outputCeilingTokens: 0, + }); + expect(denied.reserved).toBe(false); + if (!denied.reserved) expect(denied.denial.reason).toBe("journal-corrupt"); + + // Observe-only accounting is not refused by it: there is no ceiling to enforce wrongly. + const observing = createSpendReservationLedger({ journal, policy: policy(undefined), now: () => 2_000 }); + expect(observing.reserve({ + sendId: "s3", scopes: { rootId: "r1" }, inputTokens: 1, outputCeilingTokens: 0, + }).reserved).toBe(true); + }); +}); + +describe("spend reservation ledger, bounded retention", () => { + const bounded = (overrides: Partial = {}): SpendReservationPolicy => ({ + root: {}, identity: {}, pool: {}, + retentionMs: 1_000, + maxTrackedScopes: 2, + maxTrackedSends: 2, + compactAfterRecords: 6, + ...overrides, + }); + + test("cleanup runs without a caller, and the journal does not resurrect what it removed", () => { + const journal = memoryJournal(); + let clock = 0; + const ledger = createSpendReservationLedger({ journal, policy: bounded(), now: () => clock }); + // Twelve unique root ids and twelve unique send ids, which is the shape that grew both + // Maps and the journal without bound when nothing called prune(). + for (let i = 0; i < 12; i += 1) { + clock = i * 10_000; + expect(ledger.reserve({ + sendId: `s${i}`, scopes: { rootId: `r${i}` }, inputTokens: 1, outputCeilingTokens: 0, + }).reserved).toBe(true); + expect(ledger.settle(`s${i}`, { inputTokens: 1, outputTokens: 0 })).toBe(true); + } + expect(ledger.snapshot("root", "r0")).toBeUndefined(); + expect(ledger.knows("s0")).toBe(false); + expect(ledger.snapshot("root", "r11")?.settled).toBe(1); + + // The tombstones and the checkpoint are what make that durable: a restart rebuilds the + // bounded state rather than every id the process ever saw. + const restarted = createSpendReservationLedger({ journal, policy: bounded(), now: () => clock }); + expect(restarted.snapshot("root", "r0")).toBeUndefined(); + expect(restarted.knows("s0")).toBe(false); + expect(restarted.snapshot("root", "r11")?.settled).toBe(1); + expect(restarted.corruptRecords).toBe(0); + // And the file itself stayed small instead of carrying two records per unique id. + expect(journal.lines.length).toBeLessThan(12); + }); + + test("a full tracking table refuses admission rather than forgetting an exhausted scope", () => { + let clock = 1_000; + const ledger = createSpendReservationLedger({ + policy: bounded({ root: { maxTokens: 100 }, maxTrackedSends: 64 }), + now: () => clock, + }); + for (const root of ["a", "b"]) { + expect(ledger.reserve({ + sendId: `s-${root}`, scopes: { rootId: root }, inputTokens: 100, outputCeilingTokens: 0, + }).reserved).toBe(true); + expect(ledger.settle(`s-${root}`, { inputTokens: 100, outputTokens: 0 })).toBe(true); + } + clock = 9_000; + // Both tracked scopes are spent, so there is no safe eviction candidate. Making room by + // dropping one would hand it a fresh allowance under the same id. + const denied = ledger.reserve({ + sendId: "s-c", scopes: { rootId: "c" }, inputTokens: 1, outputCeilingTokens: 0, + }); + expect(denied.reserved).toBe(false); + if (!denied.reserved) expect(denied.denial.reason).toBe("tracking-capacity-exhausted"); + expect(ledger.exhausted("root", "a")).toBe(true); + expect(ledger.exhausted("root", "b")).toBe(true); + expect(ledger.snapshot("root", "c")).toBeUndefined(); + }); +}); + +describe("spend reservation ledger, privacy", () => { + test("the journal stores salted aliases, never a root header, credential or pool id", () => { + const journal = memoryJournal(); + const scopes = { rootId: "thread_0123456789", identityId: "cred-jun@example.com", poolId: "pool-prod" }; + const ledger = createSpendReservationLedger({ + journal, policy: policy(1_000), now: () => 1_000, salt: "install-one", + }); + ledger.reserve({ sendId: "send-abc", scopes, inputTokens: 10, outputCeilingTokens: 0 }); + ledger.markDispatched("send-abc"); + ledger.settle("send-abc", { inputTokens: 10, outputTokens: 0 }); + + const written = journal.lines.join("\n"); + for (const raw of ["send-abc", "thread_0123456789", "cred-jun@example.com", "pool-prod"]) { + expect(written).not.toContain(raw); + } + + // The alias is stable for one install, so a restart still finds the same spend... + const restarted = createSpendReservationLedger({ + journal, policy: policy(1_000), now: () => 2_000, salt: "install-one", + }); + expect(restarted.snapshot("root", "thread_0123456789")?.settled).toBe(10); + expect(restarted.snapshot("identity", "cred-jun@example.com")?.settled).toBe(10); + // ...and unrecoverable with anything but that install's salt. + const stranger = createSpendReservationLedger({ + journal, policy: policy(1_000), now: () => 2_000, salt: "install-two", + }); + expect(stranger.snapshot("root", "thread_0123456789")).toBeUndefined(); + }); +}); diff --git a/tests/lib/upstream-retry.test.ts b/tests/lib/upstream-retry.test.ts index af98c18552..4e014c7645 100644 --- a/tests/lib/upstream-retry.test.ts +++ b/tests/lib/upstream-retry.test.ts @@ -1,6 +1,7 @@ import { afterEach, describe, expect, spyOn, test } from "bun:test"; import { fetchWithResetRetry, + fetchWithTransientRetry, isConnectionResetError, prepareSameTarget429Wait, releaseResponseBodyBestEffort, @@ -263,15 +264,103 @@ describe("retryBackoffDelayMs", () => { })).toBe(30_000); }); - test("an honoured Retry-After is still ceilinged so it cannot park a request (#4546)", () => { + test("an honoured Retry-After is preserved in full, never shortened (#4546)", () => { const headers = new Headers({ "Retry-After": "3600" }); + // The instruction is the provider's statement of when it will serve again. Clamping it + // to a local ceiling produced a send the upstream already said it would refuse; whether + // the request can wait that long is the caller's deadline decision, not a shorter delay. expect(retryBackoffDelayMs(0, { baseDelayMs: 250, maxDelayMs: 5_000, headers, retryAfterIsLowerBound: true, retryAfterCeilingMs: 60_000, - })).toBe(60_000); + })).toBe(3_600_000); + }); + + test("an instruction past the wait deadline ends with the upstream answer intact (#4546)", async () => { + silenceWarn(); + const upstream = new Response("overloaded", { + status: 503, + headers: { "Retry-After": "3600" }, + }); + const { calls, doFetch } = mockDoFetch([upstream]); + const res = await fetchWithTransientRetry(doFetch); + // No early retry: one send, and the caller gets the real 503 with its Retry-After + // rather than a second refusal the provider already announced. + expect(calls.length).toBe(1); + expect(res.status).toBe(503); + expect(res.headers.get("retry-after")).toBe("3600"); + }); + + test("an instruction inside the wait deadline is still honoured before retrying (#4546)", async () => { + silenceWarn(); + const limited = new Response("overloaded", { + status: 503, + headers: { "Retry-After": "1" }, + }); + const ok = new Response("fine", { status: 200 }); + const { calls, doFetch } = mockDoFetch([limited, ok]); + const started = Date.now(); + const res = await fetchWithTransientRetry(doFetch); + expect(res.status).toBe(200); + expect(calls.length).toBe(2); + expect(Date.now() - started).toBeGreaterThanOrEqual(900); + }); + + test("a caller deadline shorter than the default is not slept past (#4546)", async () => { + silenceWarn(); + const limited = new Response("overloaded", { + status: 503, + headers: { "Retry-After": "1" }, + }); + const ok = new Response("fine", { status: 200 }); + const { calls, doFetch } = mockDoFetch([limited, ok]); + const started = Date.now(); + // The caller can wait 500ms; the upstream asked for 1s. Reading the module default + // instead of this deadline parked the request for the full second -- the 30s-budget / + // 45s-instruction shape, scaled down so the test does not have to sleep it. + const res = await fetchWithTransientRetry(doFetch, { retryAfterCeilingMs: 500 }); + expect(calls.length).toBe(1); + expect(res.status).toBe(503); + expect(res.headers.get("retry-after")).toBe("1"); + expect(Date.now() - started).toBeLessThan(500); + }); + + test("an instruction exactly at the caller deadline is honoured, not refused (#4546)", async () => { + silenceWarn(); + const limited = new Response("overloaded", { + status: 503, + headers: { "Retry-After": "1" }, + }); + const ok = new Response("fine", { status: 200 }); + const { calls, doFetch } = mockDoFetch([limited, ok]); + const started = Date.now(); + // Equality is inside the budget: the deadline is what the caller CAN wait, so a wait of + // exactly that length is affordable and the retry happens after it. + const res = await fetchWithTransientRetry(doFetch, { retryAfterCeilingMs: 1_000 }); + expect(res.status).toBe(200); + expect(calls.length).toBe(2); + expect(Date.now() - started).toBeGreaterThanOrEqual(900); + }); + + test("a caller deadline longer than the default waits instead of ending early (#4546)", async () => { + silenceWarn(); + const limited = new Response("overloaded", { + status: 503, + headers: { "Retry-After": "90" }, + }); + const { calls, doFetch } = mockDoFetch([limited, new Response("fine", { status: 200 })]); + const ac = new AbortController(); + // 90s is past the module default but inside this caller's 120s deadline, so the call must + // be waiting -- not returning the 503 the default ceiling used to hand back immediately. + // Aborting mid-wait is how the test observes the wait without sitting through it. + setTimeout(() => ac.abort(new DOMException("deadline probe", "AbortError")), 20); + await expect(fetchWithTransientRetry(doFetch, { + retryAfterCeilingMs: 120_000, + abortSignal: ac.signal, + })).rejects.toThrow("deadline probe"); + expect(calls.length).toBe(1); }); test("opting in never shortens a wait below the local backoff (#4546)", () => { diff --git a/tests/lib/workflow-budget.test.ts b/tests/lib/workflow-budget.test.ts new file mode 100644 index 0000000000..f21ecc1ef9 --- /dev/null +++ b/tests/lib/workflow-budget.test.ts @@ -0,0 +1,211 @@ +import { beforeEach, describe, expect, test } from "bun:test"; +import { + createSpendReservationLedger, + type SpendJournal, + type SpendReservationPolicy, +} from "../../src/lib/spend-reservation-ledger"; +import { + admitWorkflowTurn, + chargeWorkflowSends, + DEFAULT_WORKFLOW_BUDGET_POLICY, + resetWorkflowBudgetsForTest, + settleWorkflowSpend, + workflowBudgetSnapshot, + workflowSendCeilingReached, + type WorkflowBudgetPolicy, +} from "../../src/lib/workflow-budget"; + +const memoryJournal = (): SpendJournal & { lines: string[] } => { + const lines: string[] = []; + return { lines, read: () => [...lines], append: (line) => { lines.push(line); } }; +}; + +const spendPolicy = (maxTokens: number | undefined): SpendReservationPolicy => ({ + root: { maxTokens }, + identity: { maxTokens }, + pool: { maxTokens }, + retentionMs: 60_000, +}); + +const smallPolicy: WorkflowBudgetPolicy = { + maxConcurrentChildren: 2, + maxPhysicalSends: 3, + maxDistinctChildren: 2, + interactiveReserve: 1, + maxTrackedRoots: 2, +}; + +beforeEach(() => { + resetWorkflowBudgetsForTest(); +}); + +describe("workflow count caps", () => { + test("the physical-send ceiling refuses before dispatch", () => { + admitWorkflowTurn("r1", "interactive", smallPolicy); + chargeWorkflowSends("r1", 3); + expect(workflowSendCeilingReached("r1", smallPolicy)).toBe(true); + const decision = admitWorkflowTurn("r1", "interactive", smallPolicy); + expect(decision?.admitted).toBe(false); + if (decision && !decision.admitted) expect(decision.reason).toBe("workflow-sends-exhausted"); + }); + + test("a worker lane may not take the interactive reserve", () => { + const workerCeiling = smallPolicy.maxConcurrentChildren - smallPolicy.interactiveReserve; + for (let i = 0; i < workerCeiling; i += 1) { + expect(admitWorkflowTurn("r1", "worker", smallPolicy, `c${i}`)?.admitted).toBe(true); + } + const denied = admitWorkflowTurn("r1", "worker", smallPolicy, "c-extra"); + expect(denied?.admitted).toBe(false); + if (denied && !denied.admitted) expect(denied.reason).toBe("workflow-concurrency-exhausted"); + // The interactive turn that owns the fan-out still gets in. + expect(admitWorkflowTurn("r1", "interactive", smallPolicy)?.admitted).toBe(true); + }); + + test("distinct children are capped", () => { + // Concurrency is deliberately not the binding constraint here. + const policy: WorkflowBudgetPolicy = { + maxConcurrentChildren: 10, + maxPhysicalSends: 100, + maxDistinctChildren: 2, + interactiveReserve: 0, + maxTrackedRoots: 10, + }; + admitWorkflowTurn("r1", "worker", policy, "c1"); + admitWorkflowTurn("r1", "worker", policy, "c2"); + const denied = admitWorkflowTurn("r1", "worker", policy, "c3"); + expect(denied?.admitted).toBe(false); + if (denied && !denied.admitted) expect(denied.reason).toBe("workflow-children-exhausted"); + }); + + test("a full root table refuses a new root instead of evicting an exhausted one", () => { + // Fill one root to its send ceiling and let it go idle, then take the only other slot + // with an active root. maxTrackedRoots is 2, so the table is now full and neither entry + // may be forgotten. + const filled = admitWorkflowTurn("full", "interactive", smallPolicy); + chargeWorkflowSends("full", 3); + if (filled?.admitted) filled.lease.release(); + const busy = admitWorkflowTurn("n1", "interactive", smallPolicy); + expect(busy?.admitted).toBe(true); + + // Inserting a third root anyway is what made maxTrackedRoots a suggestion: the bound has + // to refuse, because the only other way to honour it is to reset a ceiling that fired. + const refused = admitWorkflowTurn("n2", "interactive", smallPolicy); + expect(refused?.admitted).toBe(false); + if (refused && !refused.admitted) expect(refused.reason).toBe("workflow-tracking-exhausted"); + expect(workflowBudgetSnapshot("n2")).toBeUndefined(); + + // The exhausted root survived, so recreating it does not reset its allowance. + const decision = admitWorkflowTurn("full", "interactive", smallPolicy); + expect(decision?.admitted).toBe(false); + if (decision && !decision.admitted) expect(decision.reason).toBe("workflow-sends-exhausted"); + + // Once the active root goes idle it becomes a safe candidate and the next root fits. + if (busy?.admitted) busy.lease.release(); + expect(admitWorkflowTurn("n2", "interactive", smallPolicy)?.admitted).toBe(true); + expect(workflowBudgetSnapshot("n1")).toBeUndefined(); + }); +}); + +describe("workflow spend reservation", () => { + test("the token cap intersects the count caps", () => { + const ledger = createSpendReservationLedger({ journal: memoryJournal(), policy: spendPolicy(100), now: () => 1_000 }); + const spend = (sendId: string) => ({ + sendId, inputTokens: 60, outputCeilingTokens: 40, + }); + expect(admitWorkflowTurn("r1", "interactive", DEFAULT_WORKFLOW_BUDGET_POLICY, + undefined, 1_000, spend("s1"), ledger)?.admitted).toBe(true); + const denied = admitWorkflowTurn("r1", "interactive", DEFAULT_WORKFLOW_BUDGET_POLICY, + undefined, 1_000, spend("s2"), ledger); + expect(denied?.admitted).toBe(false); + if (denied && !denied.admitted) { + expect(denied.reason).toBe("workflow-spend-exhausted"); + expect(denied.spendScope).toBe("root"); + } + }); + + test("identity and pool scopes hold spend across fresh root ids", () => { + const ledger = createSpendReservationLedger({ journal: memoryJournal(), policy: spendPolicy(100), now: () => 1_000 }); + const spend = (sendId: string) => ({ + sendId, identityId: "user-1", poolId: "pool-1", inputTokens: 60, outputCeilingTokens: 40, + }); + expect(admitWorkflowTurn("root-a", "interactive", DEFAULT_WORKFLOW_BUDGET_POLICY, + undefined, 1_000, spend("s1"), ledger)?.admitted).toBe(true); + const denied = admitWorkflowTurn("root-b", "interactive", DEFAULT_WORKFLOW_BUDGET_POLICY, + undefined, 1_000, spend("s2"), ledger); + expect(denied?.admitted).toBe(false); + if (denied && !denied.admitted) expect(denied.spendScope).toBe("identity"); + }); + + test("settlement is idempotent and a dispatched release without it becomes unresolved spend", () => { + const ledger = createSpendReservationLedger({ journal: memoryJournal(), policy: spendPolicy(1_000), now: () => 1_000 }); + const admitted = admitWorkflowTurn("r1", "interactive", DEFAULT_WORKFLOW_BUDGET_POLICY, + undefined, 1_000, { sendId: "s1", inputTokens: 100, outputCeilingTokens: 50 }, ledger); + expect(admitted?.admitted).toBe(true); + if (admitted?.admitted) admitted.lease.markDispatched(); + expect(settleWorkflowSpend("s1", { inputTokens: 90, outputTokens: 10 }, ledger)).toBe(true); + // Double settlement books nothing. + expect(settleWorkflowSpend("s1", { inputTokens: 90, outputTokens: 10 }, ledger)).toBe(false); + if (admitted?.admitted) admitted.lease.release(); + const settled = ledger.snapshot("root", "r1"); + expect(settled?.settled).toBe(100); + expect(settled?.unresolved).toBe(0); + + // A DISPATCHED turn released without settlement keeps its cost as unresolved spend: the + // send may have been billed even though its usage frame never arrived. + const lost = admitWorkflowTurn("r1", "interactive", DEFAULT_WORKFLOW_BUDGET_POLICY, + undefined, 2_000, { sendId: "s2", inputTokens: 30, outputCeilingTokens: 20 }, ledger); + if (lost?.admitted) { + lost.lease.markDispatched(); + lost.lease.release(); + } + const after = ledger.snapshot("root", "r1"); + expect(after?.unresolved).toBe(50); + // And a late settle for the lost send is correctly refused. + expect(settleWorkflowSpend("s2", { inputTokens: 30, outputTokens: 20 }, ledger)).toBe(false); + }); + + test("a turn that never reached upstream books no spend at all", () => { + const ledger = createSpendReservationLedger({ journal: memoryJournal(), policy: spendPolicy(1_000), now: () => 1_000 }); + const admitted = admitWorkflowTurn("r1", "interactive", DEFAULT_WORKFLOW_BUDGET_POLICY, + undefined, 1_000, { sendId: "never-sent", inputTokens: 100, outputCeilingTokens: 50 }, ledger); + expect(admitted?.admitted).toBe(true); + // Admission is not dispatch. A local validation or routing failure between the two used + // to be booked as unresolved spend, which invents debt the account never incurred. + if (admitted?.admitted) admitted.lease.release(); + const snapshot = ledger.snapshot("root", "r1"); + expect(snapshot?.reserved).toBe(0); + expect(snapshot?.unresolved).toBe(0); + expect(snapshot?.settled).toBe(0); + + // The send id stays known, so replaying it buys no second dispatch. + const replay = admitWorkflowTurn("r1", "interactive", DEFAULT_WORKFLOW_BUDGET_POLICY, + undefined, 1_000, { sendId: "never-sent", inputTokens: 100, outputCeilingTokens: 50 }, ledger); + expect(replay?.admitted).toBe(false); + if (replay && !replay.admitted) expect(replay.reason).toBe("workflow-send-replayed"); + }); + + test("a spend-exhausted idle root survives eviction pressure", () => { + const ledger = createSpendReservationLedger({ journal: memoryJournal(), policy: spendPolicy(100), now: () => 1_000 }); + const exhausted = admitWorkflowTurn("full", "interactive", smallPolicy, + undefined, 1_000, { sendId: "s1", inputTokens: 60, outputCeilingTokens: 40 }, ledger); + // The send is dispatched and settled, then the lease is released: the root is idle and + // its spend is exhausted. + expect(exhausted?.admitted).toBe(true); + if (exhausted?.admitted) { + exhausted.lease.markDispatched(); + expect(settleWorkflowSpend("s1", { inputTokens: 60, outputTokens: 40 }, ledger)).toBe(true); + exhausted.lease.release(); + } + // An idle, unspent root takes the other slot, then a third root arrives under + // maxTrackedRoots = 2. The evictable one is the unspent root, never the exhausted one. + const spare = admitWorkflowTurn("n1", "interactive", smallPolicy, undefined, 2_000, undefined, ledger); + if (spare?.admitted) spare.lease.release(); + expect(admitWorkflowTurn("n2", "interactive", smallPolicy, undefined, 3_000, undefined, ledger)?.admitted).toBe(true); + expect(workflowBudgetSnapshot("n1")).toBeUndefined(); + expect(workflowBudgetSnapshot("full")).toBeDefined(); + const denied = admitWorkflowTurn("full", "interactive", smallPolicy, + undefined, 4_000, { sendId: "s2", inputTokens: 1, outputCeilingTokens: 0 }, ledger); + expect(denied?.admitted).toBe(false); + if (denied && !denied.admitted) expect(denied.reason).toBe("workflow-spend-exhausted"); + }); +}); diff --git a/tests/routing/probe-lease.test.ts b/tests/routing/probe-lease.test.ts new file mode 100644 index 0000000000..e3cf4ece83 --- /dev/null +++ b/tests/routing/probe-lease.test.ts @@ -0,0 +1,281 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { + canAcquireTransientProbe, + clearTransientProbeLeasesForTests, + configureSharedPoolBackpressure, + createPoolBackpressureLimiter, + invalidateTransientProbe, + releaseTransientProbe, + resetSharedPoolBackpressureForTests, + resolveHeldAccountDispatch, + settleTransientProbe, + sharedPoolBackpressure, + transientProbeDiagnostics, + transientProbeStateCount, + tryAcquireTransientProbe, + MAX_TRANSIENT_PROBE_STATES, + TRANSIENT_PROBE_INTERVAL_MS, +} from "../../src/routing/probe-lease"; + +afterEach(() => { + clearTransientProbeLeasesForTests(); + resetSharedPoolBackpressureForTests(); +}); + +describe("transient probe lease", () => { + test("a held account admits exactly one in-flight probe", () => { + const now = 1_000_000; + const first = tryAcquireTransientProbe("acct-a", now); + expect(first).not.toBeNull(); + // Everyone else is refused while the holder is out. + expect(tryAcquireTransientProbe("acct-a", now)).toBeNull(); + expect(canAcquireTransientProbe("acct-a", now)).toBe(false); + // A different account is a different lease domain. + expect(tryAcquireTransientProbe("acct-b", now)).not.toBeNull(); + }); + + test("a settled lease frees the account after the pacing interval", () => { + const now = 1_000_000; + const lease = tryAcquireTransientProbe("acct-a", now)!; + expect(settleTransientProbe(lease, "failed", now + 5)).toBe("applied"); + // Settling is not a license to probe again immediately -- the interval paces retries. + expect(tryAcquireTransientProbe("acct-a", now + 10)).toBeNull(); + expect(tryAcquireTransientProbe("acct-a", now + TRANSIENT_PROBE_INTERVAL_MS)).not.toBeNull(); + }); + + test("a late result from a replaced lease is stale and mutates nothing", () => { + const now = 1_000_000; + const first = tryAcquireTransientProbe("acct-a", now, { leaseMs: 100, minIntervalMs: 0 })!; + // The first lease lapses and a second probe is issued under a new epoch. + const second = tryAcquireTransientProbe("acct-a", now + 200, { leaseMs: 100, minIntervalMs: 0 })!; + expect(second.generation).toBe(first.generation + 1); + // The late answer must not overwrite the newer lease or record an outcome. + expect(settleTransientProbe(first, "recovered", now + 250)).toBe("stale"); + const diag = transientProbeDiagnostics("acct-a", now + 250); + expect(diag.leaseId).toBe(second.leaseId); + expect(diag.lastOutcome).toBeUndefined(); + // The live holder still settles normally. + expect(settleTransientProbe(second, "recovered", now + 260)).toBe("applied"); + expect(transientProbeDiagnostics("acct-a", now + 260).lastOutcome).toBe("recovered"); + }); + + test("a result after the lease deadline is expired, not applied", () => { + const now = 1_000_000; + const lease = tryAcquireTransientProbe("acct-a", now, { leaseMs: 100 })!; + expect(settleTransientProbe(lease, "recovered", now + 101)).toBe("expired"); + }); + + test("the deadline instant itself is expired for the settle and the grant alike (#4546)", () => { + const now = 1_000_000; + const lease = tryAcquireTransientProbe("acct-a", now, { leaseMs: 100 })!; + // The lease is already gone at its deadline as far as the grant path is concerned... + expect(transientProbeDiagnostics("acct-a", now + 100).held).toBe(false); + // ...so a settle at the same instant must not apply the outcome. Disagreeing about one + // millisecond is how a probe result gets written after the lease was handed to someone + // else. + expect(settleTransientProbe(lease, "recovered", now + 100)).toBe("expired"); + expect(transientProbeDiagnostics("acct-a", now + 100).lastOutcome).toBeUndefined(); + // One millisecond earlier the holder is still live and the outcome applies. + const inside = tryAcquireTransientProbe("acct-b", now, { leaseMs: 100 })!; + expect(settleTransientProbe(inside, "recovered", now + 99)).toBe("applied"); + }); + + test("invalidation fences the epoch so an outstanding probe cannot overwrite newer state", () => { + const now = 1_000_000; + const lease = tryAcquireTransientProbe("acct-a", now)!; + // A newer failure (or a moved binding) lands through the ordinary path. + invalidateTransientProbe("acct-a"); + expect(settleTransientProbe(lease, "recovered", now + 1)).toBe("stale"); + expect(transientProbeDiagnostics("acct-a", now + 1).held).toBe(false); + }); + + test("release hands back a probe that never reached upstream", () => { + const now = 1_000_000; + const lease = tryAcquireTransientProbe("acct-a", now, { minIntervalMs: 0 })!; + releaseTransientProbe(lease); + expect(canAcquireTransientProbe("acct-a", now, { minIntervalMs: 0 })).toBe(true); + // Releasing someone else's lease is a no-op. + releaseTransientProbe({ ...lease, leaseId: "forged" }); + }); +}); + +describe("probe state retention", () => { + test("a retired account is forgotten only once it can no longer pace a probe (#4546)", () => { + const now = 1_000_000; + for (let i = 0; i < 200; i++) { + const lease = tryAcquireTransientProbe(`gone-${i}`, now, { leaseMs: 100 })!; + settleTransientProbe(lease, "failed", now + 1); + } + expect(transientProbeStateCount()).toBe(200); + + // Still inside the pacing interval: dropping these now would let the very next request + // for any of them probe early, which is the storm the interval exists to bound. + tryAcquireTransientProbe("still-paced", now + TRANSIENT_PROBE_INTERVAL_MS - 1); + expect(transientProbeStateCount()).toBe(201); + // The sweep that ran on that grant kept every entry that can still refuse a probe. + expect(canAcquireTransientProbe("gone-0", now + TRANSIENT_PROBE_INTERVAL_MS - 1)).toBe(false); + + // Past the pacing interval and the retention grace the entries cannot change an answer, + // so they are dropped instead of being remembered for the life of the process. + const retired = now + TRANSIENT_PROBE_INTERVAL_MS + 60_000 + 1; + tryAcquireTransientProbe("fresh", retired); + // Two left: the account just probed, and `still-paced`, whose lease was never settled -- + // the grace keeps that one long enough for a late settle to still be answered "expired" + // rather than silently reclassified. + expect(transientProbeStateCount()).toBe(2); + // A dropped entry is indistinguishable from one that was never probed -- which is exactly + // why it was safe to drop: by now it would admit a probe either way. + expect(canAcquireTransientProbe("gone-0", retired)).toBe(true); + }); + + test("remembered accounts stay under the ceiling when churn outruns retention (#4546)", () => { + const now = 1_000_000; + // Every probe settles at once, so nothing holds a live lease: the shape a churning + // configuration produces, and the one that used to grow one entry per account id forever. + const churn = MAX_TRANSIENT_PROBE_STATES * 2; + for (let i = 0; i < churn; i++) { + const lease = tryAcquireTransientProbe(`churn-${i}`, now + i, { leaseMs: 10 })!; + settleTransientProbe(lease, "failed", now + i + 1); + } + expect(transientProbeStateCount()).toBeLessThanOrEqual(MAX_TRANSIENT_PROBE_STATES); + // The ceiling is enforced from the oldest end: the newest accounts keep their pacing. + expect(canAcquireTransientProbe(`churn-${churn - 1}`, now + churn)).toBe(false); + }); + + test("an account holding a live lease survives the ceiling (#4546)", () => { + const now = 1_000_000; + const held = tryAcquireTransientProbe("held-through-churn", now, { leaseMs: 10_000_000 })!; + for (let i = 0; i < MAX_TRANSIENT_PROBE_STATES * 2; i++) { + const lease = tryAcquireTransientProbe(`churn-${i}`, now + i, { leaseMs: 10 })!; + settleTransientProbe(lease, "failed", now + i + 1); + } + // Evicting a live lease would hand a second concurrent probe to the same held account. + expect(transientProbeDiagnostics("held-through-churn", now + 1).leaseId).toBe(held.leaseId); + expect(canAcquireTransientProbe("held-through-churn", now + 1)).toBe(false); + }); +}); + +describe("held account dispatch", () => { + test("one caller probes while the rest keep the remembered detour", () => { + const now = 1_000_000; + const limiter = createPoolBackpressureLimiter(); + const first = resolveHeldAccountDispatch({ + boundAccountId: "acct-a", + detourAccountId: "acct-b", + now, + backpressure: limiter, + }); + expect(first.kind).toBe("probe"); + const second = resolveHeldAccountDispatch({ + boundAccountId: "acct-a", + detourAccountId: "acct-b", + now, + backpressure: limiter, + }); + // The detour is not forgotten during probing: a failed trial must not cost + // the caller its working route. + expect(second).toEqual({ kind: "detour", accountId: "acct-b" }); + }); + + test("every candidate held yields a withheld outcome, never a send", () => { + const now = 1_000_000; + const limiter = createPoolBackpressureLimiter(); + // Spend the single probe so this caller has no trial available. + resolveHeldAccountDispatch({ boundAccountId: "acct-a", now, backpressure: limiter }); + const outcome = resolveHeldAccountDispatch({ boundAccountId: "acct-a", now, backpressure: limiter }); + expect(outcome.kind).toBe("withheld"); + if (outcome.kind === "withheld") { + expect(outcome.boundAccountId).toBe("acct-a"); + expect(outcome.retryAt).toBeGreaterThan(now); + } + }); + + test("a refused probe falls back to the detour, then to withheld", () => { + const now = 1_000_000; + // Zero-allowance limiter: recovery budget is spent, so no probe may go out. + const limiter = createPoolBackpressureLimiter({ + windowMs: 10_000, + maxRetryRatio: 0, + minRecoveryAllowance: 0, + }); + const withDetour = resolveHeldAccountDispatch({ + boundAccountId: "acct-a", + detourAccountId: "acct-b", + now, + backpressure: limiter, + }); + expect(withDetour).toEqual({ kind: "detour", accountId: "acct-b" }); + const noDetour = resolveHeldAccountDispatch({ + boundAccountId: "acct-a", + now, + backpressure: limiter, + }); + expect(noDetour.kind).toBe("withheld"); + }); +}); + +describe("pool-wide backpressure", () => { + test("the initial send of a new request is never refused", () => { + const limiter = createPoolBackpressureLimiter({ + windowMs: 10_000, + maxRetryRatio: 0, + minRecoveryAllowance: 0, + }); + // Even with a zero recovery budget, initials are recorded, not gated. + for (let i = 0; i < 100; i++) limiter.recordInitialSend(i); + expect(limiter.state(100).initialSends).toBe(100); + }); + + test("recovery dispatches are capped by the ratio of observed initials", () => { + const now = 1_000_000; + const limiter = createPoolBackpressureLimiter({ + windowMs: 10_000, + maxRetryRatio: 0.2, + minRecoveryAllowance: 0, + }); + for (let i = 0; i < 10; i++) limiter.recordInitialSend(now); + // 20% of 10 initials admits exactly 2 recovery dispatches, shared by retries and probes. + expect(limiter.tryPermitRetryDispatch(now)).toBe(true); + expect(limiter.tryPermitProbeDispatch(now)).toBe(true); + expect(limiter.tryPermitRetryDispatch(now)).toBe(false); + const state = limiter.state(now); + expect(state.recoveryDispatches).toBe(2); + expect(state.allowance).toBe(2); + expect(state.refusedTotal).toBe(1); + }); + + test("the floor keeps a quiet pool recoverable", () => { + const now = 1_000_000; + const limiter = createPoolBackpressureLimiter(); + // No initials at all: the minimum allowance still admits bounded recovery. + expect(limiter.tryPermitProbeDispatch(now)).toBe(true); + expect(limiter.tryPermitRetryDispatch(now)).toBe(true); + expect(limiter.tryPermitRetryDispatch(now)).toBe(true); + expect(limiter.tryPermitRetryDispatch(now)).toBe(false); + }); + + test("the window slides: old sends stop funding new retries", () => { + const limiter = createPoolBackpressureLimiter({ + windowMs: 10_000, + maxRetryRatio: 0.5, + minRecoveryAllowance: 0, + }); + const t0 = 1_000_000; + for (let i = 0; i < 10; i++) limiter.recordInitialSend(t0); + expect(limiter.tryPermitRetryDispatch(t0)).toBe(true); + // A window later the initials have rotated out; the burst no longer funds retries. + const t1 = t0 + 11_000; + expect(limiter.state(t1).initialSends).toBe(0); + expect(limiter.tryPermitRetryDispatch(t1)).toBe(false); + }); + + test("the shared limiter is configurable and reports its state", () => { + configureSharedPoolBackpressure({ windowMs: 5_000, maxRetryRatio: 1, minRecoveryAllowance: 0 }); + const limiter = sharedPoolBackpressure(); + limiter.recordInitialSend(1_000_000); + expect(limiter.tryPermitRetryDispatch(1_000_000)).toBe(true); + const state = limiter.state(1_000_000); + expect(state.windowMs).toBe(5_000); + expect(state.ratioLimit).toBe(1); + }); +}); diff --git a/tests/routing/routing-identity-domains.test.ts b/tests/routing/routing-identity-domains.test.ts new file mode 100644 index 0000000000..fa30a83db4 --- /dev/null +++ b/tests/routing/routing-identity-domains.test.ts @@ -0,0 +1,277 @@ +import { describe, expect, test } from "bun:test"; + +import { + assessQuotaRotation, + canPortConversationState, + classifyCredential, + countQuotaCapacity, + credentialGroupIssues, + CREDENTIAL_GROUP_MEMBER_PATTERN, + relateCacheDomain, + relateQuotaDomain, + type CredentialIdentity, + type DeclaredCredentialGroup, +} from "../../src/routing/identity-domains"; + +function identity( + credentialId: string, + ref: Partial[0]> = {}, + groups: readonly DeclaredCredentialGroup[] = [], +): CredentialIdentity { + return classifyCredential({ credentialId, ...ref }, groups); +} + +const OPENAI_ORG_PROJECT = { provider: "openai", organizationId: "org-1", projectId: "proj-1" }; + +describe("credential identity domains", () => { + test("authIdentity is always the credential itself, never grouped", () => { + const a = identity("key-a", OPENAI_ORG_PROJECT); + const b = identity("key-b", OPENAI_ORG_PROJECT); + expect(a.authIdentity).toBe("key-a"); + expect(b.authIdentity).toBe("key-b"); + expect(a.authIdentity).not.toBe(b.authIdentity); + }); + + test("operator-declared groups win over the provider table for quota", () => { + const groups: DeclaredCredentialGroup[] = [ + { id: "team", credentials: ["openai:key-a", "openai:key-b"], note: "same billed org" }, + ]; + const a = identity("key-a", { provider: "openai", organizationId: "org-1", projectId: "p-1" }, groups); + const b = identity("key-b", { provider: "openai", organizationId: "org-9", projectId: "p-9" }, groups); + expect(a.quotaDomain.provenance).toBe("operator-declared"); + expect(relateQuotaDomain(a, b)).toBe("shared"); + }); + + test("a declared quota group says nothing about cache compatibility", () => { + const groups: DeclaredCredentialGroup[] = [ + { id: "team", credentials: ["openai:key-a", "openai:key-b"] }, + ]; + const a = identity("key-a", { provider: "openai" }, groups); + const b = identity("key-b", { provider: "openai" }, groups); + expect(relateQuotaDomain(a, b)).toBe("shared"); + expect(relateCacheDomain(a, b)).toBe("unknown"); + }); + + test("OpenAI quota domain is per organization and project", () => { + const a = identity("key-a", OPENAI_ORG_PROJECT); + const b = identity("key-b", OPENAI_ORG_PROJECT); + const otherProject = identity("key-c", { ...OPENAI_ORG_PROJECT, projectId: "proj-2" }); + expect(a.quotaDomain.provenance).toBe("provider-documented"); + expect(relateQuotaDomain(a, b)).toBe("shared"); + expect(relateQuotaDomain(a, otherProject)).toBe("distinct"); + }); + + test("a documented rule missing its evidence yields unknown, not a guess", () => { + const orgOnly = identity("key-a", { provider: "openai", organizationId: "org-1" }); + const same = identity("key-b", { provider: "openai", organizationId: "org-1" }); + expect(orgOnly.quotaDomain.provenance).toBe("unknown"); + expect(relateQuotaDomain(orgOnly, same)).toBe("unknown"); + }); + + test("OpenAI proves cache SEPARATION without proving cache sharing", () => { + const sameOrgRegion = { provider: "openai", organizationId: "org-1", region: "us" }; + const a = identity("key-a", sameOrgRegion); + const b = identity("key-b", sameOrgRegion); + const otherRegion = identity("key-c", { ...sameOrgRegion, region: "eu" }); + const otherOrg = identity("key-d", { ...sameOrgRegion, organizationId: "org-2" }); + // A different organization or region is documented as a different cache. + expect(relateCacheDomain(a, otherRegion)).toBe("distinct"); + expect(relateCacheDomain(a, otherOrg)).toBe("distinct"); + // The same organization and region is NOT documented as one cache: changing keys + // inside an organization is explicitly not guaranteed to hit, so the equal key is + // separation evidence only and the relation stays unknown. + expect(a.cacheDomain.key).toBe(b.cacheDomain.key); + expect(a.cacheDomain.provenance).toBe("provider-documented"); + expect(a.cacheDomain.evidence).toBe("separates"); + expect(relateCacheDomain(a, b)).toBe("unknown"); + // The quota rule for the same provider does promise sharing, and is unaffected. + const quotaA = identity("key-a", { ...sameOrgRegion, projectId: "p-1" }); + const quotaB = identity("key-b", { ...sameOrgRegion, projectId: "p-1" }); + expect(relateQuotaDomain(quotaA, quotaB)).toBe("shared"); + }); + + test("unknown is never read as shared and never as distinct", () => { + const a = identity("key-a", { provider: "obscure" }); + const b = identity("key-b", { provider: "obscure" }); + expect(relateQuotaDomain(a, b)).toBe("unknown"); + expect(relateCacheDomain(a, b)).toBe("unknown"); + expect(a.quotaDomain.key).not.toBe(b.quotaDomain.key); + }); + + test("Anthropic isolates prompt cache per workspace", () => { + const a = identity("k1", { provider: "anthropic", workspaceId: "ws-1" }); + const b = identity("k2", { provider: "anthropic", workspaceId: "ws-1" }); + const other = identity("k3", { provider: "anthropic", workspaceId: "ws-2" }); + expect(relateCacheDomain(a, b)).toBe("shared"); + expect(relateCacheDomain(a, other)).toBe("distinct"); + // Anthropic quota sharing is not one of the documented cases. + expect(relateQuotaDomain(a, b)).toBe("unknown"); + }); + + test("Azure domains are per deployment", () => { + const a = identity("d1", { provider: "azure", deploymentId: "dep-1" }); + const b = identity("d2", { provider: "azure", deploymentId: "dep-1" }); + const other = identity("d3", { provider: "azure", deploymentId: "dep-2" }); + expect(relateQuotaDomain(a, b)).toBe("shared"); + expect(relateCacheDomain(a, b)).toBe("shared"); + expect(relateQuotaDomain(a, other)).toBe("distinct"); + }); +}); + +describe("declared groups are unambiguous or they do not apply", () => { + test("a bare credential id never matches: membership is provider-scoped", () => { + const groups: DeclaredCredentialGroup[] = [{ id: "team", credentials: ["key-a"] }]; + const a = identity("key-a", { provider: "openai", organizationId: "org-1", projectId: "p-1" }, groups); + expect(a.quotaDomain.provenance).toBe("provider-documented"); + expect(credentialGroupIssues(groups)).toHaveLength(1); + expect(credentialGroupIssues(groups)[0]).toContain("provider-qualified"); + expect(CREDENTIAL_GROUP_MEMBER_PATTERN.test("key-a")).toBe(false); + expect(CREDENTIAL_GROUP_MEMBER_PATTERN.test("openai:key-a")).toBe(true); + }); + + test("the provider segment normalizes through the same aliases as a ref", () => { + const groups: DeclaredCredentialGroup[] = [{ id: "team", credentials: ["chatgpt:key-a"] }]; + const a = identity("key-a", { provider: "codex" }, groups); + expect(a.quotaDomain.provenance).toBe("operator-declared"); + expect(credentialGroupIssues(groups)).toEqual([]); + }); + + test("a credential claimed by two groups is reported, not resolved by order", () => { + const groups: DeclaredCredentialGroup[] = [ + { id: "left", credentials: ["openai:key-a"] }, + { id: "right", credentials: ["openai:key-a"] }, + ]; + const a = identity("key-a", { provider: "openai", organizationId: "org-1", projectId: "p-1" }, groups); + expect(a.declaredGroupConflict).toEqual(["left", "right"]); + // Falls back to the documented answer rather than joining whichever group came first. + expect(a.quotaDomain.provenance).toBe("provider-documented"); + expect(credentialGroupIssues(groups).join("; ")).toContain("more than one group"); + }); + + test("a duplicated group id is a conflict, because both groups key the same domain", () => { + const groups: DeclaredCredentialGroup[] = [ + { id: "team", credentials: ["openai:key-a"] }, + { id: "team", credentials: ["openai:key-b"] }, + ]; + const a = identity("key-a", { provider: "openai" }, groups); + const b = identity("key-b", { provider: "openai" }, groups); + expect(a.declaredGroupConflict).toEqual(["team"]); + expect(b.declaredGroupConflict).toEqual(["team"]); + expect(relateQuotaDomain(a, b)).toBe("unknown"); + expect(credentialGroupIssues(groups).join("; ")).toContain("duplicate group id"); + }); + + test("an empty member list and a repeated member are reported", () => { + expect(credentialGroupIssues([{ id: "team", credentials: [] }]).join("; ")) + .toContain("lists no credentials"); + expect(credentialGroupIssues([{ id: "team", credentials: ["openai:key-a", "openai:key-a"] }]).join("; ")) + .toContain("listed twice"); + }); + + test("an unambiguous declaration still applies", () => { + const groups: DeclaredCredentialGroup[] = [ + { id: "left", credentials: ["openai:key-a"] }, + { id: "right", credentials: ["azure:key-a"] }, + ]; + const openai = identity("key-a", { provider: "openai" }, groups); + const azure = identity("key-a", { provider: "azure", deploymentId: "dep-1" }, groups); + expect(openai.declaredGroupConflict).toBeUndefined(); + expect(openai.quotaDomain.key).toBe("declared:left"); + expect(azure.quotaDomain.key).toBe("declared:right"); + expect(relateQuotaDomain(openai, azure)).toBe("distinct"); + expect(credentialGroupIssues(groups)).toEqual([]); + }); +}); + +describe("quota refusal rotation", () => { + test("a refusal inside a known shared domain must not rotate within it", () => { + const a = identity("key-a", OPENAI_ORG_PROJECT); + const b = identity("key-b", OPENAI_ORG_PROJECT); + expect(assessQuotaRotation(a, b)).toBe("same-domain"); + }); + + test("a refusal may rotate to a credential in a distinct domain", () => { + const a = identity("key-a", OPENAI_ORG_PROJECT); + const b = identity("key-b", { provider: "azure", deploymentId: "dep-1" }); + expect(assessQuotaRotation(a, b)).toBe("distinct-domain"); + }); + + test("unknown domains hand the decision back to the caller", () => { + const a = identity("key-a", { provider: "obscure" }); + const b = identity("key-b", OPENAI_ORG_PROJECT); + expect(assessQuotaRotation(a, b)).toBe("unknown"); + expect(assessQuotaRotation(b, a)).toBe("unknown"); + }); +}); + +describe("quota capacity accounting", () => { + test("two credentials in one quota domain count once", () => { + const a = identity("key-a", OPENAI_ORG_PROJECT); + const b = identity("key-b", OPENAI_ORG_PROJECT); + const c = identity("key-c", { provider: "azure", deploymentId: "dep-1" }); + expect(countQuotaCapacity([a, b, c])).toEqual({ known: 2, unknown: 0 }); + }); + + test("unknown-domain credentials are reported separately, not merged", () => { + const a = identity("key-a", OPENAI_ORG_PROJECT); + const u1 = identity("u1", { provider: "obscure" }); + const u2 = identity("u2", { provider: "obscure" }); + expect(countQuotaCapacity([a, u1, u2])).toEqual({ known: 1, unknown: 2 }); + }); +}); + +describe("conversational-state portability", () => { + test("a state-free request is portable", () => { + expect(canPortConversationState({})).toEqual({ portable: true }); + expect(canPortConversationState({ + previousResponseId: null, + fileIds: [], + encryptedReasoning: undefined, + })).toEqual({ portable: true }); + }); + + test("previous_response_id refuses with a typed reason", () => { + expect(canPortConversationState({ previousResponseId: "resp_1" })).toEqual({ + portable: false, + reason: "previous-response-id", + }); + }); + + test("provider-side conversation id refuses", () => { + expect(canPortConversationState({ providerConversationId: "conv_1" })).toEqual({ + portable: false, + reason: "provider-conversation-id", + }); + }); + + test("uploaded file ids refuse", () => { + expect(canPortConversationState({ fileIds: ["file-1"] })).toEqual({ + portable: false, + reason: "uploaded-file-ids", + }); + }); + + test("encrypted reasoning payloads refuse", () => { + expect(canPortConversationState({ encryptedReasoning: ["blob"] })).toEqual({ + portable: false, + reason: "encrypted-reasoning", + }); + expect(canPortConversationState({ encryptedReasoning: "blob" })).toEqual({ + portable: false, + reason: "encrypted-reasoning", + }); + }); + + test("a shared cache domain is not portability, and portability is not a cache promise", () => { + const a = identity("k1", { provider: "anthropic", workspaceId: "ws-1" }); + const b = identity("k2", { provider: "anthropic", workspaceId: "ws-1" }); + expect(relateCacheDomain(a, b)).toBe("shared"); + // Same cache domain, still not portable once the request carries bound state. + expect(canPortConversationState({ previousResponseId: "resp_1" }).portable).toBe(false); + // Portable state, still no cache promise on an undocumented provider. + const u1 = identity("u1", { provider: "obscure" }); + const u2 = identity("u2", { provider: "obscure" }); + expect(canPortConversationState({}).portable).toBe(true); + expect(relateCacheDomain(u1, u2)).toBe("unknown"); + }); +});