fix(usage): apply cached-token discounts to all OpenAI-compatible providers#439
Conversation
…viders Cost calculation only applied per-token-type pricing (cached input, reasoning, audio) for six provider types listed in providerMappings (openai, openrouter, anthropic, gemini, groq, xai). Every other OpenAI-compatible provider — xiaomi, deepseek, zai, minimax, bailian, oracle, azure, vllm, ollama, opencode_go — got no mapping, so their cached prompt tokens were billed at the full input rate instead of the much cheaper cached-input rate, over-charging cache-heavy workloads. This is the cause of issue #435: Xiaomi MiMo bills cache-hit input at ~1/50th of cache-miss input, but the gateway charged every input token at the full rate, reporting roughly 4x the real cost. Default any provider not in providerMappings to the OpenAI-compatible token mappings, since they all speak the OpenAI usage schema. Also map DeepSeek's non-standard top-level cache fields: prompt_cache_hit_tokens (cached-input alias; the hit count is already inside prompt_tokens) and prompt_cache_miss_tokens (informational remainder, no separate rate). Note: the discount still requires the model's registry pricing to carry cached_input_per_mtok; models without it (e.g. mimo-v2.5) need a companion ai-model-list data update. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
📝 WalkthroughWalkthroughExtends ChangesProvider Token Cost Mapping Fixes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/usage/cost_test.go`:
- Around line 732-744: The current test in CalculateGranularCost only compares
fallback providers against the openai result, so it does not explicitly verify
the discounted baseline. Update the test to assert fixed expected InputCost,
OutputCost, and TotalCost values for the openai baseline before iterating
through the provider list, using CalculateGranularCost and the existing
pricing/rawData setup to lock in the cached-token discount behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: d64e0f65-bbd2-4c88-b979-683072e14b82
📒 Files selected for processing (2)
internal/usage/cost.gointernal/usage/cost_test.go
| want := CalculateGranularCost(1_000_000, 200_000, rawData, "openai", pricing) | ||
| if want.TotalCost == nil { | ||
| t.Fatal("expected a total cost for the openai baseline") | ||
| } | ||
|
|
||
| for _, provider := range []string{ | ||
| "xiaomi", "deepseek", "zai", "minimax", "bailian", | ||
| "oracle", "azure", "vllm", "ollama", "opencode_go", | ||
| } { | ||
| got := CalculateGranularCost(1_000_000, 200_000, rawData, provider, pricing) | ||
| assertCostNear(t, provider+" InputCost", got.InputCost, *want.InputCost) | ||
| assertCostNear(t, provider+" OutputCost", got.OutputCost, *want.OutputCost) | ||
| assertCostNear(t, provider+" TotalCost", got.TotalCost, *want.TotalCost) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Assert the discounted baseline explicitly.
This only proves fallback providers match "openai", not that cached tokens are actually discounted. If openAICompatibleTokenCostMappings regresses for both paths, this test still passes. Add fixed expected costs for the "openai" baseline before the provider loop.
Suggested change
want := CalculateGranularCost(1_000_000, 200_000, rawData, "openai", pricing)
- if want.TotalCost == nil {
- t.Fatal("expected a total cost for the openai baseline")
- }
+ assertCostNear(t, "openai InputCost", want.InputCost, 0.04674)
+ assertCostNear(t, "openai OutputCost", want.OutputCost, 0.174)
+ assertCostNear(t, "openai TotalCost", want.TotalCost, 0.22074)
for _, provider := range []string{📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| want := CalculateGranularCost(1_000_000, 200_000, rawData, "openai", pricing) | |
| if want.TotalCost == nil { | |
| t.Fatal("expected a total cost for the openai baseline") | |
| } | |
| for _, provider := range []string{ | |
| "xiaomi", "deepseek", "zai", "minimax", "bailian", | |
| "oracle", "azure", "vllm", "ollama", "opencode_go", | |
| } { | |
| got := CalculateGranularCost(1_000_000, 200_000, rawData, provider, pricing) | |
| assertCostNear(t, provider+" InputCost", got.InputCost, *want.InputCost) | |
| assertCostNear(t, provider+" OutputCost", got.OutputCost, *want.OutputCost) | |
| assertCostNear(t, provider+" TotalCost", got.TotalCost, *want.TotalCost) | |
| want := CalculateGranularCost(1_000_000, 200_000, rawData, "openai", pricing) | |
| assertCostNear(t, "openai InputCost", want.InputCost, 0.04674) | |
| assertCostNear(t, "openai OutputCost", want.OutputCost, 0.174) | |
| assertCostNear(t, "openai TotalCost", want.TotalCost, 0.22074) | |
| for _, provider := range []string{ | |
| "xiaomi", "deepseek", "zai", "minimax", "bailian", | |
| "oracle", "azure", "vllm", "ollama", "opencode_go", | |
| } { | |
| got := CalculateGranularCost(1_000_000, 200_000, rawData, provider, pricing) | |
| assertCostNear(t, provider+" InputCost", got.InputCost, *want.InputCost) | |
| assertCostNear(t, provider+" OutputCost", got.OutputCost, *want.OutputCost) | |
| assertCostNear(t, provider+" TotalCost", got.TotalCost, *want.TotalCost) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@internal/usage/cost_test.go` around lines 732 - 744, The current test in
CalculateGranularCost only compares fallback providers against the openai
result, so it does not explicitly verify the discounted baseline. Update the
test to assert fixed expected InputCost, OutputCost, and TotalCost values for
the openai baseline before iterating through the provider list, using
CalculateGranularCost and the existing pricing/rawData setup to lock in the
cached-token discount behavior.
Source: Coding guidelines
Confidence Score: 5/5The changes are narrowly scoped to usage cost mapping behavior and add targeted tests for the corrected provider cases. No correctness issues were identified in the changed usage cost logic, and the added tests cover the intended OpenAI-compatible fallback and DeepSeek top-level cache fields.
 T-Rex LogsWhat T-Rex did
Reviews (1): Last reviewed commit: "fix(usage): apply cached-token discounts..." | Re-trigger Greptile |
Summary
Fixes #435. Cost calculation only applied per-token-type pricing (cached input, reasoning, audio) for the six provider types listed in
providerMappings(openai,openrouter,anthropic,gemini,groq,xai). Every other OpenAI-compatible provider got no mapping at all — so their cached prompt tokens were billed at the full input rate instead of the much cheaper cached-input rate, over-charging cache-heavy workloads.Affected provider types:
xiaomi,deepseek,zai,minimax,bailian,oracle,azure,vllm,ollama,opencode_go.Why this caused the reported ~4×
Xiaomi MiMo prices cache-hit input at ~1/50th of cache-miss input (e.g. MiMo-V2.5: ¥0.02 vs ¥1.0 /Mtok). The gateway already captured the cached count (
prompt_tokens_details.cached_tokens→prompt_cached_tokens) but, becausexiaomiwasn't inproviderMappings, charged every input token at the full rate — and even flagged the cached field as an"unmapped token field"caveat. For a cache-heavy workload that lands around 4× the real bill, matching the issue report (~$9.4 vs ~$2.2).Changes
anthropic,gemini) or that report extra token types (xai) keep an explicit entry.prompt_tokens_details.cached_tokens; it reports top-levelprompt_cache_hit_tokens/prompt_cache_miss_tokenswhereprompt_tokens = hit + miss. Addedprompt_cache_hit_tokensas a cached-input alias (includedInBase) andprompt_cache_miss_tokensas an informational field (no separate rate, no spurious caveat).The dedup-by-pricing-field guard already in place means a provider emitting several cache aliases can't double-count.
Provider field-name coverage (verified against official docs)
prompt_tokens_details.cached_tokenscached_tokens(also nested)prompt_cache_hit_tokens/prompt_cache_miss_tokensNot changed (deliberate)
CachedOutputPerMtokis intentionally not added. Verified against OpenAI / Anthropic / Gemini / DeepSeek / xAI / GLM docs: prompt caching is input-only everywhere; there is no cached-output billing concept. The real cache dimensions are already modeled (CachedInputPerMtok, plus Anthropic'sCacheWritePerMtok).anthropicprovider type (e.g.opencode_gomodels inOPENCODE_GO_MESSAGES_MODELS, MiniMax's/messagessurface) reportscache_read_input_tokens/cache_creation_input_tokens, which the OpenAI default doesn't map. Narrow case; the clean fix is endpoint/shape-aware mapping selection, left out to keep this change focused.Follow-up (data, separate repo)
The discount only applies when the model's registry pricing carries
cached_input_per_mtok. 303/777ai-model-listmodels have it (incl. DeepSeek-V3.x, GLM), butmimo-v2.5does not (onlymimo-v2.5-pro). A companionENTERPILOT/ai-model-listupdate is needed to add native Xiaomi cache-hit pricing somimo-v2.5users are fully corrected.Tests
TestCalculateGranularCost_OpenAICompatibleProvidersDefaultMappings— all 10 providers now match theopenaibaseline cost for a cache-heavy response, no caveat.TestCalculateGranularCost_DeepSeekTopLevelCacheFields— DeepSeek top-level hit/miss fields priced correctly (cache-hit at cached rate, miss informational), no caveat.usage/gateway/budget/adminsuites pass;make test-raceandmake lintgreen via pre-commit.🤖 Generated with Claude Code
Summary by CodeRabbit