Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Fix two edge cases with streaming usage and raw usage #5639

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
GabrielBianconi merged 6 commits into from
Jan 14, 2026
Merged

Fix two edge cases with streaming usage and raw usage #5639

GabrielBianconi merged 6 commits into from
Jan 14, 2026

Conversation

@GabrielBianconi
Copy link
Member

@GabrielBianconi GabrielBianconi commented Jan 13, 2026

Fixes two edge cases:

  • The OpenAI-compatible endpoint buffered raw_usage but it should emit it ASAP (just like the native endpoint)
  • We reported usage of 0 (explicit) rather than omitting/null on every chunk when TensorZero cached the request. The correct behavior is to do this only in the last chunk.

@GabrielBianconi GabrielBianconi added the priority-urgent The TensorZero team will drop everything to fix this issue. label Jan 13, 2026
@GabrielBianconi
Copy link
Member Author

GabrielBianconi commented Jan 13, 2026

/merge-queue

@github-actions
Copy link
Contributor

github-actions bot commented Jan 13, 2026

🚀 Merge queue workflow triggered!

View the run: https://github.com/tensorzero/tensorzero/actions/runs/20975529133

@GabrielBianconi
Copy link
Member Author

GabrielBianconi commented Jan 13, 2026

@cursor review

@cursor
Copy link

cursor bot commented Jan 13, 2026 •
edited
Loading

PR Summary

Aligns streaming semantics across native and OpenAI-compatible APIs and strengthens test isolation.

  • OpenAI-compatible chat_completions: compute include_usage (defaults false; tensorzero::include_raw_usage implies true) and error if include_raw_usage=true with include_usage=false
  • Streaming serialization: pass through usage and tensorzero_raw_usage directly on chunks; remove synthesized trailing "usage-only" chunk; support both Chat and JSON chunks
  • Cached streaming: include usage only on the final chunk (zero values), not intermediate ones; native stream now includes zeroed usage on final chunk only when cached and original had usage
  • Tests: update Go/Node/Python to expect finish_reason/usage on the final chunk; reduce flaky timing; add e2e tests verifying usage-only-in-last-chunk and raw_usage exclusion on cache hits
  • Infra: add helpers to spin up gateways with unique ClickHouse DBs for isolated tests

Written by Cursor Bugbot for commit 2f4c166. This will update automatically on new commits. Configure here.

@tensorzero tensorzero deleted a comment from cursor bot Jan 13, 2026
chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Jan 13, 2026
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0b0f79d36c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

cursor[bot]
cursor bot reviewed Jan 13, 2026
View reviewed changes
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no bugs!

@GabrielBianconi
Copy link
Member Author

GabrielBianconi commented Jan 14, 2026

/merge-queue

@github-actions
Copy link
Contributor

github-actions bot commented Jan 14, 2026

🚀 Merge queue workflow triggered!

View the run: https://github.com/tensorzero/tensorzero/actions/runs/20984496121

@GabrielBianconi
Copy link
Member Author

GabrielBianconi commented Jan 14, 2026

/merge-queue

@github-actions
Copy link
Contributor

github-actions bot commented Jan 14, 2026

🚀 Merge queue workflow triggered!

View the run: https://github.com/tensorzero/tensorzero/actions/runs/20999279209

@GabrielBianconi
Copy link
Member Author

GabrielBianconi commented Jan 14, 2026

/merge-queue

@github-actions
Copy link
Contributor

github-actions bot commented Jan 14, 2026

🚀 Merge queue workflow triggered!

View the run: https://github.com/tensorzero/tensorzero/actions/runs/20999562817

@GabrielBianconi GabrielBianconi added this pull request to the merge queue Jan 14, 2026
Merged via the queue into main with commit f104282 Jan 14, 2026
78 of 103 checks passed
@GabrielBianconi GabrielBianconi deleted the gb/aggregate-usage branch January 14, 2026 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

@cursor cursor[bot] cursor[bot] left review comments

@Aaron1011 Aaron1011 Aaron1011 approved these changes

Assignees

@Aaron1011 Aaron1011

Labels

priority-urgent The TensorZero team will drop everything to fix this issue.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@GabrielBianconi @Aaron1011