Bug Fixes

• Fix a regression that triggered incorrect warnings about usage reporting for streaming inferences with Anthropic models.
• Fix a bug in the TensorZero Python SDK that discarded some request fields in certain multi-turn inferences with tools.

New Features

• Improve error handling across many areas: TensorZero UI, JSON deserialization, AWS providers, streaming inferences, timeouts, etc.
• Support Valkey (Redis) for improving performance of rate limiting checks (recommended at 100+ QPS).
• Support reasoning_effort for Gemini 3 models (mapped to thinkingLevel).
• Improve handling of Anthropic reasoning models in TensorZero JSON functions. Moving forward, json_mode = "strict" will use the beta structured outputs feature; json_mode = "on" still uses the legacy assistant message prefill.
• Improve handling of reasoning content in the OpenRouter and xAI model providers.
• Add extra_headers support for embedding models. (thanks @jonaylor89!)
• Support dynamic credentials for AWS Bedrock and AWS SageMaker model providers.

& multiple under-the-hood and UI improvements (thanks @ndoherty-xyz)!

New Features

• Support appending to arrays with extra_body using the /my_array/- notation.
• Handle cross-model thought signatures in GCP Vertex AI Gemini and Google AI Studio.

& multiple under-the-hood and UI improvements (thanks @ecalifornica!)

Bug Fixes

• Stop buffering raw_usage when streaming with the OpenAI-compatible inference endpoint; instead, emit raw_usage as soon as possible, just like in the native endpoint.
• Stop reporting zero usage in every chunk when streaming a cached inference; instead, report zero usage only in the final chunk, as expected.

New Features

• Support stream_options.include_usage for every model under the Azure provider.

& multiple under-the-hood and UI improvements!

New Features

• Add optional include_raw_usage parameter to inference requests. If enabled, the gateway returns the raw usage objects from model provider responses in addition to the normalized usage response field.
• Add optional --bind-address CLI flag to the gateway.
• Add optional description field to metrics in the configuration.
• Add option to fine-tune Fireworks models without automatic deployment.

& multiple under-the-hood and UI improvements (thanks @ecalifornica @achaljhawar @rguilmont)!

New Features

• Support gateway relay. With gateway relay, an LLM inference request can be routed through multiple independent TensorZero Gateway deployments before reaching a model provider. This enables you to enforce organization-wide controls (e.g. auth, rate limits, credentials) without restricting how teams build their LLM features.
• Add "Try with model" button to the datapoint page in the UI.
• Add tensorzero_inference_latency_overhead_seconds_histogram Prometheus metric for meta-observability.
• Add concurrency parameter to experimental_render_samples (defaults to 100).
• Add otlp_traces_extra_attributes and otlp_traces_extra_resources to the TensorZero Python SDK. (thanks @jinnovation!)

& multiple under-the-hood and UI improvements (thanks @ecalifornica)

New Features

• Expand the dataset builder in the UI to support complex queries (e.g. filter by tags, feedback).
• Export tensorzero_inference_latency_overhead_seconds Prometheus metric for meta-observability.
• Allow users to disable TensorZero API keys using --disable-api-key in the CLI. (thanks @jinnovation!)

& multiple under-the-hood and UI improvements (thanks @ecalifornica)!

Bug Fixes

• Fix a bug where negative tag filters (e.g. user_id != 1) matched inferences and datapoints without that tag.
• Fix a bug where metric filters covering default values (e.g. exact_match = false) matched inferences without that metric.
• Fix a regression affecting the logger in the UI.

New Features

• Improve the performance of the inference and datapoint list pages in the UI.
• Support filtering inferences by whether they have a demonstration.

& multiple under-the-hood and UI improvements (thanks @jinnovation @ecalifornica @simeonlee)!

Bug Fixes

• Fix a performance regression affecting the inference table in the UI.

New Features

• Allow users to customize the log level in the UI (TENSORZERO_UI_LOG_LEVEL).

& multiple under-the-hood and UI improvements

Bug Fixes

• Fixed a regression that broke the dataset builder in the UI.

& multiple under-the-hood and UI improvements

Bug Fixes

• Fix a regression in the "Try with..." modal in the UI that disregarded some parameters (e.g. allowed_tools).
• Fix a regression in allowed_tools when using custom display names for tools.
• Fix an edge case when using both allowed_tools and tool_choice parameters with GCP Vertex AI Gemini.

New Features

• Support free-form search and filtering (e.g. by tags, metrics) the inference and datapoint tables in the UI.
• Support creating datapoints from scratch in the UI.
• Support editing TensorZero API key descriptions in the UI (thanks @nicoestrada!).
• Support editing any kind of datapoint input and output in the UI.
• Support peeking at inferences in the episode detail page in the UI (thanks @BrianLi23!).
• Support cloning datapoints in the UI.
• Optimize the rendering performance of the code editor in the UI.
• Make mime_type optional for base64 file inputs (now inferred from magic bytes when not provided).

& multiple under-the-hood and UI improvements