Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Python: [Bug]: Streaming text content lost when usage data is present in the same chunk (affects Gemini) #3434

Description

@toshihiro-otani

[Bug]: Streaming text content lost when usage data is present in the same chunk (affects Gemini and potentially other providers)

Description

When using streaming responses with certain OpenAI-compatible providers (e.g., Gemini), text content is lost because the current implementation performs an early return when chunk.usage is present, skipping the text parsing logic.

Environment

  • agent-framework version: 1.0.0b260123
  • Python version: 3.13
  • Affected provider: Google Gemini (OpenAI-compatible API), potentially others
  • OpenAI client: Works correctly (likely due to different chunk structure)

Root Cause

In agent_framework/openai/_chat_client.py, the _parse_response_update_from_openai method (lines 307-348) performs an early return when chunk.usage is present:

def _parse_response_update_from_openai(self, chunk: ChatCompletionChunk) -> ChatResponseUpdate:
    chunk_metadata = self._get_metadata_from_streaming_chat_response(chunk)
    if chunk.usage:
        return ChatResponseUpdate(  # ← Early return!
            role=Role.ASSISTANT,
            contents=[Content.from_usage(...)],
            ...
        )
    # Text parsing below is never reached when usage is present
    contents: list[Content] = []
    for choice in chunk.choices:
        if text_content := self._parse_text_from_openai(choice):
            contents.append(text_content)
    ...

Problem: Some providers (e.g., Gemini) include both usage data and text content (delta.content) in the same chunk. The early return causes text content to be skipped.

Impact

  • Non-streaming responses: Work correctly
  • Streaming responses: Text content is lost
    • AgentResponseUpdate.text is always empty string
    • AgentResponseUpdate.contents only contains Content(type=usage)
    • Final conversation in workflows contains empty messages
  • Tool calling: Tool calls are also skipped (processed in the same loop)

Reproduction

Test Code

import asyncio
import os
from dotenv import load_dotenv
from agent_framework.openai import OpenAIChatClient

load_dotenv()

async def test_streaming():
    client = OpenAIChatClient(
        model_id="gemini-2.0-flash-lite",
        api_key=os.getenv("GEMINI_API_KEY"),
        base_url="https://generativelanguage.googleapis.com/v1beta/openai/",
    )

    agent = client.as_agent(
        name="test_agent",
        instructions="You are a helpful assistant."
    )

    async for update in agent.run_stream("Count from 1 to 3"):
        print(f"Text: '{update.text}'")  # Always empty!
        print(f"Contents: {update.contents}")  # Only usage

asyncio.run(test_streaming())

Expected Output

Text: '1'
Text: ','
Text: '2, 3'

Actual Output

Text: ''
Contents: [Content(type=usage)]
Text: ''
Contents: [Content(type=usage)]
Text: ''
Contents: [Content(type=usage)]

Raw API Verification

Direct HTTP request to Gemini shows that delta.content IS present:

{
  "choices": [
    {
      "delta": {
        "content": "1",  // ← Text is here!
        "role": "assistant"
      },
      "index": 0
    }
  ],
  "usage": {  // ← Usage is also present
    "completion_tokens": 0,
    "prompt_tokens": 18,
    "total_tokens": 18
  }
}

Proposed Fix

Remove the early return and process usage alongside text/tool calls:

def _parse_response_update_from_openai(self, chunk: ChatCompletionChunk) -> ChatResponseUpdate:
    chunk_metadata = self._get_metadata_from_streaming_chat_response(chunk)
    contents: list[Content] = []
    finish_reason: FinishReason | None = None

    # Process usage alongside text (no early return)
    if chunk.usage:
        contents.append(
            Content.from_usage(
                usage_details=self._parse_usage_from_openai(chunk.usage),
                raw_representation=chunk
            )
        )

    # Process text and tool calls
    for choice in chunk.choices:
        chunk_metadata.update(self._get_metadata_from_chat_choice(choice))
        contents.extend(self._parse_tool_calls_from_openai(choice))
        if choice.finish_reason:
            finish_reason = FinishReason(value=choice.finish_reason)
        if text_content := self._parse_text_from_openai(choice):
            contents.append(text_content)
        if reasoning_details := getattr(choice.delta, "reasoning_details", None):
            contents.append(Content.from_text_reasoning(protected_data=json.dumps(reasoning_details)))

    return ChatResponseUpdate(
        created_at=datetime.fromtimestamp(chunk.created, tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ"),
        contents=contents,
        role=Role.ASSISTANT,
        model_id=chunk.model,
        additional_properties=chunk_metadata,
        finish_reason=finish_reason,
        raw_representation=chunk,
        response_id=chunk.id,
        message_id=chunk.id,
    )

Verification After Fix

Streaming text: Correctly extracted
Tool calls: Correctly extracted
Usage data: Still collected
OpenAI compatibility: Not affected (OpenAI likely sends usage in separate chunks)

Additional Notes

  • This bug affects multi-agent workflows severely, as empty messages are stored in conversation history
  • Tool calling is also affected (tool calls are processed in the same loop)
  • The fix is backward-compatible with OpenAI (which likely doesn't include usage + text in the same chunk)

Would you like me to submit a PR?

I have already tested the fix locally and can provide:

  • ✅ Fixed implementation
  • ✅ Test cases for Gemini streaming
  • ✅ Verification that OpenAI behavior is unchanged

Metadata

Metadata

Labels

agentsUsage: [Issues, PRs], Target: Single agentpythonUsage: [Issues, PRs], Target: Python

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions