Python: Fix file_search citations breaking assistant history roundtrip#5557
Conversation
The Responses API rejects 'input_file' inside an assistant message, but the SDK was emitting it whenever an assistant Message contained a hosted_file content (which is what file_search citations become). Three coordinated fixes: 1. _prepare_content_for_openai now skips hosted_file for the assistant role instead of mapping to input_file (which the API rejects there). 2. The streaming response.output_text.annotation.added handler attaches file_citation, container_file_citation, and file_path as annotations on text content, matching the non-streaming path. Previously streaming produced standalone HostedFileContent items that always tripped (1). 3. output_text serialization preserves Annotation objects on roundtrip via a new _annotations_to_output_text helper instead of hardcoding 'annotations' to []. file_search citations now survive multi-agent forwarding. Closes microsoft#5556.
Python Test Coverage Report •
Python Unit Test Overview
|
||||||||||||||||||||||||||||||
There was a problem hiding this comment.
Pull request overview
Fixes a Responses API incompatibility where file_search citations (and other hosted-file citation artifacts) could be serialized back into assistant history as input_file, causing 400s when multi-agent workflows forward assistant messages.
Changes:
- Preserve assistant
output_text.annotationson serialization by converting frameworkAnnotation(type="citation")back into Responses API annotation dicts. - Prevent
hosted_filecontents in assistant messages from roundtripping asinput_fileby dropping them during outbound preparation. - Make streaming citation events attach as text annotations (via empty text
Content) rather than standaloneHostedFileContent, aligning streaming with non-streaming behavior; update/add regression tests.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| python/packages/openai/agent_framework_openai/_chat_client.py | Adds annotation roundtrip helper, drops assistant hosted_file on outbound, and changes streaming citation handling to produce text annotations. |
| python/packages/openai/tests/openai/test_openai_chat_client.py | Adds/updates tests to cover citation roundtrip behavior and the streaming→history regression. |
There was a problem hiding this comment.
Automated Code Review
Reviewers: 4 | Confidence: 90%
✓ Correctness
This PR fixes a roundtrip bug where streaming file_search citations were stored as HostedFileContent, then serialized as
input_fileitems in assistant history — rejected by the Responses API. The fix correctly converts these to text annotations (matching the non-streaming path) and adds a helper to reconstruct API-format annotations on output. The branching logic in_annotations_to_output_textcorrectly maps all four annotation variants (file_path, file_citation, url_citation, container_file_citation) and the priority ordering of the elif chain prevents misclassification. Droppinghosted_filefor assistant role returns{}which is properly filtered by the existingif prepared_content:guard in_prepare_message_for_openai. Tests are comprehensive and cover the main code paths.
✓ Security Reliability
This diff fixes a real bug where streaming file_search citations were represented as HostedFileContent, which then serialized as
input_filein assistant history and was rejected by the Responses API. The fix converts these to text annotations that roundtrip cleanly. The new_annotations_to_output_texthelper safely reshapes Annotation TypedDicts (all fields optional, accessed via.get()) into API-compatible dicts. The empty-dict guard forhosted_fileon assistant messages is correctly filtered by existing calers (line 1352:if prepared_content:— empty dict is falsy). Thegetattr(content, "annotations", None)is slightly redundant sincecontent.annotationsis unconditionally set inContent.__init__, but is harmlessly defensive. No injection risks, resource leaks, secrets exposure, or unhandled failure modes found.
✓ Test Coverage
The PR adds four new tests and updates three existing streaming annotation tests to reflect the behavioral change from
Content.from_hosted_filetoContent.from_textwith citation annotations. Test coverage is solid for the major paths:file_citation,url_citation, andcontainer_file_citationroundtrips are all verified, as is thehosted_file→{}guard for assistant role. However, thefile_pathbranch of_annotations_to_output_text(theelif file_id:path on diff line ~300) has no roundtrip test, even though it is one of the three streaming annotation types that changed representation in this PR. The streaming-parse side is tested (test_streaming_annotation_added_with_file_path), but no test verifies that the resulting annotation serializes back to{"type": "file_path", "file_id": ...}via_annotations_to_output_text.
✓ Design Approach
The change fixes the immediate
input_filerejection, but the streaming-side approach still breaks the underlying model of citations: it now emits annotation-added events as separate emptytextcontents instead of attaching them to the streamed text they annotate. Since assistant history serialization writes eachContentas its ownoutput_textitem, the annotation offsets are no longer relative to the actual text, unlike the non-streaming path which keeps text and annotations together. That means the PR avoids one API error while still losing the text/citation relationship during round-trip.
Automated review by moonbox3's agents
- _annotations_to_output_text: fan out one entry per annotated_region for url_citation/container_file_citation (Annotation.annotated_regions is a Sequence; the API form carries one start/end per entry). - Validate region span bounds are ints before emitting; skip otherwise. - Add test for the file_path branch (annotation with file_id only). - Add test verifying streamed citation events coalesce onto surrounding text via _finalize_response so span indices reference the merged text, not the empty-text streaming carrier.
|
Pushed 4c67642 addressing all four review comments:
|
* Python: bump package versions for 1.2.2 release PATCH bump (1.2.1 -> 1.2.2) for the released cohort. Five PRs land in this window: - agent-framework-openai: fix file_search citations breaking the assistant- message history roundtrip (#5557) — drives the released-tier PATCH - agent-framework-orchestrations: [BREAKING] standardize orchestration terminal outputs as AgentResponse (#5301) - agent-framework-core, agent-framework-declarative: preserve Workflow.run() shared state across calls, accept list[Message] in declarative start executor, and coerce Enum values when serializing PowerFx symbols (#5531) - agent-framework-foundry-hosting: add hosted Durable Workflow support (#5531) - agent-framework-azure-contentunderstanding: new alpha package — Azure AI Content Understanding context provider (#4829) - dependencies: workspace package dependency refresh (#5555) Per lockstep convention, all 21 beta packages stamp 1.0.0b260429 and all 4 alpha packages (now including the new contentunderstanding) stamp 1.0.0a260429. Date stamp reflects 2026-04-29 Pacific. Every non-core package floor on agent-framework-core is raised to >=1.2.2; the new contentunderstanding package's stale >=1.0.0 floor is brought into line. Two follow-on fixes bundled to keep validate-dependency-bounds-test green at lowest-direct resolution: - Bump agent-framework-azure-contentunderstanding's azure-ai-content understanding lower bound from >=1.0.0 to >=1.0.1 (1.0.0 ships without proper typing — pyright reports 65 unknown-type errors) - Add pyright ignore comments to core/foundry/__init__.pyi for the new alpha package's type-stub imports, since alpha packages are not in core's [all] extra and therefore aren't installed at lowest-direct * Python: add #5552 to 1.2.2 CHANGELOG Add the streaming-span observability fix to the Fixed section. PR is on upstream/main but not yet pulled into origin/main; the code itself will land via the PR merge. * Python: address PR #5561 review feedback on dependency bounds Two packaging fixes flagged in review: 1. agent-framework-azure-contentunderstanding: add agent-framework-foundry as a runtime dependency. The package's README directs users to `pip install agent-framework-azure-contentunderstanding --pre` and the basic example imports `FoundryChatClient` from `agent_framework.foundry`, so the documented install path was failing with ImportError. Pulling agent-framework-foundry into deps makes the advertised entry path self-contained. 2. agent-framework-foundry: bump agent-framework-openai lower bound from >=1.1.0 to >=1.2.2,<2. Foundry imports private modules from agent_framework_openai (`_chat_client.py:22`, `_agent.py:34`), so resolvers were free to pair foundry==1.2.2 with older OpenAI versions that lack this release's coordinated Responses/history fix. Lockstep the floor with the released cohort to prevent mismatched installs. Both changes pass `validate-dependency-bounds-test` lower + upper at their respective packages.
Motivation and Context
Closes #5556. After the RC5 → 1.0 migration to the Responses API, multi-agent flows (
SequentialBuilder,GroupChatBuilder) break with a 400 from the API whenever one agent usesfile_searchand its history is forwarded to another agent. The root cause is inOpenAIResponsesClient(inherited byFoundryChatClient):_prepare_content_for_openaimapshosted_file → input_filefor any role, butinput_fileis an input-only content type that the Responses API rejects inside an assistant message.file_citation/container_file_citation/file_pathannotations as standaloneHostedFileContentitems, while the non-streaming path attaches them as text annotations. The asymmetry means streaming users always trip (1).output_texthardcodes"annotations": [], silently dropping citation context on every roundtrip even when (1) wouldn't fire.This worked in RC5 because Chat Completions used flat text annotations and had no input/output content-type schema split.
Description
Three coordinated fixes in
packages/openai/agent_framework_openai/_chat_client.py:hosted_filefor assistant role in_prepare_content_for_openai. Hosted-file content on an assistant message is a citation reference, not a replayable input file; dropping it stops the 400.file_citation,container_file_citation, andfile_pathnow produceContent.from_text(text="", annotations=[Annotation(type="citation", ...)])instead of a standaloneHostedFileContent. URL citations (which already used this pattern) are unchanged.output_textpreserves annotations on roundtrip via a new_annotations_to_output_texthelper that converts frameworkAnnotationobjects back to Responses API annotation dicts (file_citation,url_citation,container_file_citation,file_path).Built TDD-style: 4 new behavior tests plus an end-to-end regression test that exercises the exact streaming-citation → assistant-history-forwarding flow from the bug report. 3 existing streaming-annotation tests were updated to assert the new (consistent) behavior — they previously asserted the buggy split-content shape.
Verified locally:
pytest packages/openai/tests/openai/,pytest packages/foundry/tests/foundry/,pytest packages/core/tests/,mypy, andruffall clean.Contribution Checklist