fix(llm/google): fix image handling in first user message with system text #3513
+44
−32
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Refactor GoogleMessageSerializer to improve image handling and clarify comments.
Contributor
There was a problem hiding this comment.
1 issue found across 1 file
Prompt for AI agents (all 1 issues)
Understand the root cause of the following 1 issues and fix them.
<file name="browser_use/llm/google/serializer.py">
<violation number="1" location="browser_use/llm/google/serializer.py:89">
Aggregating all user text into one part here reorders mixed content: text that originally followed an image now appears before it when include_system_in_user=True. Please keep iterating the parts in order so images remain in their original position relative to text.</violation>
</file>
React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.
browser_use/llm/google/serializer.py
Outdated
| # Add system text as the first part | ||
| message_parts.append(Part.from_text(text=system_text)) | ||
| system_parts = [] # Clear after using | ||
| message_parts.append(Part.from_text(text=f'{system_text}\n\n{getattr(message, "text", "")}')) |
Contributor
There was a problem hiding this comment.
Aggregating all user text into one part here reorders mixed content: text that originally followed an image now appears before it when include_system_in_user=True. Please keep iterating the parts in order so images remain in their original position relative to text.
Prompt for AI agents
Address the following comment on browser_use/llm/google/serializer.py at line 89:
<comment>Aggregating all user text into one part here reorders mixed content: text that originally followed an image now appears before it when include_system_in_user=True. Please keep iterating the parts in order so images remain in their original position relative to text.</comment>
<file context>
@@ -82,39 +86,31 @@ def serialize_messages(
- # Add system text as the first part
- message_parts.append(Part.from_text(text=system_text))
- system_parts = [] # Clear after using
+ message_parts.append(Part.from_text(text=f'{system_text}\n\n{getattr(message, "text", "")}'))
+ for part in message.content or []:
+ if part.type == 'image_url':
</file context>
✅ Addressed in a39cd7b
Refactor message serialization to handle text parts more effectively by prepending system text only before the first text part and managing other part types accordingly.
Contributor
There was a problem hiding this comment.
Reviewed changes from recent commits (found 1 issue).
1 issue found across 1 file
Prompt for AI agents (all 1 issues)
Understand the root cause of the following 1 issues and fix them.
<file name="browser_use/llm/google/serializer.py">
<violation number="1" location="browser_use/llm/google/serializer.py:91">
If the first user message contains no text parts (e.g., images only), this condition prevents the system instructions from being included at all. Please ensure the system text is still inserted even when there is no text part in the message.</violation>
</file>
React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.
browser_use/llm/google/serializer.py
Outdated
| elif message.content is not None: | ||
| first_text_inserted = False | ||
| for part in message.content or []: | ||
| if part.type == 'text' and not first_text_inserted: |
Contributor
There was a problem hiding this comment.
If the first user message contains no text parts (e.g., images only), this condition prevents the system instructions from being included at all. Please ensure the system text is still inserted even when there is no text part in the message.
Prompt for AI agents
Address the following comment on browser_use/llm/google/serializer.py at line 91:
<comment>If the first user message contains no text parts (e.g., images only), this condition prevents the system instructions from being included at all. Please ensure the system text is still inserted even when there is no text part in the message.</comment>
<file context>
@@ -85,15 +85,22 @@ def serialize_messages(
+ first_text_inserted = False
for part in message.content or []:
- if part.type == 'image_url':
+ if part.type == 'text' and not first_text_inserted:
+ # Prepend system text only before the first text part
+ message_parts.append(Part.from_text(text=f'{system_text}\n\n{part.text}'))
</file context>
✅ Addressed in b61454d
fix bug If the first user message contains no text parts (e.g., images only), this condition prevents the system instructions from being included at all. fix it by ensuring the system text is still inserted even when there is no text part in the message.
Ensure proper formatting by adding a newline at the end of the file.
Ensure proper file termination by adding a newline.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When include_system_in_user=True, non-text content (images) in the first
user message was not being serialized. Now properly handles mixed content
by extracting image serialization logic into a helper method.
Summary by cubic
Fixes image handling in the Google Gemini serializer so images in the first user message are preserved when system text is included. Also ensures system text is inserted before the first part, including image-only messages.
Bug Fixes
Refactors
Written for commit 8982da7. Summary will update automatically on new commits.