fix-typing #2869

MagMueller · 2025-08-30T01:22:53Z

2 Critical fixes:

Our input_text did not use code inside dispatchKeyEvent
we marked labels as interactive by default, this lead to that sometimes we clicked on the wrong element - e.g. apartments.com

- Added logging for new elements detected during actions in the Agent class. - Implemented a human-like text field clearing method in DefaultActionWatchdog, utilizing Ctrl+A and Backspace. - Improved focus handling for label elements, ensuring they are only interactive if they do not have a 'for' attribute. - Updated clickable element detection logic to account for labels pointing to inputs. These changes improve the robustness of user interactions and enhance debugging capabilities.

- Enhanced the tag check to include truly interactive elements. - Removed special handling for 'label' elements, as they are now managed by other attribute checks to prevent interference with clickable elements. These updates improve the accuracy of interactive element detection in the DOM serializer.

- Added type hint for CDPSession in the _focus_element_simple method. - Enhanced logging for focus attempts, including exception details. - Reduced sleep duration in scrollIntoViewIfNeeded for better performance. - Updated text clearing logic to ensure it only occurs after successful focus. These changes enhance the robustness of element interaction and improve debugging capabilities.

github-actions · 2025-08-30T01:24:22Z

Agent Task Evaluation Results: 2/3 (67%)

View detailed results

Task	Result	Reason
amazon_laptop	✅ Pass	The agent successfully navigated to amazon.com, performed a search for 'laptop', and returned the name and details of the first laptop result. The output includes the product title, price, rating, and number of reviews, fulfilling the task requirements.
browser_use_pip	✅ Pass	The agent correctly identified and provided the pip installation command 'pip install browser-use' as requested. The output includes the exact command and additional relevant context, meeting the success criteria.
captcha_cloudflare	❌ Fail	The agent attempted to solve the captcha on the specified page but was unable to successfully complete it and extract the 'hostname' value as requested. The extracted content only included example JSON responses and site information related to the captcha solving process, but no actual dictionary labeled 'Captcha is passed successfully!' with a hostname field was found. Additionally, the expected hostname value was 'example.com', which was not present in any of the extracted data. Therefore, the agent did not fulfill the task requirements.

Check the evaluate-tasks job for detailed task execution logs.

cubic-dev-ai

4 issues found across 3 files

_{React with 👍 or 👎 to teach cubic. You can also tag @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

cubic-dev-ai · 2025-08-30T01:31:06Z

browser_use/dom/serializer/clickable_elements.py


-		# ENHANCED TAG CHECK: Include truly interactive elements
+				# ENHANCED TAG CHECK: Include truly interactive elements
+		# Note: 'label' removed - labels are handled by other attribute checks below - other wise labels with "for" attribute can destry the real clickable element on appartments.com


Typos in the added comment reduce clarity: "other wise", "destry", and "appartments.com".

Prompt for AI agents

Address the following comment on browser_use/dom/serializer/clickable_elements.py at line 98: <comment>Typos in the added comment reduce clarity: "other wise", "destry", and "appartments.com".</comment> <file context> @@ -94,14 +94,14 @@ def is_interactive(node: EnhancedDOMTreeNode) -> bool: - # ENHANCED TAG CHECK: Include truly interactive elements + # ENHANCED TAG CHECK: Include truly interactive elements + # Note: 'label' removed - labels are handled by other attribute checks below - other wise labels with "for" attribute can destry the real clickable element on appartments.com interactive_tags = { 'button', </file context>

Suggested change

# Note: 'label' removed - labels are handled by other attribute checks below - other wise labels with "for" attribute can destry the real clickable element on appartments.com

# Note: 'label' removed - labels are handled by other attribute checks below; otherwise labels with "for" attribute can destroy the real clickable element on apartments.com

cubic-dev-ai · 2025-08-30T01:31:06Z

browser_use/browser/watchdogs/default_action_watchdog.py

+					'type': 'keyDown',
+					'key': 'a',
+					'code': 'KeyA',
+					'modifiers': 2,  # Ctrl modifier


Use Meta (Cmd) on macOS for select-all; set modifiers to 4 on Darwin to ensure Ctrl/Cmd+A works cross-platform.

(This reflects your team's feedback about using tabs for indentation in fix suggestions.)

Prompt for AI agents

Address the following comment on browser_use/browser/watchdogs/default_action_watchdog.py at line 649: <comment>Use Meta (Cmd) on macOS for select-all; set modifiers to 4 on Darwin to ensure Ctrl/Cmd+A works cross-platform. (This reflects your team's feedback about using tabs for indentation in fix suggestions.)</comment> <file context> @@ -583,80 +584,184 @@ async def _type_to_page(self, text: str): + 'type': 'keyDown', + 'key': 'a', + 'code': 'KeyA', + 'modifiers': 2, # Ctrl modifier + 'windowsVirtualKeyCode': 65, }, </file context>

cubic-dev-ai · 2025-08-30T01:31:06Z

browser_use/browser/watchdogs/default_action_watchdog.py

 						'text': char,
 						'key': char,
+						'code': key_code,
+						'windowsVirtualKeyCode': ord(char.upper()) if char.isalpha() else ord(char),


Deriving windowsVirtualKeyCode from ASCII leads to wrong virtual key codes for punctuation; either omit this field for printable chars or map to correct VK codes to avoid mis-typed input.

Prompt for AI agents

Address the following comment on browser_use/browser/watchdogs/default_action_watchdog.py at line 846: <comment>Deriving windowsVirtualKeyCode from ASCII leads to wrong virtual key codes for punctuation; either omit this field for printable chars or map to correct VK codes to avoid mis-typed input.</comment> <file context> @@ -698,145 +803,67 @@ async def _input_text_element_node_impl(self, element_node, text: str, clear_exi 'text': char, 'key': char, + 'code': key_code, + 'windowsVirtualKeyCode': ord(char.upper()) if char.isalpha() else ord(char), }, session_id=cdp_session.session_id, </file context>

cubic-dev-ai · 2025-08-30T01:31:07Z

browser_use/browser/watchdogs/default_action_watchdog.py

+			'-': 'Minus',
+			'_': 'Underscore',
+			'@': 'At',
+			'!': 'Exclamation',


Using non-standard 'Exclamation' for 'code' is incorrect; map to the proper base key code (e.g., 'Digit1') and handle Shift via modifiers for reliable typing behavior.

(This reflects your team's feedback about using tabs for indentation in fix suggestions.)

Prompt for AI agents

Address the following comment on browser_use/browser/watchdogs/default_action_watchdog.py at line 597: <comment>Using non-standard 'Exclamation' for 'code' is incorrect; map to the proper base key code (e.g., 'Digit1') and handle Shift via modifiers for reliable typing behavior. (This reflects your team's feedback about using tabs for indentation in fix suggestions.)</comment> <file context> @@ -583,80 +584,184 @@ async def _type_to_page(self, text: str): + '-': 'Minus', + '_': 'Underscore', + '@': 'At', + '!': 'Exclamation', + '?': 'Question', + ':': 'Colon', </file context>

…ionWatchdog - Updated key code mappings for special characters to reflect correct usage with modifiers. - Enhanced text field clearing method to use platform-specific modifiers (Cmd for macOS, Ctrl for others) for a more human-like interaction. - Removed unnecessary `windowsVirtualKeyCode` assignments for printable characters to prevent incorrect virtual key code usage. These changes improve the accuracy of character input handling and enhance the robustness of text field interactions.

cursor · 2025-08-30T01:54:32Z

browser_use/browser/watchdogs/default_action_watchdog.py

-				await asyncio.sleep(0.01)
+
+				# Small delay between characters to look human (realistic typing speed)
+				await asyncio.sleep(0.001)


Bug: Text Input Sequence Fails CDP Standards

The _input_text_element_node_impl method's text input sequence deviates from standard CDP. It omits the crucial char event and incorrectly places the text parameter in keyDown events, which can lead to unreliable input on some sites. Furthermore, it fails to send necessary modifier keys (e.g., Shift) for special characters, potentially causing incorrect input for characters like _ or @.

MagMueller added 5 commits August 29, 2025 16:51

Emulate real human typing

bb3055e

Use existing method to get coordinates

662a397

MagMueller added 2 commits August 29, 2025 18:25

Typos

1ac519e

Fix test

4a8d1e8

cubic-dev-ai bot reviewed Aug 30, 2025

View reviewed changes

MagMueller added 4 commits August 29, 2025 18:34

Remove CDPSession in default watchdog

42e6cc8

Test fails if 0%

723c68c

Fix test

edbfcd2

MagMueller merged commit e7a7a62 into main Aug 30, 2025
27 of 55 checks passed

MagMueller deleted the fix-typing branch August 30, 2025 01:51

cursor bot reviewed Aug 30, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix-typing #2869

fix-typing #2869

Uh oh!

MagMueller commented Aug 30, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Aug 30, 2025 •

edited

Loading

Uh oh!

cubic-dev-ai bot left a comment

Uh oh!

cubic-dev-ai bot Aug 30, 2025

Uh oh!

cubic-dev-ai bot Aug 30, 2025

Uh oh!

cubic-dev-ai bot Aug 30, 2025

Uh oh!

cubic-dev-ai bot Aug 30, 2025

Uh oh!

Uh oh!

cursor bot Aug 30, 2025

Uh oh!

Uh oh!

	# Note: 'label' removed - labels are handled by other attribute checks below - other wise labels with "for" attribute can destry the real clickable element on appartments.com
	# Note: 'label' removed - labels are handled by other attribute checks below; otherwise labels with "for" attribute can destroy the real clickable element on apartments.com

fix-typing #2869

fix-typing #2869

Uh oh!

Conversation

MagMueller commented Aug 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Agent Task Evaluation Results: 2/3 (67%)

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Aug 30, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Aug 30, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Aug 30, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Aug 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor bot Aug 30, 2025

Choose a reason for hiding this comment

Bug: Text Input Sequence Fails CDP Standards

Uh oh!

Uh oh!

MagMueller commented Aug 30, 2025 •

edited

Loading

github-actions bot commented Aug 30, 2025 •

edited

Loading