The latest updates on your projects. Learn more about Vercel for Git ↗︎

Adding support for Claude 3.7 is not as simple as I thought. Some things I've noticed so far:

1. Vercel AI SDK (@ai-sdk/anthropic 1.1.12) doesn't yet support the new bash_20250124 tool that is required by the new model. I'll open a PR with them unless someone else does it before.

   {
     "type": "error",
     "error": {
       "type": "invalid_request_error",
       "message": "'claude-3-7-sonnet-20250219' does not support tool types: bash_20241022, text_editor_20241022. Did you mean one of bash_20250124, computer_20250124, text_editor_20250124?"
     }
   }

2. The new computer_20250124 tool comes with new actions (vs computer_20241022), like: hold_key, left_mouse_down, left_mouse_up, triple_click, scroll, wait.

3. Our BrowserActionEnum only supports the existing actions (expected), plus our own custom actions (like navigate, check_email, etc). Needs to support the new actions from computer_20250124

High-level, we need some sort of tools registry to handle this:

• register all available tools supported (from providers and ours)
• AIClient uses a single interface to access available tools, which are determined based on the selected config.ai.model
• provide tool version-specific adapter that determines which actions are available for a given tool

For 1 I've created vercel/ai#5024

I've done an analysis between 2 runs to compare the models, and Claude 3.7 is 4.6 times more expensive at the first glance. For that reason, I'm not enabling it on our project, while keeping Claude 3.5 as the default model.

yt-claude-3-5-sonnet-20241022.txt
yt-claude-3-7-latest.txt

AI analysis

After comparing the two Shortest test results using Claude 3.5 Sonnet and Claude 3.7 Sonnet, I can identify several key differences in how they executed the same YouTube testing scenarios.

Main Differences

1. Testing Execution Approach

   Claude 3.7: More efficient and direct in its approach, often taking direct actions with fewer intermediate steps
   Claude 3.5: More verbose in its approach, with more screenshots, more careful checks, and more redundant actions

2. Token Usage

   Claude 3.7: Used significantly more tokens (1,253,265 tokens, ~$3.85)
   Claude 3.5: Used fewer tokens (267,581 tokens, ~$0.84)

3. Test Duration

   Claude 3.7: Longer test duration (416.62 seconds)
   Claude 3.5: Shorter test duration (151.72 seconds)

4. Decision Making

   Claude 3.7: Made more attempts to achieve the desired outcome, showing more persistence
   Claude 3.5: Often accepted initial results more readily

5. Diagnostic Behaviors

   Claude 3.7: Performed more extensive monitoring via more frequent screenshots
   Claude 3.7: Made more detailed observations about the state of the page
   Claude 3.7: Employed more waiting strategies to allow actions to complete

Why Claude 3.7 Used More Tokens

1. More Verbose Responses: Claude 3.7's messages were generally longer and more detailed about what it was observing and planning to do.
2. More Total Actions: Claude 3.7 performed more actions per test (especially screenshots), which generated more conversation turns between the AI and the testing framework.
3. More Diagnostic Messages : Claude 3.7 included more detailed analysis in its messages about what it observed on the screen and about the system's state.
4. More Persistence: When encountering issues (especially with video quality settings in the second test), Claude 3.7 made more repeated attempts, which resulted in more message exchanges.
5. More Thorough Verification: Claude 3.7 spent more time and messages verifying that actions had completed successfully.

The difference is particularly stark in the second test case (YouTube video playback settings) where Claude 3.7 used more than 1 million tokens ($3.15) compared to Claude 3.5's much more modest token usage ($0.61). This suggests that while Claude 3.7 may be more thorough and persistent, it's also significantly more expensive to run for these types of automated testing scenarios.

Cost Comparison

Claude 3.5 Sonnet (Total: ~$0.84)

Test 1 (Search for puppies): 61,558 tokens (~$0.20)
Test 2 (Change playback settings): 194,631 tokens (~$0.61)
Test 3 (Visit TED channel): 11,392 tokens (~$0.04)

Claude 3.7 Sonnet (Total: ~$3.85)

Test 1 (Search for puppies): 96,632 tokens (~$0.30)
Test 2 (Change playback settings): 1,032,598 tokens (~$3.15)
Test 3 (Visit TED channel): 124,035 tokens (~$0.39)

From this data, we can see that Claude 3.7 Sonnet is approximately 4.6 times more expensive than Claude 3.5 Sonnet for these test scenarios ($3.85 vs $0.84).
The most extreme difference is in Test 2 (changing YouTube playback settings), where Claude 3.7 used 5.3 times more tokens than Claude 3.5 ($3.15 vs $0.61). This suggests that Claude 3.7's thoroughness and persistence in handling complex interactions comes with a significantly higher cost.