feat(vscode-lm): re-apply image support with types version fix #11069
Conversation
Re-applies the image support feature from PR #11065 that was reverted, with a fix for the @types/vscode version issue. The fix keeps @types/vscode at ^1.84.0 in package.json (matching engines.vscode) while the pnpm-lock.yaml resolves to version 1.108.1 which provides the LanguageModelDataPart API. This approach: - Maintains compatibility with VS Code 1.84.0+ - Provides TypeScript types for the new LanguageModelDataPart API - Resolves the vsix build failure from the original PR Changes: - Add convertImageToDataPart() for image-to-LanguageModelDataPart conversion - Add checkModelSupportsImages() for model image capability detection - Update useSelectedModel to use models actual supportsImages capability - Add comprehensive tests for image support Relates to #11064, #11065
Reviewed changes since last review. The previous issue has been resolved by using static
Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues. |
src/api/providers/vscode-lm.ts
Outdated
| export const IMAGE_CAPABLE_MODEL_PREFIXES = [ | ||
| "gpt", // All GPT models (gpt-4o, gpt-4.1, gpt-5, gpt-5.1, gpt-5.2, gpt-5-mini, gpt-5.1-codex, etc.) | ||
| "claude", // All Claude models (claude-haiku-4.5, claude-opus-4.5, claude-sonnet-4, claude-sonnet-4.5) | ||
| "gemini", // All Gemini models (gemini-2.5-pro, gemini-3-flash-preview, gemini-3-pro-preview) | ||
| "o1", // OpenAI o1 reasoning models | ||
| "o3", // OpenAI o3 reasoning models | ||
| ] | ||
|
|
||
| /** | ||
| * Checks if a model supports image inputs based on its model ID. | ||
| * Uses prefix matching against known image-capable model families. | ||
| * | ||
| * @param _family The model family (unused, kept for API compatibility) | ||
| * @param id The model ID | ||
| * @returns true if the model supports image inputs | ||
| */ | ||
| export function checkModelSupportsImages(_family: string, id: string): boolean { | ||
| const idLower = id.toLowerCase() | ||
| return IMAGE_CAPABLE_MODEL_PREFIXES.some((prefix) => idLower.startsWith(prefix)) | ||
| } |
There was a problem hiding this comment.
The broad prefix matching here creates an inconsistency with the static vscodeLlmModels definitions in packages/types/src/providers/vscode-llm.ts. For example, gpt-3.5-turbo, gpt-4, gpt-4o-mini, o1, and o3-mini all have supportsImages: false in the static definitions, but this function returns true for them because they match the "gpt", "o1", or "o3" prefixes. This means VsCodeLmHandler.getModel() may report supportsImages: true while useSelectedModel (which uses the static definitions) reports supportsImages: false for the same model. Consider either updating the static definitions to match the dynamic behavior, or making this function more granular to exclude older models that don't support images.
Fix it with Roo Code or mention @roomote and request a fix.
There was a problem hiding this comment.
@NaccOll @hannesrudolph what do you think of this?
There was a problem hiding this comment.
@NaccOll @hannesrudolph what do you think of this?
Should use exact matching, and even consider adding a supportImage attribute to the statically defined models in vscode-lm instead of matching on the provider side. That would make adding and maintaining new models much easier.
There was a problem hiding this comment.
Fixaroo
See task on Roo Cloud
Fixed the inconsistency between checkModelSupportsImages and static vscodeLlmModels definitions. The function now checks static definitions first, then falls back to pattern matching for unknown models. All tests pass.
- Import vscodeLlmModels from @roo-code/types to check static definitions first - Use pattern matching only for unknown models not in static definitions - Add IMAGE_INCAPABLE_MODEL_PATTERNS for explicit non-vision models - Update IMAGE_CAPABLE_MODEL_PATTERNS to use RegExp for precise matching - Fix inconsistency where older models incorrectly reported supportsImages: true - Update tests to verify static definitions take precedence over pattern matching
Summary
Re-applies the image support feature for VS Code LM API provider from PR #11065 that was reverted in PR #11068, with a fix for the
@types/vscodeversion mismatch issue.The Problem
The original PR upgraded
@types/vscodefrom^1.84.0to^1.106.0insrc/package.json. This caused a mismatch with theengines.vscodefield which specifies^1.84.0, resulting inpnpm vsixbuild failures.The Solution
This PR keeps
@types/vscodeat^1.84.0inpackage.json(matchingengines.vscode) while thepnpm-lock.yamlresolves to version1.108.1which provides theLanguageModelDataPartAPI needed for image support. This approach:LanguageModelDataPartAPIChanges
convertImageToDataPart()for image-to-LanguageModelDataPart conversioncheckModelSupportsImages()for model image capability detectionTesting
Relates to #11064
Supersedes the reverted #11065
Important
Re-applies image support for VS Code LM API with a fix for
@types/vscodeversion mismatch, ensuring compatibility with VS Code 1.84.0+ and usingLanguageModelDataPartfor image handling.LanguageModelDataPart.@types/vscodeversion mismatch by keeping^1.84.0inpackage.jsonand resolving to1.108.1inpnpm-lock.yaml.convertImageToDataPart()invscode-lm-format.tsfor image conversion.checkModelSupportsImages()invscode-lm.tsfor model image capability detection.vscode-lm.spec.tsandvscode-lm-format.spec.ts.useSelectedModelinuseSelectedModel.tsto use model's actualsupportsImagescapability.This description was created by
for 2394dcd. You can customize this summary. It will automatically update as commits are pushed.