Add Granite 4.1 Vision (granite4_vision)#45597
Conversation
|
I've traced the root cause of the check_repository_consistency and tests_torch failures to a specific upstream commit: [Sam3LiteText] Remove unnecessary modules/configs (#45535) (7439ac0) This commit removed Sam3LiteTextViTConfig and Sam3LiteTextVisionConfig from the modeling file but left them referenced in auto_mappings.py, causing:
This is reproducible on main independently of our PR. Question for reviewers: Should we include a fix for this in our PR (removing the stale entries from auto_mappings.py), or would you prefer to handle it in a separate hotfix? Happy to do either. |
|
Opened a dedicated issue for the upstream regression: #45600 |
|
LMK when ready for review, and ig this PR supersedes #45350? |
|
@zucchini-nlp - yes, this PR supersedes #45350. Its our team that is responsible for producing/release IBM vision models. |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
@artem-spector great usage of modular!
Seems like the model uses granite llm as backbone with deepstack features. We will need to add an llm class in that case, since calling each backbone layer manually doesn't align well with our API. We can use modular to copy everything except for a single forward
As per adapters, can you explain how the weights are released? I am not really sure we have to manually add merge_adapters, prob I can suggest a cleaner way
| ```bibtex | ||
| @misc{granite-vision-4.1-4b, | ||
| title={Granite Vision 4.1}, | ||
| author={IBM Granite Vision Team}, | ||
| year={2026}, | ||
| url={https://huggingface.co/ibm-granite/granite-vision-4.1-4b} | ||
| } | ||
| ``` |
There was a problem hiding this comment.
nit: I think we dont need a bibtext entry and as long as there is a link to HF papers/arxiv, that is enought
There was a problem hiding this comment.
Done — removed.
| device=0, | ||
| torch_dtype=torch.bfloat16, | ||
| ) |
There was a problem hiding this comment.
nit: these two are by default "auto" so we dont need to manually set
There was a problem hiding this comment.
Done — removed.
|
|
||
| processor = AutoProcessor.from_pretrained(model_id) | ||
| model = AutoModelForImageTextToText.from_pretrained( | ||
| model_id, torch_dtype=torch.bfloat16, device_map="auto" |
There was a problem hiding this comment.
same here, torch_dtype is "auto" by default and can be deleted
There was a problem hiding this comment.
Done — removed.
| ## Notes | ||
|
|
||
| - The model includes LoRA adapters. Call `model.merge_lora_adapters()` after loading to merge them into base weights for faster inference. | ||
|
|
||
| - Set `padding_side="left"` during batched generation for more accurate results. | ||
|
|
||
| ```py | ||
| processor.tokenizer.padding_side = "left" | ||
| ``` | ||
|
|
||
| - The model supports specialized task tags for document extraction: `<chart2csv>`, `<chart2summary>`, `<chart2code>`, `<tables_html>`, `<tables_otsl>`, `<tables_json>`. Pass these as the text prompt along with a document image. | ||
|
|
||
| - For key-value pair extraction, provide a JSON schema describing the fields to extract. The model returns structured JSON matching the schema. |
There was a problem hiding this comment.
lets move this block as Usage Tips section, before the usage example code snippets
There was a problem hiding this comment.
Done — moved to a "Usage Tips" section before the code examples.
| @@ -0,0 +1,155 @@ | |||
| import math | |||
There was a problem hiding this comment.
following "one-model - one-file" philosophy, it is better put inside modular/modeling files
There was a problem hiding this comment.
Done — downsampling_granite4_vision.py deleted, all contents inlined into the modular.
| "openai-privacy-filter": "OpenAIPrivacyFilterConfig", | ||
| "lasr": "LasrCTCConfig", | ||
| "wav2vec2-with-lm": "Wav2Vec2Config", | ||
| "granite4-vision": "Granite4VisionConfig", | ||
| "hy-v3": "HYV3Config", | ||
| "slanet": "SLANetConfig", | ||
| } |
There was a problem hiding this comment.
Done — removed the stale entries introduced by bad rebases.
| WeightRenaming( | ||
| source_patterns=r"(vision_tower\.)vision_model\.", | ||
| target_patterns=r"\1", | ||
| ), |
There was a problem hiding this comment.
I think it is not needed anymore, we added PrefixWeights recently and fixed all llava models
There was a problem hiding this comment.
Done — removed the granite4_vision entry from conversion_mapping.py.
| @@ -0,0 +1,253 @@ | |||
| # Copyright 2025 IBM. All rights reserved. | |||
There was a problem hiding this comment.
Done — Copyright 2026 IBM and The HuggingFace Team.
| class Granite4VisionModelTester(VLMModelTester): | ||
| base_model_class = Granite4VisionModel | ||
| config_class = Granite4VisionConfig | ||
| conditional_generation_class = Granite4VisionForConditionalGeneration | ||
| text_config_class = GraniteConfig | ||
| vision_config_class = CLIPVisionConfig | ||
|
|
||
| def __init__(self, parent, **kwargs): |
There was a problem hiding this comment.
we need only this tester, since processing is identical to llava-next. Thanks for using VLMTester 🤩
There was a problem hiding this comment.
Done — removed test_image_processing_granite4_vision.py entirely (processing is identical to LlavaNext, no re-definition needed).
| "granite4_vision", | ||
| "falcon3", | ||
| "megatron_gpt2", | ||
| "code_llama", | ||
| "hy_v3", | ||
| "openai_privacy_filter", | ||
| "slanet", |
|
@zucchini-nlp I'm ready for the second round :) |
zucchini-nlp
left a comment
There was a problem hiding this comment.
Thanks @artem-spector ! Great work, glad to see a singlr modular file.
Left some comments on further cleaning-up, and a few questions. I just noticed that all image placeholders are filled with zeros, and not sure if that is intended. If we actually have no image features scattered, can we stop adding that many placeholders without quality degradation? Dummy and unnecessary token ids increase total length and looks like a waste of resources
| model_id = "ibm-granite/granite-vision-4.1-4b" | ||
|
|
||
| processor = AutoProcessor.from_pretrained(model_id) | ||
| model = AutoModelForImageTextToText.from_pretrained(model_id).eval() |
There was a problem hiding this comment.
nit: [...], device_map="auto") without eval, shoudl be already in eval mode when loaded
There was a problem hiding this comment.
Done — removed .eval().
| ("glm_image", {"pil": "GlmImageImageProcessorPil", "torchvision": "GlmImageImageProcessor"}), | ||
| ("glpn", {"pil": "GLPNImageProcessorPil", "torchvision": "GLPNImageProcessor"}), | ||
| ("got_ocr2", {"pil": "GotOcr2ImageProcessorPil", "torchvision": "GotOcr2ImageProcessor"}), | ||
| ("granite4_vision", {"pil": "Granite4VisionImageProcessorPil", "torchvision": "Granite4VisionImageProcessor"}), |
There was a problem hiding this comment.
can delete this, already mapped to 'LlavaNext'
There was a problem hiding this comment.
Done — removed.
| @@ -0,0 +1,734 @@ | |||
| # Copyright 2025 IBM. All rights reserved. | |||
There was a problem hiding this comment.
ig you forgot to commit and push 😄
|
|
||
|
|
||
| @dataclass | ||
| class Granite4VisionImageFeaturesOutput(ModelOutput): |
There was a problem hiding this comment.
i think same as qwen3_vl.BaseModelOutputWithDeepstackFeatures
There was a problem hiding this comment.
Done — Granite4VisionImageFeaturesOutput now inherits BaseModelOutputWithPooling (same approach as qwen3_vl.BaseModelOutputWithDeepstackFeatures), with a deepstack_features field added.
| class Granite4VisionTextConfig(PreTrainedConfig): | ||
| model_type = "granite4_vision_text" | ||
| base_config_key = "text_config" |
There was a problem hiding this comment.
not sure I am following, are we not supposed to inherit from GraniteConfig? Current class has no attributes
There was a problem hiding this comment.
Done — Granite4VisionTextConfig inherits GraniteConfig directly. It has no additional attributes because the text config is fully specified by GraniteConfig; the subclass exists only to set model_type = "granite4_vision_text" and base_config_key = "text_config".
| past_key_values=outputs.past_key_values, | ||
| hidden_states=outputs.hidden_states, | ||
| attentions=outputs.attentions, | ||
| deepstack_features=outputs.deepstack_features, |
There was a problem hiding this comment.
the only diff from llava-next is returning deepstack_features?
There was a problem hiding this comment.
Yes — the main differences from LlavaNextForConditionalGeneration are: (1) the model uses Granite4VisionModel which has deepstack injection, (2) logits are scaled by text_config.logits_scaling, and (3) deepstack_features is threaded through the output. Everything else is inherited.
| logits_to_keep=logits_to_keep, | ||
| **kwargs, | ||
| ) | ||
| model_inputs = self._init_hybrid_cache(**model_inputs) |
There was a problem hiding this comment.
we should be able to delete this line to inti cache. Correct cache should be init by generationMixin._prepare_cache_for_generation automatically
There was a problem hiding this comment.
Done — removed; GenerationMixin._prepare_cache_for_generation handles cache initialization.
| @unittest.skip("Granite4VisionImageFeaturesOutput has no hidden_states field") | ||
| def test_get_image_features_hidden_states(self): | ||
| pass | ||
|
|
||
| @unittest.skip("Granite4VisionImageFeaturesOutput has no attentions field") | ||
| def test_get_image_features_attentions(self): | ||
| pass |
There was a problem hiding this comment.
these two should be fixed when we add **kwargs and return a BaseModelOutputWithDeepstackFeatures
There was a problem hiding this comment.
Done — Granite4VisionImageFeaturesOutput now inherits BaseModelOutputWithPooling so all five tests pass. skip_test_image_features_output_shape = True remains because last_hidden_state isn't meaningful for this output type, but hidden_states, pooler_output, and field presence checks all pass.
| @unittest.skip("Base model forward returns ModelOutputWithPast, not CausalLMOutput with loss") | ||
| def test_training(self): | ||
| pass | ||
|
|
||
| @unittest.skip("QFormer submodules not initialized by init_weights from meta device") | ||
| def test_can_init_all_missing_weights(self): | ||
| pass |
There was a problem hiding this comment.
these also should be fixable, I don't think this is a valid reason to skip
There was a problem hiding this comment.
Done for test_training — the skip was stale; Granite4VisionForConditionalGeneration computes a loss, and the framework skips Granite4VisionModel automatically via MODEL_MAPPING_NAMES.\n\ntest_can_init_all_missing_weights remains skipped: Blip2QFormerModel submodules aren't initialized from meta device by our _init_weights — Blip2QFormerPreTrainedModel._init_weights only handles Blip2ForConditionalGeneration instances, not the standalone Blip2QFormerModel. Happy to investigate further if you'd like.
| "Granite4VisionConfig": [ | ||
| "multimodal_projector_bias", | ||
| "projector_hidden_act", | ||
| ], |
There was a problem hiding this comment.
if actually not used, lets just delete them from config class. When inheriting config, you can define
multimodal_projector_bias = AtributeErro()
and it will not copy this field
There was a problem hiding this comment.
Done — multimodal_projector_bias and projector_hidden_act are shadowed with AttributeError() at class level on Granite4VisionConfig, so the framework sees them as intentionally inaccessible. The SPECIAL_CASES_TO_ALLOW entry was removed.
|
Hi @zucchini-nlp, sorry for delay - I was chasing a performance problem, which turn out to be not real. Summary of changes:
Open question — test_can_init_all_missing_weights: This test verifies that every weight can be re-initialized from meta device via _init_weights. It fails because WindowQFormerDownsampler embeds a Blip2QFormerModel (a full PreTrainedModel from a different family). When from_pretrained(None, state_dict={}) walks submodules, our _init_weights has no isinstance branches for QFormer's internal layers (attention, embeddings, etc.), so they stay uninitialized. |
That is weird, usually our initializer walks down each sub-module and initializes its weights. Unless Qformer has special buffers that need to be init, it should be fine. I can check it on Monday and do another review pass |
will re-check too |
you right, |
zucchini-nlp
left a comment
There was a problem hiding this comment.
Nice, last comments on style, flagged a few that weren't addressed/pushed from prev review
Approving so we will request core maintainer review to merge after fixing left-over comments
| ("glm_image", {"pil": "GlmImageImageProcessorPil", "torchvision": "GlmImageImageProcessor"}), | ||
| ("glpn", {"pil": "GLPNImageProcessorPil", "torchvision": "GLPNImageProcessor"}), | ||
| ("got_ocr2", {"pil": "GotOcr2ImageProcessorPil", "torchvision": "GotOcr2ImageProcessor"}), | ||
| ("granite4_vision", {"pil": "LlavaNextImageProcessorPil", "torchvision": "LlavaNextImageProcessor"}), |
There was a problem hiding this comment.
this belong in image_processing_auto.py. Was it auto-generated or added manually?
There was a problem hiding this comment.
Done — removed. Not sure why it was added, but image_processing_auto.py already has it. Duplicate deleted.
| # Peek at vision hidden_size before super() to build a fully-specified qformer_config, | ||
| # avoiding any runtime field patching after super(). | ||
| if isinstance(self.vision_config, dict): | ||
| vision_hidden_size = self.vision_config.get("hidden_size", 1152) | ||
| elif self.vision_config is not None: | ||
| vision_hidden_size = self.vision_config.hidden_size | ||
| else: | ||
| vision_hidden_size = 1152 | ||
|
|
There was a problem hiding this comment.
vision/text configs cannot be None, no? In other words, we create a default config if not passes, just like with qformer below. Then we won't need a magical 1152 hardcoded here
If that is about the order, you can call super() at the beginning and it will unwrap first. Also we can completely override __post_init__ by calling PreTrainedConfig.__post_init__(**kwargs) instead of super
There was a problem hiding this comment.
Done — call super().__post_init__(**kwargs) first so vision_config is already a typed object when we build qformer_config.
| return ( | ||
| features.view(batch, side, side, channels) | ||
| .view(batch, num_win, window_size, num_win, window_size, channels) | ||
| .transpose(2, 3) | ||
| .flatten(0, 2) | ||
| .flatten(1, 2) | ||
| ) |
There was a problem hiding this comment.
ultra nit: for sake of readability I prefer to split into several lines. Same comment for unwindowed_raster
There was a problem hiding this comment.
Done — split into separate assignment lines in both methods.
| if isinstance(module, Granite4VisionTextRotaryEmbedding): | ||
| # Non-persistent buffers (inv_freq, original_inv_freq) are replaced with | ||
| # torch.empty_like() garbage by _move_missing_keys_from_meta_to_device. | ||
| # Recompute them here so _initialize_missing_keys restores correct values. | ||
| rope_type = module.config.rope_parameters.get("rope_type", "default") | ||
| if rope_type != "default": | ||
| rope_init_fn = ROPE_INIT_FUNCTIONS[rope_type] | ||
| else: | ||
| rope_init_fn = module.compute_default_rope_parameters | ||
| inv_freq, attention_scaling = rope_init_fn(module.config, module.inv_freq.device) | ||
| init.copy_(module.inv_freq, inv_freq) | ||
| init.copy_(module.original_inv_freq, inv_freq) |
There was a problem hiding this comment.
didn't push commit? 😄 Same for LN and embed modules, calling super is usually enough unless model requires special weight init
| base_model_prefix = "model" | ||
| _no_split_modules = ["Granite4VisionTextDecoderLayer"] |
There was a problem hiding this comment.
same comment, I think you have one local commit unpushed
|
run-slow: granite4_vision |
|
@zucchini-nlp , thanks for the approval! |
|
@artem-spector not the parent llava-next, it is handled in base |
| if isinstance(self.vision_config, dict): | ||
| vision_hidden_size = self.vision_config.get("hidden_size", 1152) | ||
| elif self.vision_config is not None: | ||
| vision_hidden_size = self.vision_config.hidden_size | ||
| else: | ||
| vision_hidden_size = 1152 |
There was a problem hiding this comment.
not really what I meant. Vision config has to be a config obj by this moment, so you only call vision_hidden_size = self.vision_config.hidden_size
Ordering correctly should do, move viision config code block before and that is all
There was a problem hiding this comment.
Done — three clean blocks: dict→typed conversion before super(), super().__post_init__ deserializes vision_config, then default qformer_config built from self.vision_config.hidden_size. No fallbacks or hardcoded values.
| @merge_with_config_defaults | ||
| @can_return_tuple |
There was a problem hiding this comment.
lets delete all, so it is copied completely. Current state is missing auto docstring
There was a problem hiding this comment.
The get_image_features body is entirely custom (deepstack multi-layer extraction + spatial projectors, different return type) so the method needs to stay in modular — deleting it would copy the LlavaNext version and break the model. Added @auto_docstring and @can_return_tuple.
|
A couple comments only, I think we had a misunderstanding with prev round. And requested review from a core maintainer (@vasqu) , then we can merge |
|
@zucchini-nlp, thank you! I'm working on it, hope to finish soon. BTW what hour is "until evening"? |
|
@zucchini-nlp Few questions: Re post_init super order: Re Fraction and torch-compile: Re _init_weights super call: |
Already fixed no? All subconfigs have to be an obj before calling super(), and we can do so by completely overriding
Yes, ofc. I should've also fixed that but feel free if anything is left :) |
|
Just pushed some small fixes for CI but feel to add onto them if it did not catch everything |
|
@artem-spector can you wait a second, the last force push deleted the last commits. Let me push them |
|
Should be good now |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, granite4_vision, llava_next, llava_next_video, llava_onevision, ovis2 |
- Expand abbreviations in QFormerDownsampler.forward: interp_side, enc, downsampled_w, out_w → interpolated_side, windowed_image_features, downsampled_windowed, out_windowed - Inline _deepstack_inject into Granite4VisionTextModel.forward - Use url_to_local_path for test images; add 000000001000.jpg to fetch_hub_objects_for_ci.py - Regenerate modeling_granite4_vision.py from modular Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
* Add Granite 4.1 Vision model (granite4_vision) Full implementation of IBM Granite 4.1 Vision as a built-in HF model: - Modular implementation (modular_granite4_vision.py) - Generated files: config, modeling, image processing, processing - Auto-registration: config, modeling, processing, image processing - Tests: modeling (unit + @slow), image processor, processor - Documentation (docs/source/en/model_doc/granite4_vision.md) - WeightRenaming to handle SiglipVisionModel vision_model. nesting Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix auto-registration after upstream auto_mappings refactor Upstream moved CONFIG_MAPPING_NAMES to auto_mappings.py. Add granite4_vision entry there; resolve leftover conflict markers in configuration_auto.py (granite4_vision is already in modeling_auto.py and processing_auto.py). Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix conflict marker in image_processing_auto.py Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix check-repo: remove spatial_stride (unused in modeling), fix auto_mappings duplicate Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix duplicate legacy key in conversion_mapping.py Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix check-repo failures for granite4_vision - Remove granite4_vision from MISSING_IMAGE_PROCESSOR_MAPPING_NAMES (auto-discovered via TorchvisionBackend/PilBackend) - Add granite4-vision to HARDCODED_CONFIG_FOR_MODELS in auto_docstring.py - Add granite4_vision to DOC_MODEL_NAMES_NOT_IN_AUTO in check_repo.py - Fix import sort in models/__init__.py and test file - Regenerate auto_mappings.py via check_auto.py --fix_and_overwrite - Add dates to granite4_vision.md Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * Regenerate auto_mappings.py after rebase onto upstream/main Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * Fix CI failures after upstream rebase - Fix processing_auto.py sort order (sort_auto_mappings) - Add hy-v3, openai-privacy-filter, slanet to HARDCODED_CONFIG_FOR_MODELS - Add hy_v3, openai_privacy_filter, slanet to DOC_MODEL_NAMES_NOT_IN_AUTO (new upstream models missing from these registries) Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * Fix Sam3 auto_mappings.py entries corrupted by rebase sam3_vision_model and sam3_vit_model were incorrectly mapped to Sam3LiteTextVisionConfig/Sam3LiteTextViTConfig instead of Sam3VisionConfig/Sam3ViTConfig (and sam3_lite_text module instead of sam3). These are unrelated to granite4_vision; restoring upstream/main values. Signed-off-by: artemspector <[email protected]> * Restore hy_v3, openai_privacy_filter, slanet entries dropped by bad rebase regeneration These three upstream model entries were accidentally removed from CONFIG_MAPPING_NAMES in auto_mappings.py by a previous run of check_auto.py --fix_and_overwrite during an incomplete rebase state. Restoring verbatim from upstream/main. Signed-off-by: artemspector <[email protected]> * Revert dependency_versions_table.py to match setup.py (upstream state) Signed-off-by: artemspector <[email protected]> * Fix bad rebase: remove hy-v3/slanet/openai-privacy-filter entries from auto_docstring and check_repo These entries belong to other upstream PRs and were accidentally included during a previous rebase. Our PR only owns the granite4_vision entries. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * Remove merge_lora_adapters and PEFT adapter-toggling generate override The hub checkpoint ships with pre-merged weights; PEFT-style merging doesn't fit the HF API. Regenerated modeling file from modular via converter. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * Add Granite4VisionTextModel with deepstack injection, replace manual layer loop Instead of iterating self.language_model.layers from the VLM model's forward, introduce Granite4VisionTextModel(GraniteModel) that owns the layer loop and accepts deepstack_features (dict[layer_idx -> tensor]) and vision_mask. Granite4VisionModel.forward() now calls self.language_model(...) cleanly. Pattern follows Qwen3VL. Regenerated modeling file from modular. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * Add Granite4VisionTextConfig, fix missing TextConfig import in generated file The modular converter generates a TextConfig subclass for the text model's sub-layers. Define Granite4VisionTextConfig(GraniteConfig) explicitly in modular so the converter resolves it correctly instead of creating an undefined reference. Regenerated config and modeling files. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * Fix Granite4VisionTextConfig to inherit PreTrainedConfig, add missing import Inheriting GraniteConfig caused the converter to drop the import in the generated config file. Align with Qwen3VL pattern: TextConfig inherits PreTrainedConfig directly. Also add PreTrainedConfig import to modular. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * Fix class ordering: define Granite4VisionPreTrainedModel before TextModel The converter respects source order; TextModel must come after PreTrainedModel. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * Fix inv_freq corruption in Granite4VisionTextRotaryEmbedding during from_pretrained When loading with device_map, HF's _move_missing_keys_from_meta_to_device replaces all non-persistent buffers with torch.empty_like() (garbage memory). Add a _init_weights handler for Granite4VisionTextRotaryEmbedding that recomputes inv_freq and original_inv_freq from config, so _initialize_missing_keys restores correct values after the corruption. Also adds Granite4VisionTextRotaryEmbedding as an explicit subclass in the modular file so the isinstance check resolves correctly. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Inline downsampling into modular, add qformer_config sub-config, convert to pure functions - Delete downsampling_granite4_vision.py; move WindowQFormerDownsampler, interpolate_downsample, and spatial_offset_downsample into modular - Replace stateless InterpolateDownsampler/SpatialOffsetDownsampler classes with plain functions (items 2 and 4 from reviewer feedback) - Add config.qformer_config (Blip2QFormerConfig) as a proper sub-config field on Granite4VisionConfig following the Blip2Config pattern; remove inline Blip2QFormerConfig construction from WindowQFormerDownsampler.__init__ (item 3) Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Return Granite4VisionImageFeaturesOutput from get_image_features Replace the raw list-of-tuples return from get_image_features with a proper @DataClass ModelOutput subclass (Granite4VisionImageFeaturesOutput), following the Qwen3-VL BaseModelOutputWithDeepstackFeatures pattern. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Drop Granite4Vision image processor re-definitions, delegate to LlavaNext The image processors are identical to LlavaNextImageProcessor and LlavaNextImageProcessorPil; no need to re-define them. Map 'granite4_vision' to the LlavaNext processors in image_processing_auto.py. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Address medium PR review items 8-14 - Item 8: move query/image_positions init to _init_weights (embed_std pattern) - Item 9: rename _win/_unwin to _windowed_raster/_unwindowed_raster, replace single-letter vars with descriptive names - Item 10: add deepstack_features field to Granite4VisionModelOutputWithPast and Granite4VisionCausalLMOutputWithPast instead of reusing image_hidden_states - Item 11: use TransformersKwargs instead of FlashAttentionKwargs in Granite4VisionModel.forward; remove unused FlashAttentionKwargs import - Item 12: raise ValueError instead of warning_once for patch shape mismatch; remove now-unused logger - Item 13: drop use_image_newline_parameter (not used in released checkpoint) - Item 14: read pad_token_id from config.text_config instead of top-level config Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Address nit PR review items 15-23 - Item 15: fix copyright to "2026 IBM and The HuggingFace Team" - Item 16: remove bibtex entry from docs - Item 17: remove torch_dtype/device_map from docs examples - Item 18: move Notes to "Usage Tips" section before code examples - Item 19: remove model_type from Granite4VisionProcessor - Item 20: revert AttributeError() (converter incompatible); keep del self. - Item 21: remove granite4_vision from conversion_mapping (PrefixWeights handles it) - Item 22: remove granite4_vision from check_repo DOC_MODEL_NAMES_NOT_IN_AUTO and HARDCODED_CONFIG_FOR_MODELS in auto_docstring (bad rebase entries) - Item 23: update test copyright, remove use_image_newline_parameter from tester, update skip reasons for get_image_features tests Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Address remaining review items 20 and 29 - Item 20: drop get_image_token_mask override, use parent's get_placeholder_mask - Item 29: delete test_image_processing_granite4_vision.py (identical to LlavaNext) Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix test failures found by Granite4VisionModelTest - Pass output_attentions/output_hidden_states explicitly to language_model in Granite4VisionModel.forward (were swallowed as explicit params, not forwarded via **kwargs) - Collect all_hidden_states and all_self_attns in Granite4VisionTextModel layer loop; add output_attentions/output_hidden_states params - Fix qformer_config dict→object conversion to run before super().__post_init__() so _attn_implementation.setter doesn't hit a raw dict during sub_configs iteration - Use Blip2QFormerConfig directly in sub_configs (instead of AutoConfig) so save/load round-trip resolves the type correctly; add missing import to generated configuration_granite4_vision.py Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Use lazy import for Blip2QFormerConfig; qformer_config sub_configs uses AutoConfig blip_2_qformer is registered in CONFIG_MAPPING so AutoConfig resolves it correctly. Moving the Blip2QFormerConfig import inside __post_init__ avoids a cross-model top-level import that the modular converter drops from the generated file. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Add Granite4VisionTextModel to check_repo ignore lists; document missing public classes - IGNORE_NON_TESTED + IGNORE_NON_AUTO_CONFIGURED: Granite4VisionTextModel is an internal subcomponent tested implicitly through Granite4VisionModel - Doc: add autodoc entries for Granite4VisionTextConfig, Granite4VisionTextModel, Granite4VisionImageProcessor, Granite4VisionImageProcessorPil Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix missing imports in modular: math, AutoConfig, select_best_resolution Needed for ruff F821 (undefined name) to pass under make style. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Pop output_attentions/output_hidden_states from **kwargs in Granite4VisionModel.forward Aligns with reviewer feedback: these args are not needed in the explicit signature since they flow through kwargs: Unpack[TransformersKwargs]. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix check_modeling_structure violations (TRF002, TRF009, TRF010) - TRF010: add @strict to Granite4VisionTextConfig (direct PreTrainedConfig subclass) - TRF002: set base_model_prefix = "model" on Granite4VisionTextModel (was "") - TRF009: add trf-ignore comment on Blip2QFormerModel lazy import (cross-model import is intentional — QFormer is a shared building block) Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix ruff I001 import ordering and processing consistency check - Reorder imports in modular to satisfy ruff isort (stdlib → third-party → first-party) - Sync processing_granite4_vision.py to match converter output (BatchFeature from feature_extraction_utils, no model_type on processor) Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Remove converter-regenerated files that should not exist downsampling_granite4_vision.py, image_processing_granite4_vision.py, and image_processing_pil_granite4_vision.py are regenerated by the converter but were previously intentionally deleted: image processors delegate to LlavaNext (registered in image_processing_auto.py), and downsampling is inlined in modular/modeling. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Remove autodoc entries for ImageProcessor classes that don't exist in this model Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix model card date for add_dates.py check Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix dependency_versions_table.py: sync mlinter version with setup.py Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Address review round 3: config, capture_outputs, hidden states, projector naming - Granite4VisionTextConfig restored as proper GraniteConfig subclass - Granite4VisionConfig.__post_init__: convert dict->config before super() so _attn_implementation.setter sees config objects; patch vision-size fields after super() - Use CONFIG_MAPPING/AutoModel at module top-level (no lazy imports) - Add _can_record_outputs to Granite4VisionTextModel for hidden_states/attentions capture via @capture_outputs decorator - Add Granite4VisionTextAttention/TextDecoderLayer stubs in modular so converter generates the registry entries pointing to the correct layer classes - WindowQFormerDownsampler renamed to Granite4VisionWindowQFormerDownsampler - interpolate_downsample/spatial_offset_downsample take explicit size args (not config) - Remove output_attentions/output_hidden_states from forward signatures (handled by @capture_outputs and **kwargs); use BaseModelOutputWithPast return type - Remove prepare_inputs_for_generation (handled by parent) - Remove _init_hybrid_cache (GraniteMoeHybrid leftover from 4.0) - auto_mappings.py: use LlavaNextImageProcessor(Pil) instead of model-specific copies - docs: remove .eval() from example Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * Nits: rename one-letter vars, AttributeError() for unused inherited config attrs - _windowed_raster/unwindowed_raster: x -> features, x_win -> windowed_features - __init__: q, w -> query_side_str, window_side_str - Granite4VisionConfig: shadow LlavaNextConfig's multimodal_projector_bias and projector_hidden_act with AttributeError() so check_config_attributes passes without SPECIAL_CASES_TO_ALLOW entry - Remove Granite4VisionConfig from check_config_attributes.py SPECIAL_CASES_TO_ALLOW Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * Fix qformer_config: build fully-specified at init, no post-super field patching Peek at vision_config.hidden_size (or its dict equivalent) before super() and include hidden_size, num_attention_heads, encoder_hidden_size directly in the CONFIG_MAPPING["blip_2_qformer"]() constructor call. This avoids mutating the config object after super().__post_init__() runs. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * Move _can_record_outputs and _deepstack_inject to Granite4VisionPreTrainedModel Follows qwen3_vl pattern: _can_record_outputs and _deepstack_inject belong on the shared PreTrainedModel base class, not on TextModel. TextAttention/ TextDecoderLayer stubs are defined before PreTrainedModel so the converter generates locally-scoped classes for the _can_record_outputs registry. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * Address remaining review items: capture_outputs, output class, DynamicCache - Granite4VisionImageFeaturesOutput now inherits BaseModelOutputWithPooling so the common test framework can introspect last_hidden_state/pooler_output/ hidden_states/attentions fields (removes 5 test skips) - Add @capture_outputs to Granite4VisionTextModel.forward so output_hidden_states is collected via hooks and propagated through to the causal LM output (fixes test_assisted_decoding_matches_greedy_search) - get_image_features: populate hidden_states from vision tower when output_hidden_states=True (via kwarg or config); removes test skip - Remove stale test_training skip (ForConditionalGeneration computes loss; base model is skipped automatically by MODEL_MAPPING_NAMES check) - Delete DynamicCache init block from Granite4VisionTextModel.forward; GenerationMixin._prepare_cache_for_generation handles this - Import BaseModelOutputWithPooling, capture_outputs; drop DynamicCache import Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * Regenerate modeling/processing from updated modular; fix copyright header Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix ruff formatting in modular_granite4_vision.py Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix add_dates.py: update granite4_vision model card date to 2026-04-30 Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Remove incorrect skip from test_can_init_all_missing_weights The test actually passes — framework's recursive apply() handles QFormer submodule weights correctly via the nn.Linear branch of _init_weights. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix add_dates.py: update granite4_vision model card date to 2026-05-03 Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix _init_weights: add nn.Embedding, nn.LayerNorm, Granite4VisionTextRMSNorm branches These were missing, causing test_can_init_all_missing_weights to fail in CI when weights initialized from meta device didn't match __init__ values. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix ruff F821: detect RMSNorm by attribute pattern instead of class name Granite4VisionTextRMSNorm is defined in the generated file, not modular, so referencing it by name in modular_granite4_vision.py is an undefined name. Use attribute-based detection (has weight + variance_epsilon, not Linear) instead. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * Fix ruff formatting in modular_granite4_vision.py Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * granite4_vision: address round-3 review nits - Remove duplicate image processor entry from auto_mappings.py (already in image_processing_auto.py) - Split _windowed_raster and _unwindowed_raster chains into multiple lines - Move _deepstack_inject from Granite4VisionPreTrainedModel into Granite4VisionTextModel (only used there) - Remove nn.Embedding/nn.LayerNorm branches from _init_weights (super handles them) - Remove use_spatial_sampling conditional guard — spatial projectors always active in released weights - Remove pixel_values.size(0) > 0 check (bad copy from llava-next) - Add inline comment explaining the intentional masked_fill(..., 0.0) for deepstack - Replace manual vision_feature_layer/strategy fallbacks with @merge_with_config_defaults Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * granite4_vision: fix __post_init__ ordering — call super() first, drop hardcoded 1152 super().__post_init__() deserializes vision_config from dict to a typed object, so qformer_config can read vision_config.hidden_size directly without a fallback. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * granite4_vision: address round-3 nits + fix __post_init__ ordering Changes: - __post_init__: call super() first so vision_config is deserialized before building qformer_config default; qformer_config dict converted before super() to avoid _attn_implementation setter hitting a raw dict - import merge_with_config_defaults from ...utils.generic (not ...utils) - restore _init_weights Embedding/LayerNorm branches — needed for test_can_init_all_missing_weights (LlavaNextPreTrainedModel doesn't handle them) Regenerate modeling and configuration files. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * granite4_vision: replace unbound forward call with self.language_model() The Granite4VisionTextModel.forward(self.language_model, ...) pattern was added to bypass nn.Module.__call__ overhead, but compare_implementations shows no measurable difference (1.05x total vs card, within noise). Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * granite4_vision: register Granite4VisionTextConfig in auto_mappings.py check_auto.py requires granite4_vision_text to appear in CONFIG_MAPPING_NAMES and MODEL_TYPE_TO_MODULE_NAME. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * granite4_vision: remove use_spatial_sampling from config Spatial sampling is always enabled in all released weights (use_spatial_sampling: true in config.json), so the conditional guard was already removed from modeling code. Remove the now-unused attribute from the config to fix check_config_attributes.py. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * granite4_vision: update HF Transformers date in model card to 2026-05-05 Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * granite4_vision: fix test_model_is_small — shrink qformer intermediate_size in tester QFormer defaulted to intermediate_size=3072 even in tests, pushing the tiny test model to 2.29M params (limit is 1M). Set intermediate_size=64 in get_config(). Also remove stale use_spatial_sampling from tester. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * granite4_vision: round 4 review fixes (C14, C15, C16) - C14: reorder __post_init__ — dict→typed qformer_config conversion stays before super() (needed for _attn_implementation propagation), then super().__post_init__ deserializes vision_config, then default qformer_config built from self.vision_config.hidden_size with no dict fallback or hardcoded 1152 - C15: move base_model_prefix/_no_split_modules to Granite4VisionPreTrainedModel; remove from Granite4VisionTextModel - C16: add @can_return_tuple + @auto_docstring to get_image_features override Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * granite4_vision: simplify __post_init__ — remove comments, tighten structure Three clean blocks: guard fields, dict→typed conversion before super(), default build from self.vision_config.hidden_size after super(). No fallbacks. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: artemspector <[email protected]> * Update docs/source/en/model_doc/granite4_vision.md Co-authored-by: Anton Vlasjuk <[email protected]> * Apply suggestions from code review Co-authored-by: Anton Vlasjuk <[email protected]> * commets * fix modular * config! * init weights fix, has an image newline * fix * fixup repo ci * docs fix * interesting TIL * fix auto doc * granite4_vision: re-apply round 5 fixes lost in force-push - Expand abbreviations in QFormerDownsampler.forward: interp_side, enc, downsampled_w, out_w → interpolated_side, windowed_image_features, downsampled_windowed, out_windowed - Inline _deepstack_inject into Granite4VisionTextModel.forward - Use url_to_local_path for test images; add 000000001000.jpg to fetch_hub_objects_for_ci.py - Regenerate modeling_granite4_vision.py from modular Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * granite4_vision: ruff format modular_granite4_vision.py Co-Authored-By: Claude Sonnet 4.6 <[email protected]> --------- Signed-off-by: artemspector <[email protected]> Co-authored-by: artemspector <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Raushan Turganbay <[email protected]> Co-authored-by: Anton Vlasjuk <[email protected]> Co-authored-by: raushan <[email protected]> Co-authored-by: vasqu <[email protected]>
What does this PR do?
Adds built-in support for Granite 4.1 Vision (
granite4_vision), IBM's multimodal vision-language model for enterprise document understanding.Architecture highlights
google/siglip2-so400m-patch16-384), tiled 384×384 patchesdownsample_rate="4/8")Files added
modular_granite4_vision.pyconfiguration_granite4_vision.pymodeling_granite4_vision.pyprocessing_granite4_vision.pyimage_processing_granite4_vision.pyimage_processing_pil_granite4_vision.pytests/models/granite4_vision/docs/source/en/model_doc/granite4_vision.mdAuto-registration
configuration_granite4_vision.pymodel_typeMODEL_MAPPING_NAMES+MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMESTests
pytest tests/models/granite4_vision/ -x -q)@slowintegration tests load real checkpoint and assert outputs within tolerancemake styleandmake check-repopass (3 remaining failures are pre-existing upstream issues:mlinterversion mismatch andSam3Liteincomplete model)Before submitting
Related