🚨 Modularize ProcessorMixin into smaller components#45493
Merged
zucchini-nlp merged 46 commits intoMay 20, 2026
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Co-authored-by: Anton Vlasjuk <[email protected]>
ProcessorMixin into smaller componentsProcessorMixin into smaller components
ProcessorMixin into smaller componentsProcessorMixin into smaller components
Member
Author
|
Btw, re attr, they are still present under the same name and with the same content. The only diff is that it was an instance attr and not is a property, just in case it was not clear :) There is a tiiny chance that it got saved in |
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: aria, audioflamingo3, aya_vision, blip, chameleon, cohere2_vision, cohere_asr, colmodernvbert, exaone4_5, florence2, fuyu, gemma3, gemma4, glm46v, glm4v, glmasr |
yuchenxie4645
pushed a commit
to yuchenxie4645/transformers
that referenced
this pull request
May 28, 2026
) * tmp * more * . * . * qwen * apply to video with timestamps processing * mllama with no image tokens * delete dups * split modality fn * stricter check and consistent naming * fix videos and audio * bc for non-MLLM processors * some renaming and reordering * two more models * check wih audio processor * oops * delete more similar code * two more models * a bit more * more models * fix tests * fix idefics * fix repo * now it should pass CI * omg, a typo * fix tests * comments from eustlb * oops, typo * delete url from images utils * fix * some comments * docs * what, gh ui rebased weirdly * ! * fix repo * Apply suggestions from code review Co-authored-by: Steven Liu <[email protected]> * Update src/transformers/models/aya_vision/processing_aya_vision.py Co-authored-by: Anton Vlasjuk <[email protected]> * revert gemma3 --------- Co-authored-by: Steven Liu <[email protected]> Co-authored-by: Anton Vlasjuk <[email protected]>
lucianommartins
added a commit
to lucianommartins/my-transformers
that referenced
this pull request
May 29, 2026
…ultimodal tokens - Handle multimodal tokens in text that have no corresponding replacement data (e.g. <|video|> with videos=None) - Use next(iter, None) sentinel instead of bare next() to avoid StopIteration when the replacement iterator is exhausted - Preserve the original token text in the output when no replacement is available, matching pre-huggingface#45493 behavior - Regression introduced in ba06e3f (huggingface#45493) which refactored ProcessorMixin.__call__ into get_text_with_replacements with strict iterator consumption over all regex-matched tokens - Reproduces when downstream frameworks (e.g. vLLM) send profiling prompts containing multimodal placeholder tokens without actual multimodal data attached ) Signed-off-by: Luciano Martins <[email protected]>
6 tasks
kashif
pushed a commit
to kashif/transformers
that referenced
this pull request
Jun 1, 2026
) * tmp * more * . * . * qwen * apply to video with timestamps processing * mllama with no image tokens * delete dups * split modality fn * stricter check and consistent naming * fix videos and audio * bc for non-MLLM processors * some renaming and reordering * two more models * check wih audio processor * oops * delete more similar code * two more models * a bit more * more models * fix tests * fix idefics * fix repo * now it should pass CI * omg, a typo * fix tests * comments from eustlb * oops, typo * delete url from images utils * fix * some comments * docs * what, gh ui rebased weirdly * ! * fix repo * Apply suggestions from code review Co-authored-by: Steven Liu <[email protected]> * Update src/transformers/models/aya_vision/processing_aya_vision.py Co-authored-by: Anton Vlasjuk <[email protected]> * revert gemma3 --------- Co-authored-by: Steven Liu <[email protected]> Co-authored-by: Anton Vlasjuk <[email protected]>
ArthurZucker
added a commit
that referenced
this pull request
Jun 4, 2026
Migrate the image/video/audio token expansion of these processors to the shared ProcessorMixin helper introduced in #45493 instead of bespoke logic: each processor now defines `replace_image_token`/`replace_video_token`/ `replace_audio_token` and routes expansion through `get_text_with_replacements`. Models: deepseek_ocr2, colqwen2, ernie4_5_vl_moe (image+video), glm_image, llava_next (+ granite4_vision via modular), llava_onevision, granite_speech. Note: `get_text_with_replacements` mutates the text list in place, so callers that must not edit the caller's input pass a copy.
ArthurZucker
added a commit
that referenced
this pull request
Jun 4, 2026
Replace the earlier text-mutation test with one that spies on `get_text_with_replacements` and asserts the processor routes multimodal token expansion through it (the #45493 API) rather than bespoke logic. Skip it for processors that legitimately don't expand tokens: ColQwen2/ColModernVBert (text XOR images) and Mllama (one image token per image + cross-attention).
ArthurZucker
added a commit
that referenced
this pull request
Jun 4, 2026
Route image/video/audio placeholder-token expansion through the shared ProcessorMixin.get_text_with_replacements helper (#45493) for the processors that still used bespoke `.replace`/`re.sub`/sentinel logic, each now defining replace_image_token/replace_video_token/replace_audio_token (or building the replacement list inline where the expansion is context-dependent): pixtral, internvl, qianfan_ocr, lighton_ocr, qwen2_audio, csm, video_llava, smolvlm, higgs_audio_v2, emu3, cohere2_vision, kosmos2, florence2, deepseek_vl, deepseek_vl_hybrid, gemma3, minicpmv4_6, paddleocr_vl, phi4_multimodal, qwen2_5_omni, qwen3_omni_moe. Output is byte-for-byte identical (validated against each model's existing token-count tests, or byte-equivalence harnesses where tests are gated by torchaudio/unreleased checkpoints). Mllama is intentionally left as-is (one image token per image + cross-attention, nothing to expand). gemma3n is deferred: its Hub chat template emits soft tokens, so it needs a soft-token anchored migration rather than a boi/eoi one.
lucianommartins
added a commit
to lucianommartins/my-transformers
that referenced
this pull request
Jun 4, 2026
…ultimodal tokens - Handle multimodal tokens in text that have no corresponding replacement data (e.g. <|video|> with videos=None) - Use next(iter, None) sentinel instead of bare next() to avoid StopIteration when the replacement iterator is exhausted - Preserve the original token text in the output when no replacement is available, matching pre-huggingface#45493 behavior - Regression introduced in ba06e3f (huggingface#45493) which refactored ProcessorMixin.__call__ into get_text_with_replacements with strict iterator consumption over all regex-matched tokens - Reproduces when downstream frameworks (e.g. vLLM) send profiling prompts containing multimodal placeholder tokens without actual multimodal data attached ) Signed-off-by: Luciano Martins <[email protected]>
4 tasks
This was referenced Jun 5, 2026
khushali9
pushed a commit
to khushali9/transformers
that referenced
this pull request
Jun 8, 2026
) * tmp * more * . * . * qwen * apply to video with timestamps processing * mllama with no image tokens * delete dups * split modality fn * stricter check and consistent naming * fix videos and audio * bc for non-MLLM processors * some renaming and reordering * two more models * check wih audio processor * oops * delete more similar code * two more models * a bit more * more models * fix tests * fix idefics * fix repo * now it should pass CI * omg, a typo * fix tests * comments from eustlb * oops, typo * delete url from images utils * fix * some comments * docs * what, gh ui rebased weirdly * ! * fix repo * Apply suggestions from code review Co-authored-by: Steven Liu <[email protected]> * Update src/transformers/models/aya_vision/processing_aya_vision.py Co-authored-by: Anton Vlasjuk <[email protected]> * revert gemma3 --------- Co-authored-by: Steven Liu <[email protected]> Co-authored-by: Anton Vlasjuk <[email protected]>
This was referenced Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Modularizes
ProcessorMixinto make it easier for new processors to override smaller fn rather than the whole__call__. Splits__call__into smaller functions such as validation, input preparation, replacing multimodal placeholders, and a few properties for common special tokensIn simple cases like llava or qwen2-vl, the processor only has to override one method ->
replace_image_tokens. It takes a single image input and returns the corresponding placeholder text. More complicated models can override and add their own validation and input preparation, e.g. gemma3 requires nested images and has lots of sanity checksConverted a bunch of processors with different modalities to check that it works. I think for the rest we can either ask community to contrib or do it in a separate PR. This PR is already bloating up
Best way to review:
non-model files -> llava -> gemma3 -> audioflamingo -> idefics3 -> gemma4 -> test files(already includes variety of processor types)The "breaking" part is due to moving
self.image_ids/video_ids/audio_idsfrom instance attr to properties. Thus they will not be saved on config which is the correct way we want!