Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit eb2617a

Browse files
IlyasMoutawwakilclaudeCopilot
authored and
khushali9
committed
Extract dynamic vision/audio tensors into standalone pure functions (huggingface#45396)
* Extract pure vision/audio functions into standalone utilities - Create top-level `modeling_vision_utils.py` with shared pure functions: `get_vision_cu_seqlens`, `get_rotary_pos_ids`, `get_rotary_pos_ids_interleaved`, `get_window_index`, `get_pos_embed_indices` - Move audio precompute functions (`chunk_and_pad_features`, `get_audio_cu_seqlens`, `get_valid_indices`, `get_pool_indices`) into modular files directly - Simplify `VisionRotaryEmbedding.forward` to accept `pos_ids` tensor directly via broadcast multiply, eliminating data-dependent table creation - Make vision/audio encoder forwards accept optional precomputed tensors (`cu_seqlens`, `rotary_pos_ids`, `window_index`, `embed_indices`, etc.) - Use explicit naming: `get_vision_cu_seqlens` / `get_audio_cu_seqlens` Models: qwen2_vl, qwen2_5_vl, qwen3_vl, qwen3_5, qwen3_vl_moe, qwen3_5_moe, qwen2_5_omni, qwen3_omni_moe, glm4v, glm4v_moe, glm_image, glm_ocr, ernie4_5_vl_moe, video_llama_3, mlcd, paddleocr_vl Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * Fix stale compute_ docstring references to match actual function names Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * Revert mlcd changes — not part of this PR Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix * kwargs * opt-in * fix dtype * style * guard torch import * standarize * propagate inputs * fix docs * fix docs * auto docs * more docs fixing * fix omni * fix paddle * revert paddle ocr until another time * finally fixed paddle ocr * fix review * revert chunking * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <[email protected]> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <[email protected]> * fix torch compilable check * fix docs * correct func name * fix omni * fix video llama 3 * fix video llama 3 * requires torch * add missing grid device * keep rot emb in fp32 * fix test device * fix flm4v flex attention test * rename to vision utils * only one get_rotary_pos_ids is needed * style * style * deprecate only * fix * simplify and revert processor changes * renames * move some stuff to their original place * style * style * use chunked attention * use decorator * pass kwargs and return_dict * fix missing * keep in and get from kwargs * revert some trailing commas * fix * fixes * video llama fixes * fix qwen3 vl * forgot glm ocr * address comments * drop unnecessary * use correct flash attn check * missed deprecation * empty commit 1 * empty commit 2 * revert video llama 3 config changes * style * style fix * address comments * remove unnecessary * revert TransformersKwargs and add a todo --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> Co-authored-by: Copilot Autofix powered by AI <[email protected]>
1 parent 04850b3 commit eb2617a

41 files changed

Lines changed: 1856 additions & 2219 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎src/transformers/models/ernie4_5_vl_moe/modeling_ernie4_5_vl_moe.py‎

Lines changed: 33 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
# limitations under the License.
2020

2121
import itertools
22+
import warnings
2223
from collections.abc import Callable
2324
from typing import Any, Optional
2425

@@ -39,8 +40,14 @@
3940
from ...modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
4041
from ...processing_utils import Unpack
4142
from ...utils import TransformersKwargs, auto_docstring, can_return_tuple, logging, torch_compilable_check
42-
from ...utils.generic import is_flash_attention_requested, maybe_autocast, merge_with_config_defaults
43+
from ...utils.generic import (
44+
accepts_precomputed_kwargs,
45+
is_flash_attention_requested,
46+
maybe_autocast,
47+
merge_with_config_defaults,
48+
)
4349
from ...utils.output_capturing import OutputRecorder, capture_outputs
50+
from ...vision_utils import get_vision_cu_seqlens, get_vision_position_ids
4451
from .configuration_ernie4_5_vl_moe import Ernie4_5_VLMoeConfig, Ernie4_5_VLMoeTextConfig, Ernie4_5_VLMoeVisionConfig
4552

4653

@@ -855,10 +862,8 @@ def __init__(self, dim: int, theta: float = 10000.0) -> None:
855862
inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float) / dim))
856863
self.register_buffer("inv_freq", inv_freq, persistent=False)
857864

858-
def forward(self, seqlen: int) -> torch.Tensor:
859-
seq = torch.arange(seqlen, device=self.inv_freq.device, dtype=self.inv_freq.dtype)
860-
freqs = torch.outer(seq, self.inv_freq)
861-
return freqs
865+
def forward(self, position_ids: torch.Tensor) -> torch.Tensor:
866+
return (position_ids.unsqueeze(-1) * self.inv_freq).flatten(1)
862867

863868

864869
@auto_docstring
@@ -894,32 +899,13 @@ def __init__(self, config) -> None:
894899
self.post_init()
895900

896901
def rot_pos_emb(self, grid_thw):
897-
pos_ids = []
898-
for t, h, w in grid_thw:
899-
hpos_ids = torch.arange(h).unsqueeze(1).expand(-1, w)
900-
hpos_ids = hpos_ids.reshape(
901-
h // self.spatial_merge_size,
902-
self.spatial_merge_size,
903-
w // self.spatial_merge_size,
904-
self.spatial_merge_size,
905-
)
906-
hpos_ids = hpos_ids.permute(0, 2, 1, 3)
907-
hpos_ids = hpos_ids.flatten()
908-
909-
wpos_ids = torch.arange(w).unsqueeze(0).expand(h, -1)
910-
wpos_ids = wpos_ids.reshape(
911-
h // self.spatial_merge_size,
912-
self.spatial_merge_size,
913-
w // self.spatial_merge_size,
914-
self.spatial_merge_size,
915-
)
916-
wpos_ids = wpos_ids.permute(0, 2, 1, 3)
917-
wpos_ids = wpos_ids.flatten()
918-
pos_ids.append(torch.stack([hpos_ids, wpos_ids], dim=-1).repeat(t, 1))
919-
pos_ids = torch.cat(pos_ids, dim=0)
920-
max_grid_size = grid_thw[:, 1:].max()
921-
rotary_pos_emb_full = self.rotary_pos_emb(max_grid_size)
922-
rotary_pos_emb = rotary_pos_emb_full[pos_ids].flatten(1)
902+
warnings.warn(
903+
f"`{self.__class__.__name__}.rot_pos_emb` is deprecated and will be removed in v5.11. Use `get_vision_position_ids` from `transformers.vision_utils` and apply the rotary embedding module.",
904+
FutureWarning,
905+
stacklevel=2,
906+
)
907+
position_ids = get_vision_position_ids(grid_thw, self.spatial_merge_size)
908+
rotary_pos_emb = self.rotary_pos_emb(position_ids)
923909
return rotary_pos_emb
924910

925911
@merge_with_config_defaults
@@ -931,21 +917,14 @@ def forward(
931917
grid_thw (`torch.LongTensor` of shape `(num_images, 3)`):
932918
The temporal, height and width dimensions of feature shape for each image. Each row contains [t, h, w] values.
933919
"""
920+
position_ids = get_vision_position_ids(grid_thw, self.spatial_merge_size, kwargs=kwargs)
921+
cu_seqlens = get_vision_cu_seqlens(grid_thw, kwargs=kwargs)
922+
934923
hidden_states = self.patch_embed(hidden_states)
935-
rotary_pos_emb = self.rot_pos_emb(grid_thw)
924+
rotary_pos_emb = self.rotary_pos_emb(position_ids)
936925
emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
937926
position_embeddings = (emb.cos(), emb.sin())
938927

939-
cu_seqlens = torch.repeat_interleave(grid_thw[:, 1] * grid_thw[:, 2], grid_thw[:, 0]).cumsum(
940-
dim=0,
941-
# Select dtype based on the following factors:
942-
# - FA2 requires that cu_seqlens_q must have dtype int32
943-
# - torch.onnx.export requires that cu_seqlens_q must have same dtype as grid_thw
944-
# See https://github.com/huggingface/transformers/pull/34852 for more information
945-
dtype=grid_thw.dtype if torch.jit.is_tracing() else torch.int32,
946-
)
947-
cu_seqlens = F.pad(cu_seqlens, (1, 0), value=0)
948-
949928
for block in self.blocks:
950929
hidden_states = block(
951930
hidden_states,
@@ -1263,6 +1242,7 @@ def get_rope_index(
12631242
mrope_position_deltas = torch.tensor(mrope_position_deltas, device=input_ids.device).unsqueeze(1)
12641243
return position_ids, mrope_position_deltas
12651244

1245+
@accepts_precomputed_kwargs(modality="video")
12661246
@can_return_tuple
12671247
@auto_docstring
12681248
def get_video_features(
@@ -1277,7 +1257,7 @@ def get_video_features(
12771257
video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*):
12781258
The temporal, height and width of feature shape of each video in LLM.
12791259
"""
1280-
video_outputs = self.vision_tower(pixel_values_videos, video_grid_thw, return_dict=True, **kwargs)
1260+
video_outputs = self.vision_tower(pixel_values_videos, video_grid_thw, **kwargs)
12811261
video_embeds = self.resampler_model(video_outputs.last_hidden_state, video_grid_thw)
12821262
split_sizes = (
12831263
video_grid_thw.prod(-1)
@@ -1288,6 +1268,7 @@ def get_video_features(
12881268
video_outputs.pooler_output = video_embeds
12891269
return video_outputs
12901270

1271+
@accepts_precomputed_kwargs(modality="image")
12911272
@can_return_tuple
12921273
@auto_docstring
12931274
def get_image_features(
@@ -1302,7 +1283,7 @@ def get_image_features(
13021283
image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*):
13031284
The temporal, height and width of feature shape of each image in LLM.
13041285
"""
1305-
image_outputs = self.vision_tower(pixel_values, image_grid_thw, return_dict=True, **kwargs)
1286+
image_outputs = self.vision_tower(pixel_values, image_grid_thw, **kwargs)
13061287
image_embeds = self.resampler_model(image_outputs.last_hidden_state, image_grid_thw)
13071288
split_sizes = (image_grid_thw.prod(-1) // self.vision_tower.spatial_merge_size**2).tolist()
13081289
image_embeds = torch.split(image_embeds, split_sizes)
@@ -1434,15 +1415,19 @@ def forward(
14341415
inputs_embeds = self.get_input_embeddings()(input_ids)
14351416

14361417
if pixel_values is not None:
1437-
image_embeds = self.get_image_features(pixel_values, image_grid_thw, return_dict=True).pooler_output
1418+
image_embeds = self.get_image_features(
1419+
pixel_values, image_grid_thw, return_dict=True, **kwargs
1420+
).pooler_output
14381421
image_embeds = torch.cat(image_embeds, dim=0).to(inputs_embeds.device, inputs_embeds.dtype)
14391422
image_mask, _ = self.get_placeholder_mask(
14401423
input_ids, inputs_embeds=inputs_embeds, image_features=image_embeds
14411424
)
14421425
inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)
14431426

14441427
if pixel_values_videos is not None:
1445-
video_embeds = self.get_video_features(pixel_values_videos, video_grid_thw, return_dict=True).pooler_output
1428+
video_embeds = self.get_video_features(
1429+
pixel_values_videos, video_grid_thw, return_dict=True, **kwargs
1430+
).pooler_output
14461431
video_embeds = torch.cat(video_embeds, dim=0).to(inputs_embeds.device, inputs_embeds.dtype)
14471432
_, video_mask = self.get_placeholder_mask(
14481433
input_ids, inputs_embeds=inputs_embeds, video_features=video_embeds
@@ -1598,9 +1583,7 @@ def get_video_features(
15981583
video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*):
15991584
The temporal, height and width of feature shape of each video in LLM.
16001585
"""
1601-
return self.model.get_video_features(
1602-
pixel_values_videos=pixel_values_videos, video_grid_thw=video_grid_thw, **kwargs
1603-
)
1586+
return self.model.get_video_features(pixel_values_videos, video_grid_thw, **kwargs)
16041587

16051588
@auto_docstring
16061589
def get_image_features(
@@ -1615,7 +1598,7 @@ def get_image_features(
16151598
image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*):
16161599
The temporal, height and width of feature shape of each image in LLM.
16171600
"""
1618-
return self.model.get_image_features(pixel_values=pixel_values, image_grid_thw=image_grid_thw, **kwargs)
1601+
return self.model.get_image_features(pixel_values, image_grid_thw, **kwargs)
16191602

16201603
@auto_docstring
16211604
@can_return_tuple

‎src/transformers/models/ernie4_5_vl_moe/modular_ernie4_5_vl_moe.py‎

Lines changed: 16 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@
2020
import numpy as np
2121
import torch
2222
import torch.nn as nn
23-
import torch.nn.functional as F
2423
from huggingface_hub.dataclasses import strict
2524
from torchvision.transforms.v2 import functional as tvF
2625

@@ -51,8 +50,9 @@
5150
can_return_tuple,
5251
logging,
5352
)
54-
from ...utils.generic import maybe_autocast, merge_with_config_defaults
53+
from ...utils.generic import accepts_precomputed_kwargs, maybe_autocast, merge_with_config_defaults
5554
from ...utils.output_capturing import OutputRecorder, capture_outputs
55+
from ...vision_utils import get_vision_cu_seqlens, get_vision_position_ids
5656
from ..ernie4_5_moe.configuration_ernie4_5_moe import Ernie4_5_MoeConfig
5757
from ..ernie4_5_moe.modeling_ernie4_5_moe import (
5858
Ernie4_5_MoeAttention,
@@ -701,21 +701,14 @@ def get_device(self):
701701
def forward(
702702
self, hidden_states: torch.Tensor, grid_thw: torch.Tensor, **kwargs: Unpack[TransformersKwargs]
703703
) -> tuple | BaseModelOutputWithPooling:
704+
position_ids = get_vision_position_ids(grid_thw, self.spatial_merge_size, kwargs=kwargs)
705+
cu_seqlens = get_vision_cu_seqlens(grid_thw, kwargs=kwargs)
706+
704707
hidden_states = self.patch_embed(hidden_states)
705-
rotary_pos_emb = self.rot_pos_emb(grid_thw)
708+
rotary_pos_emb = self.rotary_pos_emb(position_ids)
706709
emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
707710
position_embeddings = (emb.cos(), emb.sin())
708711

709-
cu_seqlens = torch.repeat_interleave(grid_thw[:, 1] * grid_thw[:, 2], grid_thw[:, 0]).cumsum(
710-
dim=0,
711-
# Select dtype based on the following factors:
712-
# - FA2 requires that cu_seqlens_q must have dtype int32
713-
# - torch.onnx.export requires that cu_seqlens_q must have same dtype as grid_thw
714-
# See https://github.com/huggingface/transformers/pull/34852 for more information
715-
dtype=grid_thw.dtype if torch.jit.is_tracing() else torch.int32,
716-
)
717-
cu_seqlens = F.pad(cu_seqlens, (1, 0), value=0)
718-
719712
for block in self.blocks:
720713
hidden_states = block(
721714
hidden_states,
@@ -962,6 +955,7 @@ def get_rope_index(
962955
mrope_position_deltas = torch.tensor(mrope_position_deltas, device=input_ids.device).unsqueeze(1)
963956
return position_ids, mrope_position_deltas
964957

958+
@accepts_precomputed_kwargs(modality="video")
965959
@can_return_tuple
966960
@auto_docstring
967961
def get_video_features(
@@ -970,7 +964,7 @@ def get_video_features(
970964
video_grid_thw: torch.LongTensor | None = None,
971965
**kwargs: Unpack[TransformersKwargs],
972966
) -> tuple | BaseModelOutputWithPooling:
973-
video_outputs = self.vision_tower(pixel_values_videos, video_grid_thw, return_dict=True, **kwargs)
967+
video_outputs = self.vision_tower(pixel_values_videos, video_grid_thw, **kwargs)
974968
video_embeds = self.resampler_model(video_outputs.last_hidden_state, video_grid_thw)
975969
split_sizes = (
976970
video_grid_thw.prod(-1)
@@ -981,6 +975,7 @@ def get_video_features(
981975
video_outputs.pooler_output = video_embeds
982976
return video_outputs
983977

978+
@accepts_precomputed_kwargs(modality="image")
984979
@can_return_tuple
985980
@auto_docstring
986981
def get_image_features(
@@ -989,7 +984,7 @@ def get_image_features(
989984
image_grid_thw: torch.LongTensor | None = None,
990985
**kwargs: Unpack[TransformersKwargs],
991986
) -> tuple | BaseModelOutputWithPooling:
992-
image_outputs = self.vision_tower(pixel_values, image_grid_thw, return_dict=True, **kwargs)
987+
image_outputs = self.vision_tower(pixel_values, image_grid_thw, **kwargs)
993988
image_embeds = self.resampler_model(image_outputs.last_hidden_state, image_grid_thw)
994989
split_sizes = (image_grid_thw.prod(-1) // self.vision_tower.spatial_merge_size**2).tolist()
995990
image_embeds = torch.split(image_embeds, split_sizes)
@@ -1031,15 +1026,19 @@ def forward(
10311026
inputs_embeds = self.get_input_embeddings()(input_ids)
10321027

10331028
if pixel_values is not None:
1034-
image_embeds = self.get_image_features(pixel_values, image_grid_thw, return_dict=True).pooler_output
1029+
image_embeds = self.get_image_features(
1030+
pixel_values, image_grid_thw, return_dict=True, **kwargs
1031+
).pooler_output
10351032
image_embeds = torch.cat(image_embeds, dim=0).to(inputs_embeds.device, inputs_embeds.dtype)
10361033
image_mask, _ = self.get_placeholder_mask(
10371034
input_ids, inputs_embeds=inputs_embeds, image_features=image_embeds
10381035
)
10391036
inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)
10401037

10411038
if pixel_values_videos is not None:
1042-
video_embeds = self.get_video_features(pixel_values_videos, video_grid_thw, return_dict=True).pooler_output
1039+
video_embeds = self.get_video_features(
1040+
pixel_values_videos, video_grid_thw, return_dict=True, **kwargs
1041+
).pooler_output
10431042
video_embeds = torch.cat(video_embeds, dim=0).to(inputs_embeds.device, inputs_embeds.dtype)
10441043
_, video_mask = self.get_placeholder_mask(
10451044
input_ids, inputs_embeds=inputs_embeds, video_features=video_embeds

0 commit comments

Comments
 (0)