🚨 Refactor ViT to updated standards#41693
Conversation
|
Cc @vasqu for attention mask! |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
vasqu
left a comment
There was a problem hiding this comment.
Looking already pretty good to me, I'm just really unsure if we want to have the conversion mapping; would wait for the others on their opinion here
| self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.qkv_bias) | ||
| self.k_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.qkv_bias) | ||
| self.v_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.qkv_bias) | ||
| self.o_proj = nn.Linear(config.num_attention_heads * self.head_dim, config.hidden_size, bias=config.qkv_bias) |
There was a problem hiding this comment.
There is a lot of renaming here (+ the additional o_proj). Are you sure that we are not breaking already existing models with this?
There was a problem hiding this comment.
Just seen the conversion below, a bit unsure if we want to do this
| return context_layer, attention_probs | ||
| attn_output = attn_output.reshape(*input_shape, -1).contiguous() | ||
| attn_output = self.o_proj(attn_output) | ||
| attn_output = nn.functional.dropout(attn_output, p=self.hidden_dropout, training=self.training) |
There was a problem hiding this comment.
This dropout is new no? Shouldnt exist imo
There was a problem hiding this comment.
Ah looks like it's from the separated module before, nvm - would maybe move this outside the attention tho 🤔 more of a nit
There was a problem hiding this comment.
Yes it's kind of unusual to have a dropout here, I would prefer to just get rid of it tbh 😅. I saw that SeedOssAttention also use a dropout here in the attention module, but I'm happy to move it outside if we think it makes more sense
There was a problem hiding this comment.
Depends on models that could inherit from ViT then. I don't mind it too much but it might be easier to have the dropout outside; maybe something to look out for later when you work on other models that are dependent on it
There was a problem hiding this comment.
I'm pro-setting it outside!
| _checkpoint_conversion_mapping = { | ||
| "attention.query": "q_proj", | ||
| "attention.key": "k_proj", | ||
| "attention.value": "v_proj", | ||
| "attention.output.dense": "attention.o_proj", | ||
| "intermediate.dense": "mlp.fc1", | ||
| "output.dense": "mlp.fc2", | ||
| } |
There was a problem hiding this comment.
Ah, I see so we rename stuff on the fly. I'm unsure about this would like an opinion from @zucchini-nlp or @molbap here
There was a problem hiding this comment.
+1, I don't think we should be using key-mapping unless there is a strong reason for it. It usually can cause weird side-effects iwith 3rd party libraries, adapters, trainer classes.
If some frameworks do not use our weight loading, and overwrite it, it'll also raise issues. It was the case for vLLM but we copied the same conversion mapping so the models can still be loaded correctly
There was a problem hiding this comment.
Ah I see, that's annoying. I still think this would be worth it in the long run, we have too many disparities in our vision models implementation, it's hard to understand which models/modules are different or the same as each other. + this would be the occasion to standardize how we write models in transformers between modalities. Right now it seems that vision models have their separate way of implementing attention, mlp, positional embeddings etc. when we could use the same or similar implementations as in language models.
There was a problem hiding this comment.
Both sides make sense. Changing the keys here creates a difference between the json (external API) and the signatures (internal API), which breaks the Consistent Public Surface we want (and can create issues with external adapters). However, as Yoni says the current implementation is very far from being Standard across the lib, so there's a tension between these two principles.
What I suggest is we go the other way around: I DO want to standardize these keys, but the reason is that the implementations are so wildly different and hard to read. Hence, let's first map the implementations to what exists in language models, and then we have a stronger motive to do on-the-fly conversion.
TL;DR let's move that to another PR and update implementations, then we can change the keys.
|
Thanks for the review!
It seems reasonable for v5 imo, and will be well worth the cost in the long run. + @ArthurZucker and @Cyrilvallez are working on a way to make weights conversion on the fly much faster |
|
Gotcha, I'd still wait for the opinion of others here but it would indeed make things much easier! |
molbap
left a comment
There was a problem hiding this comment.
daang I had published my review yesterday but it disappeared 😩
| class BeitPatchEmbeddings(ViTPatchEmbeddings): | ||
| def __init__(self, config): | ||
| super().__init__() | ||
| image_size, patch_size = config.image_size, config.patch_size | ||
|
|
||
| image_size = image_size if isinstance(image_size, collections.abc.Iterable) else (image_size, image_size) | ||
| patch_size = patch_size if isinstance(patch_size, collections.abc.Iterable) else (patch_size, patch_size) | ||
| patch_shape = (image_size[0] // patch_size[0], image_size[1] // patch_size[1]) | ||
| self.patch_shape = patch_shape | ||
|
|
||
| def forward(self, pixel_values: torch.Tensor) -> torch.Tensor: | ||
| num_channels = pixel_values.shape[1] | ||
| if num_channels != self.num_channels: | ||
| raise ValueError( | ||
| "Make sure that the channel dimension of the pixel values match with the one set in the configuration." | ||
| ) | ||
|
|
||
| embeddings = self.projection(pixel_values).flatten(2).transpose(1, 2) | ||
|
|
||
| return embeddings |
There was a problem hiding this comment.
There's like 10 other models with almost identical PatchEmbeddings, that's good, but could we inherit directly the ViT one? It's too bad to have to redefine because of a raise, I'd be in favor of removing it
| if not interpolate_pos_encoding: | ||
| if height != self.image_size[0] or width != self.image_size[1]: | ||
| raise ValueError( | ||
| f"Input image size ({height}*{width}) doesn't match model" | ||
| f" ({self.image_size[0]}*{self.image_size[1]})." | ||
| ) |
There was a problem hiding this comment.
if this goes away or is under an Optional and not a hard check, it's easier to inherit
| return context_layer, attention_probs | ||
| attn_output = attn_output.reshape(*input_shape, -1).contiguous() | ||
| attn_output = self.o_proj(attn_output) | ||
| attn_output = nn.functional.dropout(attn_output, p=self.hidden_dropout, training=self.training) |
There was a problem hiding this comment.
I'm pro-setting it outside!
| _checkpoint_conversion_mapping = { | ||
| "attention.query": "q_proj", | ||
| "attention.key": "k_proj", | ||
| "attention.value": "v_proj", | ||
| "attention.output.dense": "attention.o_proj", | ||
| "intermediate.dense": "mlp.fc1", | ||
| "output.dense": "mlp.fc2", | ||
| } |
There was a problem hiding this comment.
Both sides make sense. Changing the keys here creates a difference between the json (external API) and the signatures (internal API), which breaks the Consistent Public Surface we want (and can create issues with external adapters). However, as Yoni says the current implementation is very far from being Standard across the lib, so there's a tension between these two principles.
What I suggest is we go the other way around: I DO want to standardize these keys, but the reason is that the implementations are so wildly different and hard to read. Hence, let's first map the implementations to what exists in language models, and then we have a stronger motive to do on-the-fly conversion.
TL;DR let's move that to another PR and update implementations, then we can change the keys.
|
I think we can revisit this now @yonigozlan (given the weight converter is now merged into main), oh and #41549 as well |
…ied from and add todos
|
run-slow: aimv2, audio_spectrogram_transformer, beit, bit, clap, conditional_detr, convnext, convnextv2, cvt, d_fine, dab_detr, data2vec, deit, dinat, dinov2, dinov2_with_registers, dinov3_convnext, dinov3_vit, donut, dpt, eomt, eomt_dinov3, flava, florence2, focalnet, glpn, grounding_dino, hiera, ijepa, lw_detr, maskformer, mgp_str, mm_grounding_dino, pixio, poolformer, pp_doclayout_v2, pp_doclayout_v3, pp_lcnet, pp_lcnet_v3, pp_ocrv5_mobile_rec, pp_ocrv5_server_rec, pvt, pvt_v2, qianfan_ocr, resnet, rt_detr, rt_detr_v2, segformer, seggpt, slanet, swiftformer, swin, swin2sr, swinv2, timesformer, uvdoc, videomae, videomt, vilt, vit, vit_mae, vit_msn, vitdet, vitpose_backbone, vivit, vjepa2, x_clip, yolos |
|
This comment contains models: ["models/aimv2", "models/audio_spectrogram_transformer", "models/beit", "models/bit", "models/clap", "models/conditional_detr", "models/convnext", "models/convnextv2", "models/cvt", "models/d_fine", "models/dab_detr", "models/data2vec", "models/deit", "models/dinat", "models/dinov2", "models/dinov2_with_registers", "models/dinov3_convnext", "models/dinov3_vit", "models/donut", "models/dpt", "models/eomt", "models/eomt_dinov3", "models/flava", "models/florence2", "models/focalnet", "models/glpn", "models/grounding_dino", "models/hiera", "models/ijepa", "models/lw_detr", "models/maskformer", "models/mgp_str", "models/mm_grounding_dino", "models/pixio", "models/poolformer", "models/pp_doclayout_v2", "models/pp_doclayout_v3", "models/pp_lcnet", "models/pp_lcnet_v3", "models/pp_ocrv5_mobile_rec", "models/pp_ocrv5_server_rec", "models/pvt", "models/pvt_v2", "models/qianfan_ocr", "models/resnet", "models/rt_detr", "models/rt_detr_v2", "models/segformer", "models/seggpt", "models/slanet", "models/swiftformer", "models/swin", "models/swin2sr", "models/swinv2", "models/timesformer", "models/uvdoc", "models/videomae", "models/videomt", "models/vilt", "models/vit", "models/vit_mae", "models/vit_msn", "models/vitdet", "models/vitpose_backbone", "models/vivit", "models/vjepa2", "models/x_clip", "models/yolos"] |
CI ResultsCommit Info
Model CI Report❌ 181 new failed tests from this PR 😭
|
|
run-slow: aimv2, audio_spectrogram_transformer, beit, bit, clap, conditional_detr, convnext, convnextv2, cvt, d_fine, dab_detr, data2vec, deit, dinat, dinov2, dinov2_with_registers, dinov3_convnext, dinov3_vit, donut, dpt, eomt, eomt_dinov3, flava, florence2, focalnet, glpn, grounding_dino, hiera, ijepa, lw_detr, maskformer, mgp_str, mm_grounding_dino, pixio, poolformer, pp_doclayout_v2, pp_doclayout_v3, pp_lcnet, pp_lcnet_v3, pp_ocrv5_mobile_rec, pp_ocrv5_server_rec, pvt, pvt_v2, qianfan_ocr, resnet, rt_detr, rt_detr_v2, segformer, seggpt, slanet, swiftformer, swin, swin2sr, swinv2, timesformer, uvdoc, videomae, videomt, vilt, vit, vit_mae, vit_msn, vitdet, vitpose_backbone, vivit, vjepa2, x_clip, yolos |
|
This comment contains models: ["models/aimv2", "models/audio_spectrogram_transformer", "models/beit", "models/bit", "models/clap", "models/conditional_detr", "models/convnext", "models/convnextv2", "models/cvt", "models/d_fine", "models/dab_detr", "models/data2vec", "models/deit", "models/dinat", "models/dinov2", "models/dinov2_with_registers", "models/dinov3_convnext", "models/dinov3_vit", "models/donut", "models/dpt", "models/eomt", "models/eomt_dinov3", "models/flava", "models/florence2", "models/focalnet", "models/glpn", "models/grounding_dino", "models/hiera", "models/ijepa", "models/lw_detr", "models/maskformer", "models/mgp_str", "models/mm_grounding_dino", "models/pixio", "models/poolformer", "models/pp_doclayout_v2", "models/pp_doclayout_v3", "models/pp_lcnet", "models/pp_lcnet_v3", "models/pp_ocrv5_mobile_rec", "models/pp_ocrv5_server_rec", "models/pvt", "models/pvt_v2", "models/qianfan_ocr", "models/resnet", "models/rt_detr", "models/rt_detr_v2", "models/segformer", "models/seggpt", "models/slanet", "models/swiftformer", "models/swin", "models/swin2sr", "models/swinv2", "models/timesformer", "models/uvdoc", "models/videomae", "models/videomt", "models/vilt", "models/vit", "models/vit_mae", "models/vit_msn", "models/vitdet", "models/vitpose_backbone", "models/vivit", "models/vjepa2", "models/x_clip", "models/yolos"] |
CI ResultsCommit Info
Model CI Report❌ 3 new failed tests from this PR 😭
|
|
[For maintainers] Suggested jobs to run (before merge) run-slow: aimv2, audio_spectrogram_transformer, beit, bit, clap, conditional_detr, convnext, convnextv2, cvt, d_fine, dab_detr, data2vec, deimv2, deit, dinat, dinov2 |
|
View the CircleCI Test Summary for this PR: https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41693&sha=2d424a |
* refactor attention * use mlp instead of intermediate and output * use modular for deit * nit * fix deit (to check) and refactor beit * use modular for beit * style * refactor vit deit beit segformer * add pos_embed_utils, refactor swin * Fix after merge + refactor pixio * refactor most drop_path in transformers * refactor vit_msn * fix modular * refactor vivit and audio_spectrogram_transformer, remove obsolete copied from and add todos * Fix repo * fix nit * Fix drop_path_schedule * nit * use modular for pos embed * reverse DropPath as a util * add conversion ijepa + fix offload vit_mae * fix modular * fix qkv bias in output proj + swin attention per stage * address review comments * fix swinn * improve swin and beit * last fixes after review * standardize backbone beit * fix beit * fix beit * fix repo * changes after review * Fix beit * fix after review * fix unexpected key swin * Address review * fix repo * Fix after review * fix modular * fix depth pro * Fix conversion mappings * fix style * Fix failing tests * fix ci * Fix two slow failing tests
What does this PR do?
This PR aims at refactoring ViT as part of an effort to standardize vision models in the library, similarly to #41549
Vision Models Refactoring - Chronological Todo List
vit(2021-04-01) 🚨 Refactor ViT to updated standards #41693deit(2021-04-13) 🚨 Refactor ViT to updated standards #41693detr(2021-06-09) 🚨 Refactor DETR to updated standards #41549beit(2021-08-04) 🚨 Refactor ViT to updated standards #41693segformer(2021-10-28) 🚨 Refactor ViT to updated standards #41693imagegpt(2021-11-18)vit_mae(2022-01-18) 🚨 Refactor ViT to updated standards #41693swin(2022-01-21) 🚨 Refactor ViT to updated standards #41693convnext(2022-02-07)poolformer(2022-02-17)maskformer(2022-03-02)dit(2022-03-10)resnet(2022-03-14)glpn(2022-03-22)dpt(2022-03-28)regnet(2022-04-07)yolos(2022-05-02)cvt(2022-05-18)levit(2022-06-01)mobilevit(2022-06-29)swinv2(2022-07-27)deformable_detr(2022-09-14) 🚨 Refactor DETR to updated standards #41549conditional_detr(2022-09-22) 🚨 Refactor DETR to updated standards #41549vit_msn(2022-09-22) 🚨 Refactor ViT to updated standards #41693table-transformer(2022-10-18)mobilenet_v2(2022-11-14)dinat(2022-11-18)mobilenet_v1(2022-11-21)bit(2022-12-07)swin2sr(2022-12-16)mask2former(2023-01-16)upernet(2023-01-16)efficientnet(2023-02-20)convnextv2(2023-03-14)sam(2023-04-19)focalnet(2023-04-23)swiftformer(2023-05-12)mobilevitv2(2023-06-02)dinov2(2023-07-18)pvt(2023-07-24)vitdet(2023-08-29)vitmatte(2023-09-19)depth_anything(2024-01-25)seggpt(2024-02-26)pvt_v2(2024-03-13)superpoint(2024-03-19)rt_detr(2024-06-22) 🚨 Refactor DETR to updated standards #41549depth_anything_v2(2024-07-05)zoedepth(2024-07-08)hiera(2024-07-12)ijepa(2024-12-05) 🚨 Refactor ViT to updated standards #41693dinov2_with_registers(2024-12-24)textnet(2025-01-08)vitpose(2025-01-08)superglue(2025-01-20)dab-detr(2025-02-04)rt_detr_v2(2025-02-06) 🚨 Refactor DETR to updated standards #41549depth_pro(2025-02-10)prompt_depth_anything(2025-03-21)mlcd(2025-04-15)sam_hq(2025-04-28)d_fine(2025-04-29) 🚨 Refactor DETR to updated standards #41549hgnet_v2(2025-04-29)lightglue(2025-06-17)eomt(2025-06-27)aimv2(2025-07-08)efficientloftr(2025-07-22)dinov3(2025-08-14)sam2(2025-08-14)sam3_tracker(2025-11-19)pixio(2025-12-16) 🚨 Refactor ViT to updated standards #41693lw_detr(2026-01-12)eomt_dinov3(2026-02-01)chmv2(2026-03-11)Total: 74 vision models
Additional multimodal/other modalities refactor:
audio_spectrogram_transformer🚨 Refactor ViT to updated standards #41693vivit🚨 Refactor ViT to updated standards #41693pp_doclayout_v3🚨 Refactor DETR to updated standards #41549