Codestin Search App

Leon0402 · 2025-07-20T07:42:17Z

This a rebase based on the original PR: #22289

Potentially Missing:

Using Scaled Dot Product Attention (SDPA)
BeitImageProcessorFast Support
Beit3Backbone

Leon0402 · 2025-07-20T12:52:00Z

@NielsRogge Would you be willing to support me in this second attempt (#22289) to finally bring BEIT3 support to HF?

Leon0402 · 2025-07-21T19:50:42Z

+
+
+def get_tokenizer():
+    return XLMRobertaTokenizer.from_pretrained("https://github.com/addf400/files/releases/download/beit3/beit3.spm")


Is there any way how I can convert this directly in a XMLRobertaTokenizerFast? I know I can use the old one here and then when loading the tokenizer from the hf checkpoint it can be converted on the fly.
But I think it would be preferable to convert it directly here.

Also it seems that model_max_length of the tokenizer / preprocessor is not set automatically as typically with hf models. Why is that the case? Should that not be stored in the beit3.spm or do I need to set that somewhere explicitly? For instance, I couldn't find where the VILT model sets this explicitly, thus I was assuming that huggingface can get it automatically from somewhere

Leon0402 · 2025-07-21T19:59:44Z

+    parser.add_argument(
+        "--beit3_model_type",
+        default=None,
+        type=str,
+        help="Beit3 model type, it has to be one of image_classification, vqa, visual_reasoning,"
+        "image_captioning,image_text_retrieval",
+    )


One thing I am overall wondering is: What is with the "base pretrained" BEiT3 image? I think with BEiT3 it is common to finetune the whole model for downstream tasks like vqa or image captioning. Thus, I think it makes sense to provide the base model too and not only the finetuned versions.

But technically the base model of course also has a head. It would probably be something like Beit3ForMaskedImageModelling. Even though it is not advertised like this. So what would be best practice here? Load the standard Beit3Model without any head and ignore certain parameters? This is what I tried locally:

Modify lines 277ff:

model_state_dict = ulilm_state_dict["model"] model_state_dict = rename_keys(model_state_dict, beit3_model_type) if isinstance(model, Beit3Model): # For Beit3Model, we need filter beit3 keys and remove them model_state_dict = {k.replace("beit3.", ""): v for k, v in model_state_dict.items() if k.startswith("beit3.")} model.load_state_dict(model_state_dict)

Is this okay? What would be best practise?

See newest commits

Leon0402 · 2025-07-22T09:10:21Z

+class Beit3MultiwayFeedForwardNetwork(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.text = Beit3FeedForwardNetwork(config)
+        self.image = Beit3FeedForwardNetwork(config)
+
+    def forward(self, hidden_states: torch.Tensor, split_position: int = -1):
+        if split_position == -1:
+            return self.image(hidden_states)
+        if split_position == 0:
+            return self.text(hidden_states)
+        image_hidden, text_hidden = torch.split(
+            hidden_states,
+            [split_position, hidden_states.size(1) - split_position],
+            dim=1,
+        )
+        image_out, text_out = self.image(image_hidden), self.text(text_hidden)
+        return torch.cat([image_out, text_out], dim=1)


Would it make sense to create a parent class MultiwayLayer that takes two networks for text and image?
I think this logic is 2 or 3 three times duplicated here.

I think that would be great, yeah!
Adding a level of abstraction here would make the code shorter while maintaining readability.

Leon0402 · 2025-07-23T06:34:18Z

class Beit3ForQuestionAnswering(Beit3PreTrainedModel):
    def __init__(self, config):
        super().__init__(config)
        embed_dim = config.hidden_size
        self.num_labels = config.num_labels
        self.beit3 = Beit3Model(config, add_pooling_layer=True)

        self.classifier = nn.Sequential(
            nn.Linear(embed_dim, embed_dim * 2),
            nn.LayerNorm(embed_dim * 2, eps=config.layer_norm_eps),
            nn.GELU(),
            nn.Linear(embed_dim * 2, config.num_labels),
        )
        self.post_init()

    def load_state_dict(self, state_dict, *args, **kwargs):
        pos_embed_checkpoint = state_dict["beit3.encoder.embed_positions.image.weight"]
        embedding_size = pos_embed_checkpoint.shape[-1]

        num_patches = self.beit3.vision_embedding.num_patches
        num_extra_tokens = self.beit3.vision_embedding.num_position_embeddings() + 2 - num_patches

        orig_size = int((pos_embed_checkpoint.shape[-2] - num_extra_tokens) ** 0.5)
        new_size = int(num_patches**0.5)

        if orig_size != new_size:
            extra_tokens = pos_embed_checkpoint[:num_extra_tokens].unsqueeze(0)
            pos_tokens = pos_embed_checkpoint[num_extra_tokens:]
            pos_tokens = pos_tokens.reshape(1, orig_size, orig_size, embedding_size).permute(0, 3, 1, 2)
            pos_tokens = torch.nn.functional.interpolate(
                pos_tokens, size=(new_size, new_size), mode="bicubic", align_corners=False
            )
            pos_tokens = pos_tokens.permute(0, 2, 3, 1).reshape(1, new_size * new_size, embedding_size)
            new_pos_embed = torch.cat((extra_tokens, pos_tokens), dim=1).squeeze(0)
            state_dict["beit3.encoder.embed_positions.image.weight"] = new_pos_embed

        return super().load_state_dict(state_dict, *args, **kwargs)

@NielsRogge Could you help me here. I need something along the lines of this in the code. If someone wants to process larger or smaller images, instead of reinitializing the weights, this will interpolate the new embedding weights. It is done here https://github.com/microsoft/unilm/blob/master/beit3/utils.py#L521.
I believe this should be part of the HF model and not left to the user. But I am not sure where to put it. It seems from_pretrained does not use load_state_dict. Is there any hook in HF that could be overridden, where I can put this?

CharlesAttend · 2025-07-28T15:24:08Z

Hello @Leon0402,

Thank you for working on this! This implementation could also be really useful for me at work.

I’d love to help, but I’m not quite sure how to contribute effectively. I could take care of the small subclassing tasks for Beit3MultiwayFeedForwardNetwork, Beit3AttentionLinearProjection, and Beit3LayerNorm if you’d like (though I realize it’s not much work).

Regarding interpolation, I noticed that it's done during the forward pass in the DeiT implementation.
Could that also be a viable approach for us?

Leon0402 · 2025-07-28T16:11:31Z

Hi @CharlesAttend,

yes, feel free to implement the abstraction for the muliway logic. Maybe the best option would be to create a PR against my branch that I can merge it, once it looks good?

Regarding interpolation logic: I think doing it in the forward pass does not work too well, because huggingface will default initialise the parameters during from_pretrained and we then do not have access anymore to the old embeddings, right?

Rocketknight1 · 2025-09-29T12:53:31Z

cc @yonigozlan @molbap maybe?

molbap

I'm down to review this, left a couple guiding comments to modernize the API!

molbap · 2025-09-29T15:34:03Z

+BEIT3_START_DOCSTRING = r"""
+    This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it
+    as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
+    behavior.
+
+    Parameters:
+        config ([`Beit3Config`]): Model configuration class with all the parameters of the model.
+            Initializing with a config file does not load the weights associated with the model, only the
+            configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights.
+"""
+
+
+BEIT3_MODEL = r"""
+    Args:
+        input_ids (`torch.LongTensor` of shape `({0})`, *optional*):
+            Indices of input sequence tokens in the vocabulary.
+
+            Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
+            [`PreTrainedTokenizer.__call__`] for details.
+
+            [What are input IDs?](../glossary#input-ids)
+        pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`, *optional*):
+            Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See
+            [`BeitImageProcessor.__call__`] for details.
+        attention_mask (`torch.LongTensor` of shape `({0})`, *optional*):
+            Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
+
+            - 1 indicates the token is **not masked**,
+            - 0 indicates the token is **masked**.
+
+            [What are attention masks?](../glossary#attention-mask)
+        image_text_mask (`torch.FloatTensor` of shape `({0})`, *optional*):
+            Mask to avoid performing attention on image-text tokens. Mask values selected in `[0, 1]`:
+
+            - 1 for tokens that are **not masked**,
+            - 0 for tokens that are **masked**.
+
+        vision_masked_position (`torch.LongTensor` of shape pixel_values, *optional*):
+            Padding mask for input tokens , of same shape as `pixel_values`
+
+            - 1 indicates the token is **not masked**,
+            - 0 indicates the token is **masked**.
+        past_key_values (`Tuple`, *optional*):
+            A Tuple containing the incremental states layerwise.This can be used to when generating next token in
+            case of image captioning.
+        output_hidden_states (`bool`, *optional*):
+            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
+            more detail.
+        output_attentions (`bool`, *optional*):
+            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
+            tensors for more detail.
+        return_dict (`bool`, *optional*):
+            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
+"""
+
+
+BEIT3_FOR_VISUAL_REASONING_INPUTS_DOCSTRING = r"""
+    Args:
+        input_ids (`torch.LongTensor` of shape `({0})`):
+            Indices of input sequence tokens in the vocabulary.
+
+            Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
+            [`PreTrainedTokenizer.__call__`] for details.
+
+            [What are input IDs?](../glossary#input-ids)
+        pixel_values (`torch.FloatTensor` of shape `(batch_size, 2, num_channels, height, width)`):
+            Pixel values. Pixel values can be obtained by combining two images after preprocessing using
+             [`AutoImageProcessor`]. See [`BeitImageProcessor.__call__`] for details. Use torch.cat with two images,
+             with dim=1.
+        attention_mask (`torch.LongTensor` of shape `({0})`,*optional*):
+            Padding mask for input tokens , of same shape as `input_ids`
+            - 1 indicates the token is **not masked**,
+            - 0 indicates the token is **masked**.
+        output_hidden_states (`bool`, *optional*):
+            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
+            more detail.
+        output_attentions (`bool`, *optional*):
+            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
+            tensors for more detail.
+        return_dict (`bool`, *optional*):
+            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
+        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
+            config.num_labels - 1]`. A classification loss is computed (Cross-Entropy) against these labels.
+"""
+
+
+BEIT3_FOR_IMAGE_CLASSIFICATION_INPUTS_DOCSTRING = r"""
+    Args:
+        pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`):
+            Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See
+            [`BeitImageProcessor.__call__`] for details.
+        output_hidden_states (`bool`, *optional*):
+            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
+            more detail.
+        output_attentions (`bool`, *optional*):
+            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
+            tensors for more detail.
+        return_dict (`bool`, *optional*):
+            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
+        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for computing the classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. A
+            classification loss is computed (Cross-Entropy) against these labels.
+"""
+
+
+BEIT3_FOR_CAPTIONING_INPUTS_DOCSTRING = r"""
+    Args:
+        input_ids (`torch.LongTensor` of shape `({0})`):
+            Indices of input sequence tokens in the vocabulary.
+
+            Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
+            [`PreTrainedTokenizer.__call__`] for details.
+
+            [What are input IDs?](../glossary#input-ids)
+        pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`):
+            Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See
+            [`BeitImageProcessor.__call__`] for details.
+        attention_mask (`torch.LongTensor` of shape `({0})`, *optional*):
+            Padding mask for input tokens , of same shape as `input_ids`
+
+            - 1 indicates the token is **not masked**,
+            - 0 indicates the token is **masked**.
+        language_masked_pos (`torch.LongTensor` of shape `({0})`, *optional*):
+            language_masked_pos for denoting tokens for captioning
+
+            - 1 indicates the token is **present**,
+            - 0 indicates the token is **absent**.
+        text_len (`torch.LongTensor` of shape `({0})`, *optional*):
+            Length of text for captioning, this is the length of the final caption to be generated, includes the
+            input_ids and tokens marked as 64001 (token id marked as to be filled).
+        past_key_values (`Tuple`, *optional*):
+            A Tuple containing the incremental states layerwise
+        output_hidden_states (`bool`, *optional*):
+            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
+            more detail.
+        output_attentions (`bool`, *optional*):
+            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
+            tensors for more detail.
+        return_dict (`bool`, *optional*):
+            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
+        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for computing the classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. A
+            classification loss is computed (Cross-Entropy) against these labels.
+"""
+
+
+BEIT3_FOR_VQA_INPUTS_DOCSTRING = r"""
+    Args:
+        input_ids (`torch.LongTensor` of shape `({0})`):
+            Indices of input sequence tokens in the vocabulary.
+
+            Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
+            [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids)
+        pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`):
+            Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See
+            [`BeitImageProcessor.__call__`] for details.
+        attention_mask (`torch.LongTensor` of shape `({0})`, *optional*):
+            Padding mask for input tokens , of same shape as `input_ids`
+
+            - 1 indicates the token is **not masked**,
+            - 0 indicates the token is **masked**.
+        output_hidden_states (`bool`, *optional*):
+            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
+            more detail.
+        output_attentions (`bool`, *optional*):
+            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
+            tensors for more detail.
+        return_dict (`bool`, *optional*):
+            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
+        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for computing the classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. A
+            classification loss is computed (Cross-Entropy) against these labels.
+"""
+
+
+BEIT3_FOR_TEXT_RETRIEVAL_INPUTS_DOCSTRING = r"""
+    Args:
+        input_ids (`torch.LongTensor` of shape `({0})`):
+            Indices of input sequence tokens in the vocabulary.
+
+            Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
+            [`PreTrainedTokenizer.__call__`] for details.
+
+            [What are input IDs?](../glossary#input-ids)
+        pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`):
+            Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See
+            [`BeitImageProcessor.__call__`] for details.
+        attention_mask (`torch.LongTensor` of shape `({0})`, *optional*):
+            Padding mask for input tokens , of same shape as `input_ids`
+
+            - 1 indicates the token is **not masked**,
+            - 0 indicates the token is **masked**.
+        return_loss (`bool`, *optional*):
+            Whether or not to return the contrastive loss.
+        output_hidden_states (`bool`, *optional*):
+            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
+            more detail.
+        output_attentions (`bool`, *optional*):
+            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
+            tensors for more detail.
+        return_dict (`bool`, *optional*):
+            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
+"""


We now have an auto_docstring decorator that should alleviate a portion of these

molbap · 2025-09-29T16:03:48Z

In terms of API changes, one big change is the addition of modular transformers: https://huggingface.co/docs/transformers/v4.45.1/en/modular_transformers
Here I'm seeing a lot of components that are very similar to existing ones, or identical (contrastive_loss for instance). What's # Copied from should be removed, and a modular_beit3.py file should be created, with the modeling_beit3.py file being automatically generated from it. Feel free to look at examples across the lib!

Leon0402 · 2025-10-05T15:14:41Z

Thanks @molbap. Do you happen to have any advice on my previous comment:

class Beit3ForQuestionAnswering(Beit3PreTrainedModel):
    def __init__(self, config):
        super().__init__(config)
        embed_dim = config.hidden_size
        self.num_labels = config.num_labels
        self.beit3 = Beit3Model(config, add_pooling_layer=True)

        self.classifier = nn.Sequential(
            nn.Linear(embed_dim, embed_dim * 2),
            nn.LayerNorm(embed_dim * 2, eps=config.layer_norm_eps),
            nn.GELU(),
            nn.Linear(embed_dim * 2, config.num_labels),
        )
        self.post_init()

    def load_state_dict(self, state_dict, *args, **kwargs):
        pos_embed_checkpoint = state_dict["beit3.encoder.embed_positions.image.weight"]
        embedding_size = pos_embed_checkpoint.shape[-1]

        num_patches = self.beit3.vision_embedding.num_patches
        num_extra_tokens = self.beit3.vision_embedding.num_position_embeddings() + 2 - num_patches

        orig_size = int((pos_embed_checkpoint.shape[-2] - num_extra_tokens) ** 0.5)
        new_size = int(num_patches**0.5)

        if orig_size != new_size:
            extra_tokens = pos_embed_checkpoint[:num_extra_tokens].unsqueeze(0)
            pos_tokens = pos_embed_checkpoint[num_extra_tokens:]
            pos_tokens = pos_tokens.reshape(1, orig_size, orig_size, embedding_size).permute(0, 3, 1, 2)
            pos_tokens = torch.nn.functional.interpolate(
                pos_tokens, size=(new_size, new_size), mode="bicubic", align_corners=False
            )
            pos_tokens = pos_tokens.permute(0, 2, 3, 1).reshape(1, new_size * new_size, embedding_size)
            new_pos_embed = torch.cat((extra_tokens, pos_tokens), dim=1).squeeze(0)
            state_dict["beit3.encoder.embed_positions.image.weight"] = new_pos_embed

        return super().load_state_dict(state_dict, *args, **kwargs)

@NielsRogge Could you help me here. I need something along the lines of this in the code. If someone wants to process larger or smaller images, instead of reinitializing the weights, this will interpolate the new embedding weights. It is done here https://github.com/microsoft/unilm/blob/master/beit3/utils.py#L521. I believe this should be part of the HF model and not left to the user. But I am not sure where to put it. It seems from_pretrained does not use load_state_dict. Is there any hook in HF that could be overridden, where I can put this?

github-actions · 2025-10-05T18:36:20Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, beit3

molbap · 2025-10-09T08:13:33Z

Hey @Leon0402, for your question: right now we don't add hooks to from_pretrained from within a modeling file. However if you check out

transformers/src/transformers/models/siglip2/modeling_siglip2.py

Line 133 in 7aa888b

def resize_positional_embeddings(

you'll find a method we previously used that resizes positional embeddings. Something similar could be ok I think.

Feel free to ping me again for another review!

Leon0402 force-pushed the feature/beit3 branch from 3608964 to f605be6 Compare July 20, 2025 07:44

Leon0402 commented Jul 21, 2025

View reviewed changes

Comment thread src/transformers/models/beit3/modeling_beit3.py

Leon0402 commented Jul 21, 2025

View reviewed changes

Leon0402 commented Jul 22, 2025

View reviewed changes

Leon0402 force-pushed the feature/beit3 branch from 77e4516 to d822a1d Compare September 28, 2025 16:12

Implement Beit3 model

18efe70

Leon0402 force-pushed the feature/beit3 branch from d822a1d to 18efe70 Compare September 28, 2025 16:16

Leon0402 marked this pull request as ready for review September 28, 2025 16:16

Leon0402 added 5 commits September 28, 2025 16:21

Fix issue in auto import

4492010

Format

c5b470b

Fix copies

7115de7

Remove accidental changes again

3bdffc4

Update init

e20cccc

molbap reviewed Sep 29, 2025

View reviewed changes

molbap mentioned this pull request Oct 2, 2025

Add modular detector #41289

Merged

Include __all__ to files

60a361a

Use autodoc

a2b7a91

evalstate mentioned this pull request Apr 29, 2026

Cumulative feature and defect updates from recent Transformers PRs evalstate/transformers#42

Open



		def get_tokenizer():
		return XLMRobertaTokenizer.from_pretrained("https://github.com/addf400/files/releases/download/beit3/beit3.spm")

Uh oh!

Conversation

Leon0402 commented Jul 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Leon0402 commented Jul 20, 2025

Uh oh!

Uh oh!

Leon0402 Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

Leon0402 Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Leon0402 Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

Leon0402 Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

Leon0402 Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CharlesAttend Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Leon0402 commented Jul 23, 2025

Uh oh!

CharlesAttend commented Jul 28, 2025

Uh oh!

Leon0402 commented Jul 28, 2025

Uh oh!

Rocketknight1 commented Sep 29, 2025

Uh oh!

molbap left a comment

Choose a reason for hiding this comment

Uh oh!

molbap Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

molbap Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

Leon0402 commented Oct 5, 2025

Uh oh!

github-actions Bot commented Oct 5, 2025

Uh oh!

molbap commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Leon0402 commented Jul 20, 2025 •

edited

Loading

Leon0402 Jul 21, 2025 •

edited

Loading

Leon0402 Jul 22, 2025 •

edited

Loading

molbap commented Oct 9, 2025 •

edited

Loading