Hi @Cyrilvallez ! I'm encountering a blocker while trying to fix the missing initializations for the Data2VecAudio model. Below is the updated initialization for this model:

def _init_weights(self, module: nn.Module):
        """Initialize the weights"""
        if isinstance(module, Data2VecAudioFeatureProjection):
            k = math.sqrt(1 / module.projection.in_features)
            nn.init.uniform_(module.projection.weight, a=-k, b=k)
            nn.init.uniform_(module.projection.bias, a=-k, b=k)
        elif isinstance(module, Data2VecAudioForXVector):
            nn.init.normal_(module.objective.weight)
        elif isinstance(module, Data2VecAudioPositionalConvLayer):
            nn.init.constant_(module.conv.bias, 0)
        elif isinstance(module, Data2VecAudioModel):
            # model only needs masking vector if mask prob is > 0.0
            if self.config.mask_time_prob > 0.0 or self.config.mask_feature_prob > 0.0:
                nn.init.uniform_(module.masked_spec_embed)
        elif isinstance(module, nn.Linear):
            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
            if module.bias is not None:
                module.bias.data.zero_()
        elif isinstance(module, nn.LayerNorm):
            if module.bias is not None:
                module.bias.data.zero_()
            if module.weight is not None:
                module.weight.data.fill_(1.0)
        elif isinstance(module, nn.Conv1d):
            nn.init.kaiming_normal_(module.weight)
            if module.bias is not None:
                k = math.sqrt(module.groups / (module.in_channels * module.kernel_size[0]))
                nn.init.uniform_(module.bias, a=-k, b=k)

When generating the modeling file from the modular file, I run into a code verification error during the make fixup process.

src/transformers/models/data2vec/modeling_data2vec_audio.py:797:35: F821 Undefined name `Data2VecAudioPreTrainedModel`
    |
795 |     """
796 | )
797 | class Data2VecAudioForPreTraining(Data2VecAudioPreTrainedModel):
    |                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ F821
798 |     def __init__(self, config: Data2VecAudioConfig):
799 |         super().__init__(config)
    |

Found 1 error.

The error occurs because Data2VecAudioPreTrainedModel is declared after Data2VecAudioForPreTraining, and I don't know how to skip the creation of Data2VecAudioForPreTraining during the modular_model_converter.py process.

Hey @bvantuan! Sorry for the delay! The issue comes from the fact that Data2VecAudioModeland Data2VecAudioForXVector are now referenced before their redefinition inside the modular file, so modular imports them and their dependencies before it sees how they should be redefined... 🥲 We should indeed add forward checks, not only backward ones for such cases.

I'll try to look into it, for now we can just skip this model, or perform the checks in a way which does not use those module name (e.g. if the actual module name is unique, we can simply use if hasattr(module, "module_name")

Hi @Cyrilvallez ! Thank you so much for reviewing and responding to my issue, as always😊. Yes, indeed, when using f hasattr(module, "module_name") for those two modules Data2VecAudioModel and Data2VecAudioForXVector, it worked.

For the term factor * 0.02, first we need to clarify the difference between config.initializer_factor and config.initializer_range. As far as I know, config.initializer_factor typically defaults to 1.0 and serves as a scaling factor for the initialization of all weight matrices within a model. config.initializer_range defines the standard deviation used for initializing the weights of a model (a common default value is 0.02). It is unusual to initialize the weights using config.initializer_factor as the standard deviation, because that value is excessively large for this purpose. Also, the term factor * 0.02 is sometimes used in the repository, for example in CLVP.

The original implementation of X-CLIP also hardcodes 0.02 for the Linear module:
https://github.com/microsoft/VideoX/blob/1744b2159c3d11694f94dab4fcde8660c685e79a/X-CLIP/models/mit.py#L40-L49

[For maintainers] Suggested jobs to run (before merge)

run-slow: altclip, bert, blip, bridgetower, camembert, chinese_clip, clip, clipseg, conditional_detr, cvt, data2vec, deformable_detr, detr, ernie, esm, groupvit

Yeah but unfortunately, once again if those old models were badly implemented at the time, it's hard to change it now. So this would be another PR, which would be quite breaking, so need to be separate. Happy to review such a PR in the future if you are open to it! In the meantime, this PR should keep its focus on adding missing initializations only, i.e. modules that are not taken into account - it should not change the default std of existing inits 🤗