refactor(modeling_llama): make RotaryEmbedding default path explicit#39831
refactor(modeling_llama): make RotaryEmbedding default path explicit#39831pco111 wants to merge 6 commits into
Conversation
Following the suggestion in issue huggingface#39753, this commit refactors the LlamaRotaryEmbedding (and its copies across other models) to make the initialization for the default rope_type more explicit. Instead of relying on the ROPE_INIT_FUNCTIONS dictionary for the default case, the code now uses a direct call to _compute_default_rope_parameters, reserving the dictionary lookup for non-default rope types. This improves code readability and maintainability. Closes huggingface#39753
| if self.rope_type == "default": | ||
| self.rope_init_fn = _compute_default_rope_parameters | ||
| else: | ||
| self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type] |
There was a problem hiding this comment.
hey! the idea is not to impor new fucntion, but to explicitly write what the default is! SO torch.arange(......) 😉
…and clarify the processing method of the default rope_type. By directly calculating the default parameters, improve the readability and maintainability of the code, while retaining dictionary search for non-default types.
…and clarify the processing method of the default rope_type. By directly calculating the default parameters, improve the readability and maintainability of the code, while retaining dictionary searches for non-default types.
…and clarify the processing method of the default rope_type. By directly calculating the default parameters, improve the readability and maintainability of the code, while retaining dictionary searches for non-default types.
|
Sorry, but let me give it a couple more tries😂 |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: arcee, aria, bamba, bitnet, cohere, cohere2, csm, deepseek_v3, dia, diffllama, doge, dots1, emu3, ernie4_5, ernie4_5_moe, evolla |
|
Hey @pco111 👋 @zucchini-nlp is working on a larger refactor of RoPE, which may conflict with this PR 🤔 (@zucchini-nlp can you confirm whether this part of the problem can be worked on independently?) |
| self.config = config | ||
| if self.rope_type == "default": | ||
| self.rope_init_fn = _compute_default_rope_parameters | ||
| base = self.config.rope_theta |
There was a problem hiding this comment.
Perfect, the only issue I have is that llama does not do partial! and should always have head dim!
It's gonna be a little bit of manual work but let's remove what was not there with the model: no partial when model does not do partial, no head dim if model does not have head dim in config!
What does this PR do?
This PR refactors
LlamaRotaryEmbeddingto make the initialization for the defaultrope_typemore explicit, as suggested in issue #39753.Instead of relying on the
ROPE_INIT_FUNCTIONSdictionary for the default case, the code now uses a direct call to_compute_default_rope_parameters. The dictionary lookup is reserved for non-defaultrope_typevalues. This change improves code readability and maintainability by making the default execution path clearer, aligning with the "explicit is better than implicit" philosophy.Fixes #39753
Before submitting
Pull Request section?
to it if that's the case. (Link: Inv frequency has not default, going against our philosophy #39753)
documentation guidelines, and
here are tips on formatting docstrings. (Not applicable, as this is an internal refactor with no user-facing changes).
Who can review?
@ArthurZucker @gante