Add HumanV: decoder-only causal LM#42982
Conversation
|
View the CircleCI Test Summary for this PR: https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42982&sha=206397 |
|
This looks like it should be possible to implement under existing model architectures, right? Have you tried just converting the model checkpoints to |
We agree that parts of HumanV overlap with existing decoder-only architectures (RoPE, RMSNorm, SiLU, and optional GQA/MHA). The motivation for a separate architecture PR is not to rebrand a LLaMA-like baseline, but to provide a stable upstream home for ongoing attention research we are actively implementing and benchmarking. What is new / why not “just convert to LLaMA”? HumanV is designed as a research-friendly attention sandbox within transformers, where attention patterns can be selected per-layer (e.g., full_attention vs sliding_attention), and where we can introduce non-standard attention backends without bending existing model classes beyond their intended scope. We already implemented an opt-in Local–Global Block Sparse Attention mode (via config), and we are actively validating it for numerical stability (no NaNs) and correctness with small training runs (including TPU settings). Sparse attention roadmap Our goal is to integrate and evaluate modern sparse attention approaches (e.g., block-sparse / local-global hybrids). The sparse path is fully behind configuration flags (e.g. use_sparse_attention=True, sparse_attention_impl="local_global_block"), so the default remains standard dense attention. Training & reproducibility We are currently training a fresh checkpoint using the new sparse attention algorithm, and once training is complete we will upload the resulting model checkpoint and update this PR with a direct Hugging Face link for reproducibility. In short: this PR establishes the base HumanV architecture in transformers and enables a clean, opt-in path for sparse attention research that cannot be represented by a simple “convert to LLaMA” checkpoint when the attention mechanism itself is actively evolving. |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, humanv |
ArthurZucker
left a comment
There was a problem hiding this comment.
Hey! We don't mind adding a new model_type with your custom model name.
Just note that for us, a new architecture is a new model.
HumanV is designed as a research-friendly attention sandbox within transformers, where attention patterns can be selected per-layer (e.g., full_attention vs sliding_attention), and where we can introduce non-standard attention backends without bending existing model classes beyond their intended scope.
This is something you can already do with models that have layer_type in the config!
What matters the most for us is just standardization, so making sure your code uses https://huggingface.co/docs/transformers/main/en/modular_transformers !
Just isolate the changes relate to sparse attention for example for attention mask creation.
The main hickup is that for us, a new architecture == a new model (we don't bloat old code / code if it does not support the feature)!
🤗 Happy to help
What does this PR do?
This PR adds HumanV, a lightweight decoder-only causal language model architecture, to 🤗 Transformers.
Key additions:
humanvmodel implementation (HumanVConfig,HumanVModel,HumanVForCausalLM)AutoModelForCausalLM) once mergedA small trained checkpoint is available on the Hugging Face Hub:
Fixes # (issue)
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
Suggested reviewers for text model additions:
@ArthurZucker @Cyrilvallez