Qwen3 ASR and Forced Aligner#43838
Conversation
|
@mbtariq82 thanks for opening the PR! we're definitely interested to add the model and were planning to work on it. Could you go ahead with the rest of the model? And I can iterate with you on it. I see you started with a modular file which is great. Below are some pointers of recent audio LM models that may help you with the other files / get an idea of our conventions: thanks 🤗 |
Create tester class and test processor initialization
create methods for common tests
|
I'm struggling to get test_apply_chat_template_audio from test_processing_common.py to pass. Specifically, the final part of the test where we apply_chat_template with continue_final_message=True fails with ValueError(continue_final_message is set but the final message does not appear in the chat after applying the chat template!...) I've verified that the chat_template is being correctly loaded from the model checkpoint: Qwen/Qwen3-ASR-0.6B. According to ChatGPT, the chat_template provided by Qwen is not correctly rendering the final assistant message so I think the only way to solve this is to override the apply_chat_template method and add some custom logic before calling super().apply_chat_template? |
|
@mbtariq82 it's only about getting the test to pass but the model is behaving as expected, let's avoid overwriting For now you can leave the test failing, and if necessary we can overwrite or even skip the test later on |
|
when you finish the modeling, and integration tests that produce equivalent outputs to the original, e.g. for audio flamingo:
I can already take a look to give some feedback! and we can take a look at the test after that |
Create integration test Setup Qwen3ASRModelTester
|
So between the current version and v4.57.6, the "default" key was removed from ROPE_INIT_FUNCTIONS. Qwen3-ASR was built using v4.57.6 and the checkpoint uses the "default" key. I've changed the rope_type to "linear" in Qwen3ASRThinkerTextRotaryEmbedding for now but I'm not sure if this is correct. I also changed the "attentions" PyTorch hooks, it was set to Qwen3ASRThinkerTextAttention which is not used at all in the base class - maybe they plan to use it in the future but I'm not sure. I've changed it to Qwen3ASRTextAttention to get the tests to pass. I've added the entire model. All the tests are passing. |
Add property methods to config Add base_model_prefix and wrapper method to generation class
…ion weights CLEANUP NEEDED
…n to Qwen3ASRTextAttention, Qwen3ASRThinkerTextAttention is never instantiated and so 'attentions' was not being properly propogated Fix integration tests
ebezzam
left a comment
There was a problem hiding this comment.
Hi @mbtariq82 thanks for working on this integration! I'm doing a small review because I noticed you started a modular file, but aren't making full use of its functionality to generate the configuration, processing, and modeling from existing components in Transformers. I gave some pointers for the configuration and processing but will let you check out the rest for the modeling components.
I encourage reading this page on using modular to contribute models: https://huggingface.co/docs/transformers/en/modular_transformers
And for practical examples you can see other modular files:
- Qwen3OmniMoe, which has a lot of similarity with the ASR model (namely removing the vision modalities): https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen3_omni_moe/modular_qwen3_omni_moe.py
- A recent Audio LM addition: https://github.com/huggingface/transformers/blob/main/src/transformers/models/glmasr/modular_glmasr.py
Note: I think you will have to use "Asr" instead of "ASR" in your model naming because the modular script prefers camelcase.
…gn RoPE position handling with cache_position Refactor position_ids construction to be fully cache_position-driven and generation-safe. - Compute batch_size/seq_length from inputs_embeds - Initialize cache_position when absent - Build 3D position_ids from cache_position - Compute rope_deltas once during prefill - Reuse rope_deltas for subsequent decode steps Removes legacy attention_mask-dependent branch that was incompatible with static cache generation. Ensures correct RoPE offsets for multimodal inputs under both dynamic and static cache modes.
|
I made some big changes in the base model's forward in this commit: 0b3248d. I also removed get_rope_index. |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: audioflamingo3, auto, dac, glmasr, qwen2_5_omni, qwen3_asr, qwen3_omni_moe, vibevoice_asr |
1 similar comment
|
[For maintainers] Suggested jobs to run (before merge) run-slow: audioflamingo3, auto, dac, glmasr, qwen2_5_omni, qwen3_asr, qwen3_omni_moe, vibevoice_asr |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: audioflamingo3, auto, dac, glmasr, qwen2_5_omni, qwen3_asr, qwen3_omni_moe, vibevoice_asr |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: audioflamingo3, auto, dac, glmasr, qwen2_5_omni, qwen3_asr, qwen3_omni_moe, vibevoice_asr |
|
run-slow: qwen3_asr |
|
This comment contains models: ["models/qwen3_asr"] |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: audioflamingo3, auto, dac, glmasr, qwen2_5_omni, qwen3_asr, qwen3_omni_moe, vibevoice_asr |
1 similar comment
|
[For maintainers] Suggested jobs to run (before merge) run-slow: audioflamingo3, auto, dac, glmasr, qwen2_5_omni, qwen3_asr, qwen3_omni_moe, vibevoice_asr |
|
run-slow: qwen3_asr |
|
This comment contains models: ["models/qwen3_asr"] |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: audioflamingo3, auto, dac, glmasr, qwen2_5_omni, qwen3_asr, qwen3_omni_moe, vibevoice_asr |
What does this PR do?
This PR adds Qwen3-ASR to the Transformers library.
Fixes #43837
Before submitting
Pull Request section?
to it if that's the case. Proposal to add Qwen3-ASR support #43837
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.