Codestin Search App

ebezzam · 2026-07-06T14:58:20Z

What does this PR do?

A few patches for Qwen3 ASR:

add support for custom hotwords, e.g. passing custom vocabulary to help transcription like VibeVoice. Original model card and repo didn't show such examples... but a user comment on the new checkpoints brought up this feature not handled by current code. And looking into original code, I see this context arg.
language hint should be handled differently
while doing above, I noticed a training bug which was copied from other models, because AudioFlamingo3, GlmASR, and MusicFlamingo were affected by this refactor (I'll open a separate PR to fix them)

ebezzam · 2026-07-06T15:00:37Z

+# - assistant turns: rendered verbatim, wrapped in `{% generation %}` for assistant token masking.
+#   Used to prefill the forced language ("language <NAME><asr_text>") via
+#   `apply_chat_template(..., continue_final_message=True)` and for training targets.
+ASR_CHAT_TEMPLATE = (


would need to update chat templates of Qwen/Qwen3-ASR-1.7B-hf and Qwen/Qwen3-ASR-0.6B-hf. I can see with model authors if we do this direction

OR we could go with this approach to not have to update the chat template (but perhaps unconventional approach with apply_chat_template)

Imo its best to update the template to avoid headaches. This cries workaround an potentially messy maintenance

ebezzam · 2026-07-06T15:03:27Z

        audio: AudioInput | list[AudioInput],
        language: str | list[str] | None = None,
-        **kwargs,
+        prompt: str | list[str] | None = None,


adding prompt arg like VibeVoice ASR here

ebezzam · 2026-07-06T15:04:55Z

+        # Following the original implementation, the language is forced by prefilling the assistant
+        # turn with "language <NAME><asr_text>" so that the model only generates the transcription.
+        if self._template_renders_assistant_turns():
+            for messages, lang in zip(conversations, languages):
+                prefill = f"language {lang}<asr_text>" if lang is not None else ""
+                messages.append({"role": "assistant", "content": [{"type": "text", "text": prefill}]})
+            return self.apply_chat_template(
+                conversations,
+                tokenize=True,
+                return_dict=True,
+                continue_final_message=True,
+                **kwargs,
+            )


if we push the new chat template, we can call self.apply_chat_template with tokenize=True

ebezzam · 2026-07-06T15:06:12Z

+        # Legacy chat templates (before assistant-turn rendering was added) cannot express the
+        # prefill, so splice the forced-language suffix into the rendered text manually.
+        text = self.apply_chat_template(conversations, tokenize=False, add_generation_prompt=True)
+        text = [
+            rendered if lang is None else f"{rendered}language {lang}<asr_text>"
+            for rendered, lang in zip(text, languages)
+        ]
+
+        sampling_rate = kwargs.get(
+            "sampling_rate", kwargs.get("audio_kwargs", {}).get("sampling_rate", self.feature_extractor.sampling_rate)
        )
+        audio_arrays = [load_audio(item, sampling_rate=sampling_rate) for item in audio_items]
+        return self(text=text, audio=audio_arrays, **kwargs)


Otherwise we could leave the chat template as-is, but we wouldn't call self.apply_chat_template with tokenize=True and need to call the processor __call__ manually -> this would be the least breaking approach

ebezzam · 2026-07-06T15:10:08Z

-            return_dict=True,
-            **kwargs,
+        # Following the original implementation, the language is forced by prefilling the assistant
+        # turn with "language <NAME><asr_text>" so that the model only generates the transcription.


original: https://github.com/QwenLM/Qwen3-ASR/blob/7c6daf77a2421100f5fb066495372c00129d39ff/qwen_asr/inference/qwen3_asr.py#L463

ebezzam · 2026-07-06T15:11:31Z

+            mm_token_type_ids = model_inputs.pop("mm_token_type_ids")
+            labels = model_inputs["input_ids"].clone()
+            labels[mm_token_type_ids != 0] = -100  # audio positions
            for token_id in [


bug in preparing labels!

AudioFlamingo3 also affected here, and as a result GlmASR and MusicFlamingo which do modular from this processor.

yep same comment util worthy!

HuggingFaceDocBuilderDev · 2026-07-06T15:12:22Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ebezzam · 2026-07-06T15:13:57Z

run-slow: qwen3_asr

github-actions · 2026-07-06T15:15:18Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/qwen3_asr"]
quantizations: []

github-actions · 2026-07-06T15:28:28Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	d167b695	workflow commit (merge commit)
PR	e4cf30a3	branch commit (from PR)
main	1da9d1d4	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

vasqu

Some initial comments. I would nudge towards updating the templates remotely tbh as it will get messy otherwise

vasqu · 2026-07-07T08:52:02Z

    print(text)
 ```

+The language can be forced through the chat template by *prefilling* the assistant turn with `language <NAME><asr_text>` and passing `continue_final_message=True`, which is what `apply_transcription_request` does under the hood:


could we kind of combine it with the previous instead with a small comment?

Iiuc then its possible not only to pass to the chat template but also give it as prefix?

vasqu · 2026-07-07T08:53:24Z

+# - assistant turns: rendered verbatim, wrapped in `{% generation %}` for assistant token masking.
+#   Used to prefill the forced language ("language <NAME><asr_text>") via
+#   `apply_chat_template(..., continue_final_message=True)` and for training targets.
+ASR_CHAT_TEMPLATE = (


Imo its best to update the template to avoid headaches. This cries workaround an potentially messy maintenance

vasqu · 2026-07-07T08:54:26Z

+            mm_token_type_ids = model_inputs.pop("mm_token_type_ids")
+            labels = model_inputs["input_ids"].clone()
+            labels[mm_token_type_ids != 0] = -100  # audio positions
            for token_id in [


yep same comment util worthy!

vasqu · 2026-07-07T08:56:19Z

+        if prompt is None:
+            prompts = [None] * batch_size
+        elif isinstance(prompt, str):
+            prompts = [prompt] * batch_size
+        elif isinstance(prompt, (list, tuple)):
+            if len(prompt) != batch_size:
+                raise ValueError(
+                    f"Received {len(prompt)} prompt(s) for {batch_size} audio sample(s); counts must match."
+                )
+            prompts = list(prompt)
+        else:
+            raise TypeError("`prompt` must be a string, a sequence of strings, or `None`.")


I would check whether something similar already exists? this looks failry standard and should work for most standard primitives

I couldn't find one, but creating prepare_prompt_input in audio utils because this was copied from VibeVoice, and future audio LMs may use this!

vasqu · 2026-07-07T08:57:07Z

-        audio_items = make_list_of_audio_chat_template(audio)
+        audio_items = list(make_list_of_audio_chat_template(audio))
+        if is_torch_available():
+            audio_items = [el.detach().cpu().numpy() if isinstance(el, torch.Tensor) else el for el in audio_items]


why is this needed?

don't need, copied from vibevoice but we can keep as tensors

vasqu · 2026-07-07T08:57:45Z

+        # Following the original implementation, the language is forced by prefilling the assistant
+        # turn with "language <NAME><asr_text>" so that the model only generates the transcription.
+        if self._template_renders_assistant_turns():
+            for messages, lang in zip(conversations, languages):
+                prefill = f"language {lang}<asr_text>" if lang is not None else ""
+                messages.append({"role": "assistant", "content": [{"type": "text", "text": prefill}]})
+            return self.apply_chat_template(
+                conversations,
+                tokenize=True,
+                return_dict=True,
+                continue_final_message=True,
+                **kwargs,
+            )


vasqu · 2026-07-07T08:58:24Z

@@ -36,7 +36,7 @@ class Qwen3ASRProcessorTest(ProcessorTesterMixin, unittest.TestCase):
    @classmethod
    @require_torch
    def setUpClass(cls):
-        cls.checkpoint = "Qwen/Qwen3-ASR-0.6B-hf"
+        cls.checkpoint = "bezzam/Qwen3-ASR-0.6B-hf"


we can open a PR and add a revision for now no? Ideally no revision before merge then

vasqu · 2026-07-07T08:59:50Z

+        self.assertTrue(decoded.endswith("<|im_start|>assistant\nlanguage English<asr_text>"))
+
+    @require_torch
+    def test_apply_transcription_request_with_prompt(self):


What happens if we have both language and prompt? Should they interact in a specific order or should we make it mutually exclusive etc? I think it would make sense to allow both but lan -> prompt

both are allowed! as seen in the chat template example (input repeated below)

prompts are in the user role

language is in the assistant role

their model was trained to have language forcing in the assistant role, so I guess we need to keep it that way?

chat_template = [ [ # Context/hotwords as system message {"role": "system", "content": [{"type": "text", "text": "Vocabulary: Quilter, apostle, gospel."}]}, { "role": "user", "content": [ { "type": "audio", "path": "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav", }, ], }, # empty prefill since forcing language in the other sample {"role": "assistant", "content": [{"type": "text", "text": ""}]}, ], [ { "role": "user", "content": [ { "type": "audio", "path": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_zh.wav", }, ], }, {"role": "assistant", "content": [{"type": "text", "text": "language Chinese<asr_text>"}]}, ], ]

ebezzam · 2026-07-07T17:13:56Z

    @require_torch
    def setUpClass(cls):
        cls.checkpoint = "Qwen/Qwen3-ASR-0.6B-hf"
+        cls.revision = "refs/pr/3"  # TODO: set to main after merge


@vasqu Opened PRs to the model cards:

https://huggingface.co/Qwen/Qwen3-ASR-1.7B-hf/discussions/2

https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf/discussions/3

When we are ready to merge:

I will tell model authors to merge

we set revision to main (or remove altogether cls.revision)

ebezzam · 2026-07-07T17:18:27Z

run-slow: qwen3_asr

github-actions · 2026-07-07T17:20:35Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/qwen3_asr"]
quantizations: []

github-actions · 2026-07-07T17:36:07Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	b5cbed78	workflow commit (merge commit)
PR	caf37300	branch commit (from PR)
main	4af901d5	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

vasqu

LGTM, would be nice to wait for the merge on the hub before merging this one tho 🤗

vasqu · 2026-07-08T19:30:09Z

+        for prompt_text, audio_item in zip(prompts, audio_items):
            messages = []
-            if lang is not None:
-                messages.append({"role": "system", "content": [{"type": "text", "text": lang}]})
+            if prompt_text is not None:
+                messages.append({"role": "system", "content": [{"type": "text", "text": prompt_text}]})
            messages.append({"role": "user", "content": [_audio_content_item(audio_item)]})
            conversations.append(messages)

+        # The language is forced by prefilling the assistant turn with "language <NAME><asr_text>"
+        # so that the model only generates the transcription text in the desired language
+        for messages, lang in zip(conversations, languages):
+            prefill = f"language {lang}<asr_text>" if lang is not None else ""
+            messages.append({"role": "assistant", "content": [{"type": "text", "text": prefill}]})


Ah ok one is the user and one is the assistant I think I missed that last time. Really weird template

vasqu · 2026-07-08T19:30:59Z

    return make_list_of_audio(audio)


+def prepare_prompt_input(


maybe even processing utils or similar? not really sure if its audio only but super nit

vasqu · 2026-07-08T19:31:55Z

Lets be careful that the processor is not too heavy for CI?

We can make a dummy one imo

FYI checked with @ydshieh. He's working on a script/workflow to make tiny model copies for the processor tests (e.g. see the tiny-processor models here). So if it passes with the current checkpoint, we'll let him create the tiny processor in a follow-up PR

Sounds good, feel free to merge then

thanks! waiting for the model card PRs to be merged

github-actions · 2026-07-09T13:24:56Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: musicflamingo, qwen3_asr, vibevoice_asr

github-actions · 2026-07-09T13:33:55Z

CI recap

Dashboard: View test results in Grafana
Latest run: 29020934031:1
Result: failure | Jobs: 12 | Tests: 169,853 | Failures: 2 | Duration: 16h 51m

Add hotword parsing, and fix language parsing and training.

e4cf30a

ebezzam commented Jul 6, 2026

View reviewed changes

ebezzam requested a review from vasqu July 6, 2026 15:15

vasqu reviewed Jul 7, 2026

View reviewed changes

ebezzam added 5 commits July 7, 2026 15:11

update doc

281852a

Merge branch 'main' into qwenasr_hotwords

f56cc06

Add temporary revision.

92f439f

Merge branch 'main' into qwenasr_hotwords

e24ef81

Address comments.

7afd31a

ebezzam commented Jul 7, 2026

View reviewed changes

Remove legacy chat template path.

caf3730

gemini-code-assist Bot mentioned this pull request Jul 8, 2026

Sync AutoSubs support and Qwen3-ASR tracker modelscope/FunASR#3195

Merged

ebezzam requested a review from vasqu July 8, 2026 14:47

vasqu approved these changes Jul 8, 2026

View reviewed changes

Move prompt util to processing utils.

b023162

Uh oh!

Conversation

ebezzam commented Jul 6, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

ebezzam Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Jul 6, 2026

Uh oh!

ebezzam commented Jul 6, 2026

Uh oh!

github-actions Bot commented Jul 6, 2026

Uh oh!

github-actions Bot commented Jul 6, 2026

CI Results

Commit Info

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebezzam Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebezzam commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 7, 2026

CI Results

Commit Info

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebezzam commented Jul 6, 2026 •

edited by github-actions Bot

Loading

ebezzam Jul 6, 2026 •

edited

Loading

ebezzam Jul 7, 2026 •

edited

Loading