Codestin Search App

harshaljanjani · 2026-06-22T20:02:22Z

What does this PR do?

→ This PR adds Canary 1B v2 to Transformers!

References:
→ Model Checkpoints
→ GitHub Repository
→ Transformers Converted Checkpoints
→ Research Paper
→ Testing Script

Rebased on top of @eustlb's #40756

cc: @ebezzam @Rocketknight1

Multilingual Transcription and Test Results

transformers version: 5.13.0.dev0
Platform: Linux-6.8.0-1060-gcp-x86_64-with-glibc2.35
Python version: 3.11.15
huggingface_hub version: 1.20.1
safetensors version: 0.8.0
accelerate version: 1.14.0
DeepSpeed version: not installed
PyTorch version (accelerator?): 2.11.0+cu130 (CUDA)
GPU type: NVIDIA L4
CUDA version: 13.0

Before submitting

This PR adds a new model to Transformers.
Did you read the contributor guidelines, specifically the Pull Request section?
Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
Did you add any necessary tests?

github-actions · 2026-06-22T20:03:15Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

harshaljanjani · 2026-06-26T04:19:13Z

Good day @eustlb @ebezzam, just a gentle ping in regard to model reviews!

harshaljanjani · 2026-07-08T05:46:11Z

Good day @ebezzam! Just bumping this up for whenever you find the time; thanks!

ebezzam

hi @harshaljanjani thanks for the contribution! Here are some initial comments to start iterating. I didn't have time for an in-depth review, but hopefully these help to get things in the expected direction!

Main idea is to see how existing models define / implement the various parts, and trying to use modular as much as possible!

ebezzam · 2026-07-01T12:34:46Z

+
+    def __init__(
+        self,
+        embed_dim: int,


I wonder if super_kwargs could be used here? I learned about this for here

Thanks for the precedent link; done!

ebezzam · 2026-07-01T12:35:46Z

+
+
+@auto_docstring
+class CanaryPreTrainedModel(PreTrainedModel):


(nit but good practice) could we do modular from an existing model? to save some lines

Done as well.

ebezzam · 2026-07-01T12:39:08Z

+        if encoder_outputs is None:
+            encoder_outputs = self.encoder(
+                input_features=input_features,
+                attention_mask=attention_mask,
+                **kwargs,
+            )
+
+        encoder_attention_mask = getattr(encoder_outputs, "attention_mask", None)


could we define and use get_audio_features like this for calling the encoder

Thanks for the precedent, done.

ebezzam · 2026-07-01T12:43:35Z

+    speech-to-text translation.
+    """
+)
+class CanaryForConditionalGeneration(CanaryPreTrainedModel, GenerationMixin):


can we do modular from WhisperForConditionalGeneration?

IMO since WhisperForConditionalGeneration inherits from the custom WhisperGenerationMixin with tons of custom methods specific to Whisper (long-form chunking, timestamp extraction for Whisper-specific token IDs, etc., also bias = False), we'd be adding breakage here if we inherited from the class. Moonshine sets precedent as well by being standalone with the same WhisperModel derived shape. Happy to know if I'm missing something here though!

ebezzam · 2026-07-01T12:44:49Z

+            init.copy_(module.positional_embeddings, module._build_table())
+
+
+class CanaryDecoder(CanaryPreTrainedModel):


TO CHECK: there isn't a similar module elsewhere in the lib? for example Whisper

Deduped it with WhisperDecoder, thanks for the nudge!

ebezzam · 2026-07-08T14:29:25Z

+class CanaryModel(CanaryPreTrainedModel):
+    def __init__(self, config: CanaryConfig):
+        super().__init__(config)
+        self.encoder = AutoModel.from_config(config.encoder_config)
+        self.decoder = CanaryDecoder(config)
+        self.post_init()


could we do modular from WhisperModel?

ebezzam · 2026-07-08T14:33:59Z

+        audio = make_list_of_audio(audio)
+        inputs = self.feature_extractor(audio, **output_kwargs["audio_kwargs"])
+
+        prompt_tokens = self._build_prompt_tokens(source_lang, target_lang, pnc, timestamps)


can we use a chat template for specifying these prompts? See VibeVoice ASR or Qwen 3 ASR. And so we likely need to define an additional method called apply_transcription_request that will call apply_chat_template which then calls this __call__.

The purpose of this __call__ should be simply:

apply feature extrator to audio and insert audio tokens

tokenize text

(optionally) prepare output labels

you will need to add this chat template to your checkpoint in the conversion script

Done following the aforementioned lib patterns, happy to adjust if anything comes up!

ebezzam · 2026-07-08T14:36:32Z

can we generate this via modular from an existing config?

ebezzam · 2026-07-08T14:39:23Z

+
+Canary reuses the [Fast Conformer](https://huggingface.co/papers/2305.05084) encoder from [Parakeet](./parakeet.md) (loaded through [`ParakeetEncoder`] / [`ParakeetEncoderConfig`]) and pairs it with a Transformer decoder that uses fixed sinusoidal positional embeddings, cross-attention to the encoder outputs and tied input/output embeddings. The task is selected through a decoder prompt prefix built by [`CanaryProcessor`] of the form `<|startofcontext|> <|startoftranscript|> <source_lang> <target_lang> <pnc|nopnc> <timestamp|notimestamp> ...`, where `source_lang == target_lang` selects transcription and otherwise selects translation.
+
+The original implementation can be found in [NVIDIA NeMo](https://github.com/NVIDIA/NeMo). A Transformers-compatible checkpoint is available at [harshaljanjani/canary-1b-v2-hf](https://huggingface.co/harshaljanjani/canary-1b-v2-hf).


NOTE for later: we'll eventually want to transfer a checkpoint to NVIDIA's org

ebezzam · 2026-07-08T14:40:35Z

+
+The original implementation can be found in [NVIDIA NeMo](https://github.com/NVIDIA/NeMo). A Transformers-compatible checkpoint is available at [harshaljanjani/canary-1b-v2-hf](https://huggingface.co/harshaljanjani/canary-1b-v2-hf).
+
+## Usage


other example usage, we'd like to cover:

batch

training (simply showing forward/backward)

torch compile

model features such as timestamp, translation, diariarization

Check out existing models like AudioFlamingo3, VIbevoice ASR, Qwen3 ASR

Done, fleshed out all the tasks with examples :)

harshaljanjani · 2026-07-09T09:48:22Z

Thanks for your time @ebezzam; left replies, addressed the review comments and verified that there are no regressions across the test suite or in generation. I suspect the failures are alluding to cache warmup and will be fixed with this review when we get to it?

github-actions · 2026-07-09T09:51:33Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, canary

github-actions · 2026-07-09T10:09:17Z

CI recap

Dashboard: View test results in Grafana
Latest run: 29009401108:2
Result: failure | Jobs: 15 | Tests: 171,039 | Failures: 8 | Duration: 19h 53m

eustlb and others added 2 commits June 22, 2026 13:02

CanaryConverter draft

36767d1

init: Add files (canary-1b-v2)

403a6df

fix: Resolve Bandit warning

c9a82bf

harshaljanjani marked this pull request as ready for review June 23, 2026 03:48

github-actions Bot requested review from ArthurZucker and Rocketknight1 June 23, 2026 03:48

harshaljanjani marked this pull request as draft June 23, 2026 04:30

fix: Make CI green 1

86a5cf1

harshaljanjani marked this pull request as ready for review June 23, 2026 04:50

harshaljanjani mentioned this pull request Jul 1, 2026

feat[vLLM x v5]: Rename audio_processor to feature_extractor on Granite Speech and Phi-4 Multimodal processors #46992

Closed

3 tasks

ebezzam added Audio New model labels Jul 1, 2026

ebezzam self-assigned this Jul 1, 2026

ebezzam reviewed Jul 8, 2026

View reviewed changes

harshaljanjani added 4 commits July 8, 2026 19:00

Merge branch 'main' into add-canary

210d2ee

refactor: Address review round 1 - 1

9c2e8b3

Merge branch 'main' into add-canary

769b816

refactor: Address review round 1 - 2

d471d82

harshaljanjani requested a review from ebezzam July 9, 2026 10:07



		@auto_docstring
		class CanaryPreTrainedModel(PreTrainedModel):

		init.copy_(module.positional_embeddings, module._build_table())


		class CanaryDecoder(CanaryPreTrainedModel):


		Canary reuses the [Fast Conformer](https://huggingface.co/papers/2305.05084) encoder from [Parakeet](./parakeet.md) (loaded through [`ParakeetEncoder`] / [`ParakeetEncoderConfig`]) and pairs it with a Transformer decoder that uses fixed sinusoidal positional embeddings, cross-attention to the encoder outputs and tied input/output embeddings. The task is selected through a decoder prompt prefix built by [`CanaryProcessor`] of the form `<\|startofcontext\|> <\|startoftranscript\|> <source_lang> <target_lang> <pnc\|nopnc> <timestamp\|notimestamp> ...`, where `source_lang == target_lang` selects transcription and otherwise selects translation.

		The original implementation can be found in [NVIDIA NeMo](https://github.com/NVIDIA/NeMo). A Transformers-compatible checkpoint is available at [harshaljanjani/canary-1b-v2-hf](https://huggingface.co/harshaljanjani/canary-1b-v2-hf).


		The original implementation can be found in [NVIDIA NeMo](https://github.com/NVIDIA/NeMo). A Transformers-compatible checkpoint is available at [harshaljanjani/canary-1b-v2-hf](https://huggingface.co/harshaljanjani/canary-1b-v2-hf).

		## Usage

Uh oh!

Conversation

harshaljanjani commented Jun 22, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Multilingual Transcription and Test Results

Before submitting

Uh oh!

github-actions Bot commented Jun 22, 2026

CI Security Gate — automatic approval blocked

Uh oh!

harshaljanjani commented Jun 26, 2026

Uh oh!

harshaljanjani commented Jul 8, 2026

Uh oh!

ebezzam left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harshaljanjani commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 9, 2026

CI recap

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

harshaljanjani commented Jun 22, 2026 •

edited by github-actions Bot

Loading

harshaljanjani commented Jul 9, 2026 •

edited

Loading