See huggingface/transformers#39847

See huggingface/transformers#39847

See huggingface/transformers#39847

…os/sin) (huggingface#47403)

* Fix Qwen2.5-Omni Token2Wav DiT rotary embedding layout

The Token2Wav DiT applies an interleaved rotate (`rotate_half_codec`,
pairing channels (2i, 2i+1)), but since huggingface#39847 the DiT rotary embedding
built cos/sin with the half-split `torch.cat((freqs, freqs))` layout
inherited from `LlamaRotaryEmbedding`. A half-split cos/sin combined with
an interleaved rotate is not a valid rotation: RoPE's translation
invariance is destroyed on head 0 of every DiT attention layer, degrading
the predicted mel-spectrogram (and thus the synthesized audio).

Override `Qwen2_5OmniDiTRotaryEmbedding.forward` to emit the interleaved
`repeat_interleave(freqs, 2)` layout, restoring the pre-huggingface#39847 behavior
that matches `rotate_half_codec` and the released weights (originally
integration-tested in huggingface#36752). Add CPU regression tests asserting the
interleaved layout and RoPE translation invariance.

* Use kernelized apply_rotary_pos_emb with deinterleaved q/k in Token2Wav DiT

* Replace DiT rotary unit tests with Token2Wav audio regression test

* Replace audio-regression fixture with inline Expectations-based test

* Add expected outputs for our CI.

---------

Co-authored-by: Eric B <[email protected]>
Co-authored-by: Eric Bezzam <[email protected]>

…os/sin) (huggingface#47403)

* Fix Qwen2.5-Omni Token2Wav DiT rotary embedding layout

The Token2Wav DiT applies an interleaved rotate (`rotate_half_codec`,
pairing channels (2i, 2i+1)), but since huggingface#39847 the DiT rotary embedding
built cos/sin with the half-split `torch.cat((freqs, freqs))` layout
inherited from `LlamaRotaryEmbedding`. A half-split cos/sin combined with
an interleaved rotate is not a valid rotation: RoPE's translation
invariance is destroyed on head 0 of every DiT attention layer, degrading
the predicted mel-spectrogram (and thus the synthesized audio).

Override `Qwen2_5OmniDiTRotaryEmbedding.forward` to emit the interleaved
`repeat_interleave(freqs, 2)` layout, restoring the pre-huggingface#39847 behavior
that matches `rotate_half_codec` and the released weights (originally
integration-tested in huggingface#36752). Add CPU regression tests asserting the
interleaved layout and RoPE translation invariance.

* Use kernelized apply_rotary_pos_emb with deinterleaved q/k in Token2Wav DiT

* Replace DiT rotary unit tests with Token2Wav audio regression test

* Replace audio-regression fixture with inline Expectations-based test

* Add expected outputs for our CI.

---------

Co-authored-by: Eric B <[email protected]>
Co-authored-by: Eric Bezzam <[email protected]>

…os/sin) (huggingface#47403)

* Fix Qwen2.5-Omni Token2Wav DiT rotary embedding layout

The Token2Wav DiT applies an interleaved rotate (`rotate_half_codec`,
pairing channels (2i, 2i+1)), but since huggingface#39847 the DiT rotary embedding
built cos/sin with the half-split `torch.cat((freqs, freqs))` layout
inherited from `LlamaRotaryEmbedding`. A half-split cos/sin combined with
an interleaved rotate is not a valid rotation: RoPE's translation
invariance is destroyed on head 0 of every DiT attention layer, degrading
the predicted mel-spectrogram (and thus the synthesized audio).

Override `Qwen2_5OmniDiTRotaryEmbedding.forward` to emit the interleaved
`repeat_interleave(freqs, 2)` layout, restoring the pre-huggingface#39847 behavior
that matches `rotate_half_codec` and the released weights (originally
integration-tested in huggingface#36752). Add CPU regression tests asserting the
interleaved layout and RoPE translation invariance.

* Use kernelized apply_rotary_pos_emb with deinterleaved q/k in Token2Wav DiT

* Replace DiT rotary unit tests with Token2Wav audio regression test

* Replace audio-regression fixture with inline Expectations-based test

* Add expected outputs for our CI.

---------

Co-authored-by: Eric B <[email protected]>
Co-authored-by: Eric Bezzam <[email protected]>

Transformers 5 moved rotary embedding settings from direct config
attributes into rope_parameters. Pinned llm-awq still reads the removed
rope_theta and rope_scaling attributes while starting TinyChat.

Read rope_parameters when available and retain the legacy fallback. This
preserves the pinned Llama-3 VILA model theta of 500000.0 and its default
unscaled RoPE behavior.

Related: huggingface/transformers#39847
Signed-off-by: Jean-Christophe Fillion-Robin <[email protected]>

Transformers 5 stores rotary embedding settings in rope_parameters. The
pinned TinyChat Llama implementation reads the legacy rope_theta and
rope_scaling fields in both attention and context-stage setup.

Read both schemas through one helper without mutating the model
configuration. Default RoPE retains the legacy scale of 1.0. Scaled
variants retain their model-provided theta and inverse-factor behavior.

Related: huggingface/transformers#39847
Signed-off-by: Jean-Christophe Fillion-Robin <[email protected]>

…ernal report)

Reported by @venvoo (Wenbin Wu) on the HF repo discussion -- the first external check of
these artifacts by anyone outside the project, and it found a real error.

VERIFIED, every checkable claim. Our uv.lock resolves transformers 5.9.0. In that range
the OLMo-3 modeling code applied the config YaRN rope_scaling to every attention layer, a
regression from huggingface/transformers#39847 fixed in #46911 (v5.13.0). Both PRs exist
and are merged, by zucchini-nlp, with titles matching the report. Correct behaviour per
the OLMo 3 paper 3.6.4 and OLMo-core exporter applies YaRN to full-attention layers only.

CHECKED THE CONFIG DIRECTLY, which makes it worse than reported: OLMo-3-7B has 32 layers,
24 sliding_attention and 8 full_attention, every 4th layer full. So 24 of 32 layers --
75% of the model -- ran a positional encoding the model was not trained with. Our 11
lenses were fitted 2026-07-22 to 08-02, after the fix shipped on 07-03 but against a lock
predating it.

Added one hypothesis the report does not make: our fits used max_seq_len 128, where YaRN
frequency interpolation is position-scaled and therefore small, which plausibly explains
the 0.96-0.99 per-prompt correlation despite 75% of layers being affected. BUT YaRN also
applies attention_factor 1.2079, a POSITION-INDEPENDENT softmax temperature, on 24 layers
that should not have it -- so this is not a long-context-only artifact, and that is our
best guess at the residual 4-6% against Neuronpedia convergence.csv. Flagged as
hypothesis, not finding.

Actions taken: results/ROPE_ERRATUM.md written with the full account and a checklist;
erratum banner prepended to all 10 posttrain result files and the cross-validation;
pyproject pinned transformers>=5.13 (was >=5.5). Refit and re-run of the ladder still
pending.

THE EXPENSIVE LESSON, now PITFALLS #26 in the curriculum repo.
results/neuronpedia_crossvalidation.md calls itself "the first external check" of our
pipeline. It was not. Both lenses were produced by the same library version with the same
wrong convention, so the comparison could only ever measure sampling noise. A comparison
between two implementations tests correctness only if the implementations differ in the
way that could be wrong. What would have caught this is a cross-VERSION check, or
validation against the model training-time definition -- not another consumer of the same
library. That file now says so about itself.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_012FgR72xAnrpk7XnGwvZTyW