Loading RTDetrModel from a detection checkpoint (or SEWDForCTC from a SEW-D checkpoint) gives a model with every weight randomly initialized #48722

System Info

- `transformers` version: 5.18.0.dev0
- Platform: Linux-7.0.0-31-generic-x86_64-with-glibc2.39
- Python version: 3.10.20
- Huggingface_hub version: 1.15.0
- Safetensors version: 0.8.0
- Accelerate version: 1.13.0
- Accelerate config: not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.12.0+cu130 (CUDA)
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: no, the script below is CPU-only
- GPU type: NVIDIA GeForce RTX 3060

Who can help?

@Cyrilvallez @molbap

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I wanted RT-DETR's encoder features without the detection head, so I ran the RTDetrModel docstring example: load the base model from PekingU/rtdetr_r50vd. It runs and returns the right shapes, but none of the weights come from the checkpoint — all 502 are reported missing and re-initialized. AutoModel does the same. The other direction is broken too: SEWDForCTC from the pretrained asapp/sew-d-tiny-100k starts from random weights, so a CTC fine-tune silently starts from scratch.

main at 9c944e9762, CPU:

import torch
from transformers import AutoModel, RTDetrForObjectDetection, RTDetrModel, SEWDForCTC, SEWDModel

for cls, repo in (
    (AutoModel, "PekingU/rtdetr_r18vd"),
    (RTDetrModel, "PekingU/rtdetr_r18vd"),
    (SEWDForCTC, "asapp/sew-d-tiny-100k"),
    (SEWDModel, "asapp/sew-d-tiny-100k-ft-ls100h"),
):
    model, info = cls.from_pretrained(repo, output_loading_info=True)
    n = len(model.state_dict())
    print(f"{type(model).__name__:14} <- {repo:34} missing {len(info['missing_keys']):3}/{n:<3} unexpected {len(info['unexpected_keys']):3}  base_model_prefix={model.base_model_prefix!r}")

head = RTDetrForObjectDetection.from_pretrained("PekingU/rtdetr_r18vd")
print("the head stores the base model under:", [n for n, m in head.named_children() if isinstance(m, RTDetrModel)])

RTDetrModel    <- PekingU/rtdetr_r18vd               missing 502/502 unexpected 526  base_model_prefix='rt_detr'
RTDetrModel    <- PekingU/rtdetr_r18vd               missing 502/502 unexpected 526  base_model_prefix='rt_detr'
SEWDForCTC     <- asapp/sew-d-tiny-100k              missing 222/222 unexpected 220  base_model_prefix='sew-d'
SEWDModel      <- asapp/sew-d-tiny-100k-ft-ls100h    missing 220/220 unexpected 222  base_model_prefix='sew-d'
the head stores the base model under: ['model']

The last line is the reason. The head keeps its base model as self.model but declares base_model_prefix = "rt_detr".
The loader adds or strips that name to move weights between a base checkpoint and a head
(core_model_loading.py:1473-1480), so a name that matches no attribute can never line the keys up: every weight is missing on one side and unexpected on the other. SEW-D declares "sew-d", which cannot even be an attribute name. DETR declares "model" and loads both ways.

The same mismatch is in every model built on RT-DETR (rt_detr_v2, d_fine, deimv2, pp_doclayout_v2/v3, the pp_ocrv5/pp_ocrv6 detectors — AutoModel on PekingU/rtdetr_v2_r18vd and ustc-community/dfine-small-coco loses 505/505 and 740/740), in clipseg, timesfm, CLIP's and MetaCLIP-2's …WithProjection wrappers and VideoPrism's video models. It has been so since each model was added; nothing raises, only the usual "newly initialized" warning, so it went unnoticed.

Expected behavior

Loading a base model from a head checkpoint, or a head from a base checkpoint, should give every shared
weight — as for DETR and BERT — with only the head's own parameters reported missing.

This is quiet but wide: rt_detr has 167 Hub repos (top one 742k downloads/month), rt_detr_v2 186 (top
1.6M), d_fine 95, plus anyone fine-tuning SEW-D for CTC from the pretrained checkpoints.

I have the fix — the prefix constant corrected in each of these models — plus a common test that loads a
base checkpoint into every head and checks the weights arrive: red on main, green with the fix, the rest of
the suites unchanged. Glad to open the PR if a maintainer wants it.