Fix silently random-initializing RTDetrModel/SEWDForCTC loads (wrong base_model_prefix) - #48744
Fix silently random-initializing RTDetrModel/SEWDForCTC loads (wrong base_model_prefix)#48744abhinav-phi wants to merge 1 commit into
RTDetrModel/SEWDForCTC loads (wrong base_model_prefix)#48744Conversation
3e9b709 to
2b48a1e
Compare
|
@Cyrilvallez @ydzork friendly ping for a review when you get a chance 🙏 This is a small, self-contained loading fix: the only hand-edited source line is |
…-initialized loads
`RTDetrModel.from_pretrained("PekingU/rtdetr_r18vd")` and
`SEWDForCTC.from_pretrained("asapp/sew-d-tiny-100k")` silently returned a
fully randomly-initialized model (502/502 and 222/222 missing keys
respectively), because the weight loader adds/strips `base_model_prefix`
when crossing the base-model/head-model boundary, and for these families the
declared prefix never matches the actual attribute:
- `RTDetrForObjectDetection` nests the base model in `self.model`, but the
prefix was `"rt_detr"` (materialized as `"rt_detr_v2"`, `"pp_doclayout_v2"`,
`"pp_doclayout_v3"`, `"d_fine"`, `"deimv2"` for the subclasses through the
modular converter). Detection checkpoints store every weight under `model.`,
which was never stripped when loading the base `*Model` classes.
- `SEWDForCTC` nests the base model in `self.sew_d` while the prefix was
`"sew-d"` (with a dash), which can never match a Python attribute name, so
base checkpoints (unprefixed keys) never got the `sew_d.` prefix added.
Align the prefix with the actual attribute (`"model"` in the RT-DETR family,
`"sew_d"` for SEW-D), matching what sibling architectures already do
(`DeformableDetr`/`TableTransformer`/`RfDetr` use `"model"`; `SEW` uses
`"sew"`). For the RT-DETR family the fix lives in `modular_rt_detr.py` and is
propagated to all generated modeling files.
This also un-breaks `model.base_model` for these classes (it used to return
the head model itself), so the generic `test_correct_missing_keys`
(`test_missing_keys = False`) is re-enabled for rt_detr, rt_detr_v2,
pp_doclayout_v2, pp_doclayout_v3, d_fine and deimv2.
Add cross-class save/load regression tests for all seven families, plus
hub-level tests for the exact checkpoints reported in the issue.
Fixes huggingface#48722
2b48a1e to
77e6a1e
Compare
|
[For maintainers] Suggested jobs to run (before merge) run-slow: d_fine, deimv2, pp_doclayout_v2, pp_doclayout_v3, rt_detr, rt_detr_v2, sew_d |
CI recapDashboard: View test results in Grafana |
What does this PR do?
Fixes #48722
RTDetrModel.from_pretrained("PekingU/rtdetr_r18vd")returned a model with all 502 weights randomly initialized (526 unexpected keys), andSEWDForCTC.from_pretrained("asapp/sew-d-tiny-100k")likewise left all 222 weights random — silently, with only a warning.Root cause. When loading across the base-model/head-model boundary, the weight loader adds or strips
base_model_prefix(seerename_source_keyincore_model_loading.py, step 3) based on what the target model's state dict expects. For these families the declared prefix never matches the attribute that actually nests the base model:RTDetrForObjectDetectionnests the base model inself.model, butbase_model_prefixwas"rt_detr"(same forrt_detr_v2,pp_doclayout_v2,pp_doclayout_v3,d_fine,deimv2via the modular converter). Detection checkpoints store every weight undermodel., so loading*Modelstripped nothing → 100% missing/unexpected.SEWDForCTCnests the base model inself.sew_d, butbase_model_prefixwas"sew-d"with a dash — an impossible Python attribute name — so base checkpoints (unprefixed keys) never got thesew_d.prefix added.Fix. Align
base_model_prefixwith the actual attribute ("model"in the RT-DETR family — changed once inmodular_rt_detr.pyand propagated to the generated files —"sew_d"for SEW-D), matching what sibling architectures already do:DeformableDetr,TableTransformer,RfDetrall use"model",SEWuses"sew".Before / after
Verified locally against the exact checkpoints from the issue:
RTDetrModel←PekingU/rtdetr_r18vdclass_embed/bbox_embedheads)RTDetrForObjectDetection← sameSEWDForCTC←asapp/sew-d-tiny-100klm_head.weight/biasmissing, all 220 encoder weights loaded and equal toSEWDModel'sSEWDModel← sameSame class of bug, same fix — also verified on real hub checkpoints for
RTDetrV2Model(PekingU/rtdetr_v2_r50vd: base 0 missing),PPDocLayoutV2/V3Model, and confirmed themodel.-prefixed hub layout ford_fine/deimv2(ustc-community/dfine-*-coco). (ForPP-DocLayoutV2/V3the remaining "missing" entries arenum_batches_trackedbuffers the converted checkpoints do not store;PP-DocLayoutV3additionally has a pre-existing size mismatch ondenoising_class_embedbetween the base model and what its head stores — a modeling inconsistency unrelated to key mapping.)Tests
rt_detr,rt_detr_v2,pp_doclayout_v2,pp_doclayout_v3,d_fine,deimv2,sew_d; tiny configs, no network).@slowhub tests pinning the exact issue repro:RTDetrModelfromPekingU/rtdetr_r50vd,SEWDForCTCfromasapp/sew-d-tiny-100k.test_correct_missing_keys(test_missing_keys = Falsewas needed only becausemodel.base_modelused to return the head model itself with the broken prefix; it now returns the base model and the generic test passes) for rt_detr, rt_detr_v2, pp_doclayout_v2, pp_doclayout_v3, d_fine and deimv2.Locally: full non-slow suites pass for all touched families + controls (
deformable_detr,sew);make qualityclean (ruffcheck/format onsrc+tests,check_copies,check_repo,check_modular_conversion --check_all).@huggingface/transformers-core