Fix CLIPTextModel compatibility with transformers > 5.5#1548
Fix CLIPTextModel compatibility with transformers > 5.5#1548emanuelenurra78-byte wants to merge 3 commits into
Conversation
In transformers 5.x, CLIPTextModel was flattened: the .text_model sub-module no longer exists. Layers (embeddings, encoder, final_layer_norm) are now direct attributes of CLIPTextModel itself. This broke SD1.5 (and SDXL) loading via diffusers convert_from_ckpt.py, which calls accelerate set_module_tensor_to_device() with parameter paths containing text_model.* names. The fix adds a text_model property to CLIPTextModel (if absent) that returns self, so all existing code paths work on both transformers 4.x and 5.x. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
|
There is more to it. See here for details: #1506 If you think this will fix all issues mentioned there or are willing to add more fixes, please extend this PR. Otherwise, please close it. I changed your headline because we have already upgraded to transformers 5.5. |
The previous patch only handled the 5.6+ flat layout (text_model removed). With transformers 5.5.x (nested layout), diffusers 0.38+ still accesses embeddings/encoder/final_layer_norm directly on CLIPTextModel, which fails because those attrs live under .text_model on the instance. Now detects the transformers version at startup: - >=5.6 (flat): adds text_model = self property (unchanged) - <=5.5 (nested): exposes embeddings/encoder/final_layer_norm as flat properties forwarding to self._modules["text_model"] to avoid circular lookups through the property itself. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Backups created with transformers >=5.6 use a flat CLIPTextModel layout where the key-conversion table cannot match the nested path, leaving the TE LoRA keys absent from the saved state dict. Loading with strict=True raises a RuntimeError on resume; strict=False keeps those layers at zero initialisation (same as training from scratch) and lets the run proceed. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
|
Thanks for the detailed pointer — agreed on all counts. You're right that this is the "shim reproducing the old non-flat CLIP" option, which #1506 already evaluated and marked Skip. The shim doesn't address loading (#1) or SD1.x single-file (#6), and it carries the recurring-maintenance cost you flagged. On top of that, the Since the project is pinned to transformers 5.5.4 (#1524), there's no active >=5.6 breakage to fix here, and the proper fix is the deferred 6-item plan in #1506, which is yours to own. Closing this rather than half-implementing a discarded approach. The WIP stays on the branch if any of it is useful later. 🤖 Addressed by Claude Code |
What changed
Added a
text_modelcompatibility shim inscripts/util/import_util.pythat patchesCLIPTextModelat startup.Why
In transformers 5.x,
CLIPTextModelwas flattened: the.text_modelsub-module no longer exists as a named child. The layers that previously lived undertext_model(embeddings, encoder, final_layer_norm) are now direct attributes of the model itself.This silently broke loading of SD 1.5 and SDXL checkpoints because
diffusers.pipelines.stable_diffusion.convert_from_ckptpasses parameter names liketext_model.embeddings.position_idstoaccelerate.utils.set_module_tensor_to_device(), which then tries to resolvetext_modelas a sub-module and raises:Fix
Adding
text_model = property(lambda self: self)makesmodel.text_modelreturn the model itself, so all existing attribute paths resolve correctly. Thehasattrguard means this is a no-op on transformers 4.x where.text_modelalready exists.Testing
Verified with
transformers==5.12.1,diffusers==0.38.0,accelerate==1.8.1on Python 3.14 (Windows): SD 1.5.safetensorscheckpoint loads and training starts successfully after this patch.