Convert kohya/CivitAI LoRA safetensors for Anima#1
Open
ghunkins wants to merge 2 commits into
Open
Conversation
CivitAI Anima LoRAs are trained with kohya-ss/sd-scripts which writes keys
as `lora_unet_<munged_name>.lora_{down,up}.weight` (+ optional `.alpha`),
with dots in module paths replaced by underscores. The existing Anima
LoRA loader only handled the `diffusion_model.*` prefix with peft-style
`lora_A`/`lora_B` suffixes, so kohya keys slipped through `load_lora_weights`
silently. No PEFT adapter was registered for the file, surfacing later as
`Adapter name(s) {...} not in the list of present adapters` from
`set_adapters`.
Adds `_convert_kohya_anima_lora_to_diffusers`, wires it into
`AnimaLoraLoaderMixin.lora_state_dict`, and extends the existing
`diffusion_model.*` converter to also accept kohya `.lora_down`/`.lora_up`/
`.alpha` suffixes (some trainers combine the two). The kohya converter
routes DiT blocks to `transformer.*`, LLM-adapter blocks to
`text_conditioner.*` (preserving `o_proj` and `mlp.{0,2}`, which differ
from the DiT's `output_proj` and `mlp.{layer1,layer2}` — verified against
the live anima-base-v1.0 safetensors header), and skips Qwen3
`lora_te_*` keys with a warning since the Anima loader does not load
LoRAs into the text encoder.
Adds `test_kohya_lora_state_dict_conversion` and
`test_load_lora_weights_kohya_format`.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
`utils/check_support_list.py` requires every LoRA mixin defined in `src/diffusers/loaders/lora_pipeline.py` to appear in `docs/source/en/api/loaders/lora.md`. Add the missing `AnimaLoraLoaderMixin` entry and `[[autodoc]]` block. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Author
|
Still testing, but here is a more robust conversion script: https://gist.github.com/ghunkins/05839b2e6b0111be3a98ba2af13e3dda |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CivitAI Anima LoRAs are trained with kohya-ss/sd-scripts, which writes keys as
lora_unet_<munged_name>.lora_{down,up}.weight(+ optional.alpha), with dots in module paths replaced by underscores. The existingAnimaLoraLoaderMixin.lora_state_dictonly handled thediffusion_model.*prefix with peft-stylelora_A/lora_Bsuffixes, so kohya-format keys slipped throughload_lora_weightssilently and no PEFT adapter got registered. The user then sees a confusingAdapter name(s) {...} not in the list of present adapterslater fromset_adapters.Changes
_convert_kohya_anima_lora_to_diffusers(new) inlora_conversion_utils.py:.lora_down.weight/.lora_up.weight/.alphaby base key.lora_unet_blocks_{i}_*totransformer.transformer_blocks.{i}.*(full self/cross attn, MLPlayer1/layer2, adaLN modulation).lora_unet_llm_adapter_blocks_{i}_*totext_conditioner.blocks.{i}.*. Note: the LLM adapter useso_proj(not the DiT'soutput_proj) andmlp.{0,2}(not the DiT'smlp.{layer1,layer2}). Verified against the livecirclestone-labs/Anima/anima-base-v1.0.safetensorsheader.x_embedder_proj_1,t_embedder_1_linear_{1,2},final_layer_linear,final_layer_adaln_modulation_{1,2}.alpha / rankscaling, distributing across down/up to avoid overflow (same formula as_convert_kohya_flux_lora_to_diffusers).lora_te_*Qwen3 text-encoder keys with a warning since the Anima loader does not load LoRAs into the text encoder._convert_non_diffusers_anima_lora_to_diffusers(extended): also accepts kohya.lora_down/.lora_up/.alphasuffixes when the prefix isdiffusion_model.(some trainers combine the two).AnimaLoraLoaderMixin.lora_state_dict: dispatches to the kohya converter when any key starts withlora_unet_orlora_te_.Test plan
test_lora_state_dict_conversion(existing) — still passes.test_load_lora_weights(existing) — still passes.test_kohya_lora_state_dict_conversion(new) — covers DiT, LLM adapter, alpha scaling, and the TE-skip path.test_load_lora_weights_kohya_format(new) — end-to-end PEFT registration with kohya keys.anima-base-v1.0.safetensors: 515/516 Linear modules round-trip cleanly; the 1 skipped isllm_adapter.embed(nn.Embedding), which kohya excludes by default anyway.🤖 Generated with Claude Code