Question about LoHa and LoKr support during inference.

[Scorpinaus]

Hello Diffusers Team, great work on the Lora and lycoris support.

I see that there are newer formats such as LoHa and LoKr, which is supported in PEFT for training. Furthermore, there are some loha loras which can be used for SD1.5 and SDXL. Is loading of Loha and lokr formats supported when we load them via peft for inference?

Thank you!

[sayakpaul]

It should be possible via but not via load_lora_weights(). LoKR could be supported but haven't verified it.

Let me elaborate this a bit.

Let's say you have conducted LoHA or LoKR training on SD/SDXL via peft.

Now, to be able to peft formatted checkpoint during inference, you need to do something like:

inference_model = PeftModel.from_pretrained(model, repo_name)

where the model is the base model on which PEFT training was conducted and repo_name is the identifier of the Hub repository that has the PEFT checkpoint.

Once this is done, you can do:

pipeline.unet = inference_model

(assuming you only conducted PEFT training on the unet).

Does this help?

[Scorpinaus]

Hi Sayakapaul,

Thank you for the response. Does this mean LoHA/LoKr that is not trained on peft will likely not work with this alternate method?

This also implies the loading method would affect the unet but not on the clip/textual encoder layer?

[sayakpaul]

Loading method (load_lora_weights()) should affect both UNet and the text encoders when the underlying adapter checkpoint has keys for both.

Does this mean LoHA/LoKr that is not trained on peft will likely not work with this alternate method?

Alternate method, meaning?

[BenjaminBossan]

Note that we have a script to convert kohya checkpoints to PEFT format: https://github.com/huggingface/peft/blob/main/examples/stable_diffusion/convert_sd_adapter_to_peft.py

[Scorpinaus]

Hi sayakapaul, thank you for answering. Had some doubt there if it only affects the UNet layers or both. Is it possible at this current time to affect the clip/textual encoder layer and unet by different amounts during inference time using the PEFT integrated loading method? Similar to how its possible in comfy?

For the first question, thank you BenjaminBossan for raising up that there's a conversion script. So technically once we convert the kohya checkpoints to PEFT format, we can load it using the integrated PEFT adapter loading?

[sayakpaul]

Had some doubt there if it only affects the UNet layers or both.

Do you mean during training? With peft, you do have the option to target both UNet and text encoder, like so (an example):

from peft import LoHaConfig

unet_loha_config = LoHaConfig(
    r=args.rank,
    target_modules=["to_k", "to_q", "to_v", "to_out.0", "add_k_proj", "add_v_proj"],
)
unet.add_adapter(unet_loha_config)

text_loha_config = LoraConfig(
    r=args.rank,
    target_modules=["q_proj", "k_proj", "v_proj", "out_proj"],
)
text_encoder.add_adapter(text_loha_config)

And then proceed with training accordingly. Here's a training script you could refer to.

• LoHA documentation
• LoKr documentation

Is it possible at this current time to affect the clip/textual encoder layer and unet by different amounts during inference time using the PEFT integrated loading method? Similar to how its possible in comfy?

I am not sure if I understand this. Could you elaborate? Do you mean you weigh the outputs coming from the text encoder differently than the outputs of the UNet?

So technically once we convert the kohya checkpoints to PEFT format, we can load it using the integrated PEFT adapter loading?

I think there's some confusion here. load_lora_weights(), under the hood, might error out when exposed to certain state dicts that don't respect the checks we have under load_lora_weights().

What this script does is that it yields a model with the following workflow:

• the supplied adapter is first converted to the peft model format.
• serializes the adapter in the peft format.

To be able to use the output adapter models in diffusers, you could do:

from diffusers import StableDiffusionPipeline
from peft import PeftModel

pipeline = StableDiffusionPipeline.from_pretrained("pipeline-id-or-path")

unet_adapter_id = "path-to-the-unet-adapter"
text_encoder_id = "path-to-the-text-encoder-adapter"
pipeline.unet = PeftModel.from_pretrained(pipeline.unet, unet_adapter_id)
pipeline.text_encoder = PeftModel.from_pretrained(pipeline.text_encoder, text_encoder_id)

image = pipeline(prompt).images[0]

@BenjaminBossan I hope I didn't provide any incorrect information here.

[BenjaminBossan]

I hope I didn't provide any incorrect information here.

No, this looks quite correct to me.

Just to clarify, I linked the conversion script in case the user wants to use kohya LoHa/LoKr weights in PEFT. If they train their own adapters with PEFT, running the script is of course not necessary.