Support for OFT (and problem with bundle format)

[cyber-meow]

OFT from Controlling Text-to-Image Diffusion by Orthogonal Finetuning is now working in webui after Kohaku fixes inference: AUTOMATIC1111/stable-diffusion-webui#14300

To make it work with OFT, I initially tried adding "NetworkModuleOFT": "oft_blocks" to LORAANDSOON. This runs but does not give the good results because OFT performs multiplication instead of addition.
What really fixes the problem is to add this in line 981

if ltype == "NetworkModuleOFT":
    oft_parameters = torch.nn.Parameter(getattr(lora.modules[key],LORAANDSOON[ltype]))
    I = torch.eye(oft_parameters.size()[1]).to(oft_parameters.device)
    oft_parameters = oft_parameters * ratio + I * (1-ratio)
    setattr(lora.modules[key], "oft_blocks", oft_parameters)
    set = True

Illustration:

This is from this OFT bundle: https://civitai.com/models/173081

---

By the way, the same issue for bundle format (i.e. an additional network with built-in embeddings) that I mentioned earlier for regional prompter hako-mikan/sd-webui-regional-prompter#258 applies to LBW as well. This is not so critical but may worth some investigation if you have some time.

Thank you for the great plugins in any case.

[KohakuBlueleaf]

I recommend to modify scale instead of oft_blocks since kohya/lycoris have different implementation and have different math form for oft_blocks

[hako-mikan]

Fixed.
Thanks @KohakuBlueleaf for advice.

[cyber-meow]

Indeed. Both issues are solved now. Thanks.