Add lora-embedding bundle system

[KohakuBlueleaf]

Description

Some Trainer (like hcp-diffusion, naifu) have implemented a method of training called "pivtoal tuning", which is basically trained Embedding and LoRA(DreamBooth) at the same time.
But if the trainer want to train multiple concept within one model, it will produce lot of embedding files which is not very good for user and trainer to managing them.

So I developed this bundle system which can store the bundled embedding withing the lora files and load them with built-in lora extension.

The state_dict key name format:
"bundle_emb.EMBEDDING_NAME.KEY_NAME"
The KEY_NAME here means the "key" in the embedding state_dict, the EMBEDDING_NAME means the "trigger word" of that embedding. With this format we can store multiple embedding within 1 lora file, which is good for pivotal tuning.

Thx to @narugo1992 and @cyber-meow for this idea.

Checklist:

• I have read contributing wiki page
• I have performed a self-review of my own code
• My code follows the style guidelines
• My code passes tests

[narugo1992]

Yes, indeed. Pivotal tuning is a proven effective training method.

Check out my Civitai profile: narugo1992 on Civitai, where over 500 models were trained using LoRA+pt. User feedback indicates excellent quality, currently ranking second on the site.

Clearly, pivotal tuning has demonstrated its effectiveness and potential.

Additionally, we conducted a quantitative analysis of existing pivotal tuning, referenced in this article: Article on pivotal tuning analysis.

[cyber-meow]

I have written a post in civitai to explain the benefits of pivotal tunining
https://civitai.com/articles/2494/making-better-loras-with-pivotal-tuning

Not to mention that this seems to be the go-to method for many papers:

[1] Kumari, N., Zhang, B., Zhang, R., Shechtman, E., & Zhu, J. Y. (2023). Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1931-1941).
[2] Smith, J. S., Hsu, Y. C., Zhang, L., Hua, T., Kira, Z., Shen, Y., & Jin, H. (2023). Continual diffusion: Continual customization of text-to-image diffusion with c-lora. arXiv preprint arXiv:2304.06027.
[3] Gu, Y., Wang, X., Wu, J. Z., Shi, Y., Chen, Y., Fan, Z., ... & Shou, M. Z. (2023). Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models. arXiv preprint arXiv:2305.18292.
[4] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personalization. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1–11, 2023.

[KohakuBlueleaf]

Need to add support for strin_to_param dict
draft now

[AUTOMATIC1111]

Any files to test this on?

The huge copied block of code in networks.py definitely should not be there. The code in textual_inversion.py should be placed into a function and that function should be used instead. Although if there is an example file to play with I would do it myself.

[KohakuBlueleaf]

@AUTOMATIC1111 Here is the demo of lora-emb bundle for this model

rosmontis_arknights_bundle.zip
(Github dont' allow me to upload .safetensors so I made zip)