Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Everything is Working Great, quick question.... #19

Closed
MushroomFleet opened this issue Dec 12, 2022 · 27 comments
Closed

Everything is Working Great, quick question.... #19

MushroomFleet opened this issue Dec 12, 2022 · 27 comments
Labels
question Further information is requested

Comments

@MushroomFleet
Copy link

as seen here, (example outputs included)
https://twitter.com/MushroomFleet/status/1602341447952437249
and below (just the merge experiments)

What are your advised Training settings? I did use a bunch that i used in other Dreambooth methods, but it would be interesting to know if you have a recommended setting for LORA because it is a little different.

Thanks for your hard work in bringing this to us :)

MichaelFassbender-LORA-AiNodes

@cloneofsimo
Copy link
Owner

cloneofsimo commented Dec 12, 2022

I really haven't experimented with LoRA to conclude anything, but for me,
using lr=1e-4, prior_preservation = False worked OK. here are 10 LoRA results from training 10 illustration images with 30000 steps.
image

@nitrosocke
Copy link

nitrosocke commented Dec 12, 2022

I haven't done any style training with it (yeah I know, how is this possible?..)
I used it mainly for subject training and refining a few celebrities in base SD, here are my results in conjunction with my dreambooth models:
nitrosocke-lora-comp-01
nitrosocke-lora-comp-04
nitrosocke-lora-comp-05

The settings I used where the following:
--pretrained_model_name_or_path=$MODEL_NAME --weight_name=$WEIGHT_NAME --instance_data_dir=$INSTANCE_DIR --output_dir=$OUTPUT_DIR --instance_prompt="[celeb name]" --resolution=768 --train_batch_size=4 --mixed_precision="fp16" --use_8bit_adam --gradient_accumulation_steps=1 --learning_rate=1e-4 --lr_scheduler="constant" --save_steps=200 --max_train_steps=1000

@pedrogengo
Copy link

@nitrosocke Do you mind to share the process you did to get these results? Im trying to use only LORA to finetune SD, but the results were not that good.

@nitrosocke
Copy link

@pedrogengo Sure thing here is my workflow:

  • Choose 5-10 images of a person
  • Crop/resize to 768x768 for SD 2.1 training
  • Following settings worked for me: train_batch_size=4, mixed_precision="fp16", use_8bit_adam, learning_rate=1e-4, lr_scheduler="constant", save_steps=200, max_train_steps=1000
  • for systems with less VRAM than 24GB use a lower train_batch_size like 2 or 1.
  • make sure to use xformers and 8bit adam
  • for subjects already know to SD images*100 worked great, for subjects unknown to SD more steps or a higher LR are required
  • training on a 3090 takes ~20 min for 1k steps

@pedrogengo
Copy link

@nitrosocke
Did you try with some unknown subjects? My problem is when Im trying to finetune one with images from myself (as you may know, Im not a celeb haha)

And thanks for sharing! What prompt did you use to generate the last two images from your grid?

@nitrosocke
Copy link

@pedrogengo I did try training my wife and the settings didn't work. I haven't done any more testing on that but I assume it's something like 2k steps for 5 images. Would have to try though.

Prompt for Joseph was "modern disney style joseph gordon levitt", it's using my custom model and the JGL training .pt on top.

@MushroomFleet
Copy link
Author

Thanks for all the responses everyone!!

I generally only overwrite known concepts (like actors) with photos of myself. I'll take a look at all the settings you all posted and tweak mine accordingly :)

I'm helping to test a local installed UI implementation that now included LORA dreambooth so any information is very useful for guiding users !!

@cloneofsimo
Copy link
Owner

Now you can also fine-tune CLIP encoder with LoRA as well, just like dreambooth option

Checkout fine-tuning with this shell-code:
https://github.com/cloneofsimo/lora/blob/master/run_lora_db_w_text.sh

Checkout using LoRAs this notebook (on example LoRA i've made): https://github.com/cloneofsimo/lora/blob/master/scripts/run_with_text_lora_also.ipynb

@amrrs
Copy link

amrrs commented Dec 12, 2022

@cloneofsimo Thanks for the work. Do you have any Colab / Notebook for basic training. I currently saw 4 scripts on the scripts folder, but not sure if you got the example there or somewhere else.

Basically, if i've got 4 768x768 images and I want to fine-tune SD2.0, then which is the script I should look at?

@cloneofsimo
Copy link
Owner

You can run:
https://github.com/cloneofsimo/lora/blob/master/run_lora_db_w_text.sh
with few parameters changed. defaults worked OK for me, and I even stopped at 2500 steps because it was overfitting.

@amrrs
Copy link

amrrs commented Dec 12, 2022

Thank you @cloneofsimo sorry for a dumb question. So the input images should be inside this folder which is assigned as the instance directory INSTANCE_DIR="./data_example_text" ?

@pedrogengo
Copy link

Yes @amrrs

@pedrogengo
Copy link

@amrrs I did this Colab notebook if you want to perform all the steps on Colab:

https://colab.research.google.com/drive/1iSFDpRBKEWr2HLlz243rbym3J2X95kcy?usp=sharing

@cloneofsimo If you like, you can update the README with it :)

@Daniel-Kelvich
Copy link

Daniel-Kelvich commented Dec 13, 2022

@pedrogengo is it possible to run colab training with batch_size=5? It is the default setting but I'm getting CUDA_OUT_OF_MEMORY

@amerkay
Copy link

amerkay commented Dec 13, 2022

@pedrogengo is it possible to run colab training with batch_size=5? It is the default setting but I'm getting CUDA_OUT_OF_MEMORY

Which GPU are you using? T4 15GB (standard) or A100 40GB (premium)?

I'm trying to figure out ideal settings for the A100.

@pedrogengo
Copy link

Sorry, I didn't test with batch_size=5. I was using 1 during my experiments. What you can do is use 1 and the gradient_step can be 5, so you update the gradients only after 5 steps, which is the same of use batch_size=5. I will create a field for this info too on Colab

@pedrogengo
Copy link

@amerkay @Daniel-Kelvich I just updated the Colab with Gradient Accumulation Steps! Enjoy :)

@cloneofsimo
Copy link
Owner

Nice work @pedrogengo! I'll update the readme.
Meanwhile it would be nice if there was also text encoder training. Would you like to update it?

@pedrogengo
Copy link

Sure thing! I can to it until EOD

@brian6091
Copy link
Collaborator

brian6091 commented Dec 13, 2022

@pedrogengo @cloneofsimo the script train_lora_dreambooth.py seems to be missing a call for accelerate to manage the accumulation context:

https://huggingface.co/docs/accelerate/v0.13.2/en/package_reference/accelerator#accelerate.Accelerator.accumulate

so I'm not sure passing the parameter will do anything.

@cloneofsimo
Copy link
Owner

Right gradient accumulation doesn't work now because it implicitly updates all other params wrapped inside it. So i removed it.

@brian6091
Copy link
Collaborator

Even if you set return_grad=None or filter the parameters?

@cloneofsimo
Copy link
Owner

Yes I think so. But Im not really used to accelerate package so it was probably wasn't the way to fix it. I'll try to make it work with grad accumulation

@brian6091
Copy link
Collaborator

Just gonna drop a link to more training/tuning discussion here:
#37

@pedrogengo
Copy link

pedrogengo commented Dec 14, 2022

@cloneofsimo I just updated the colab and as a workaround to gradient accumulation I'm making lr / batch_size. I know it is not the best approach but it is better than nothing haha

https://colab.research.google.com/drive/1iSFDpRBKEWr2HLlz243rbym3J2X95kcy?usp=sharing#scrollTo=ZDMXQnKfat1-

@scaraffe
Copy link

Is there a way to train new images with manually added captions?

@brian6091
Copy link
Collaborator

@scaraffe

You can try this colab notebook:
https://github.com/brian6091/Dreambooth

that allows using captions with @cloneofsimo's lora training

@cloneofsimo cloneofsimo added the question Further information is requested label Dec 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

9 participants