Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debias Estimation loss #889

Merged
merged 16 commits into from
Oct 23, 2023
Merged

Debias Estimation loss #889

merged 16 commits into from
Oct 23, 2023

Conversation

sdbds
Copy link
Contributor

@sdbds sdbds commented Oct 19, 2023

paper:https://arxiv.org/pdf/2310.08442.pdf

just change for SNR weight like min-snr-gamma

@vvern999
Copy link
Contributor

It seems you forgot to add

                   if args.debiased_estimation_loss:
                        loss = apply_debiased_estimation(loss, timesteps, noise_scheduler)

in train_network.py

@sdbds
Copy link
Contributor Author

sdbds commented Oct 20, 2023

It seems you forgot to add

                   if args.debiased_estimation_loss:
                        loss = apply_debiased_estimation(loss, timesteps, noise_scheduler)

in train_network.py

thank u for reviewing,i add it.

@kohya-ss kohya-ss merged commit 202f2c3 into kohya-ss:dev Oct 23, 2023
1 check passed
@kohya-ss
Copy link
Owner

Thank you for this!

I only run the test training yet, but it seems to be very promising. I will do more testing and merge this into main.

@feffy380
Copy link
Contributor

@sdbds any idea what adjustment needs to be made for v_prediction and zero_terminal_snr? ZSNR in particular will break the weight calculation due to division by zero

@sdbds
Copy link
Contributor Author

sdbds commented Oct 31, 2023

v_prediction and zero_terminal_snr

i think v_prediction and zero_terminal_snr used in SD V2 so it cant use this at the same time...

for some histroy,not all loss function implement in all script,so i think just not use this in SD V2 model.

@FurkanGozukara
Copy link

What is this and how to use with sdxl or sd 1.5 can you give some more info please?

@sdbds
Copy link
Contributor Author

sdbds commented Nov 5, 2023

What is this and how to use with sdxl or sd 1.5 can you give some more info please?

u can think it is min-SNR plus version....

@FurkanGozukara
Copy link

What is this and how to use with sdxl or sd 1.5 can you give some more info please?

u can think it is min-SNR plus version....

sadly i don't know that too

could you give little info? and how to use?

@sdbds
Copy link
Contributor Author

sdbds commented Nov 5, 2023

What is this and how to use with sdxl or sd 1.5 can you give some more info please?

u can think it is min-SNR plus version....

sadly i don't know that too

could you give little info? and how to use?

You can imagine it as an operation that automates the processing of noise, allowing for faster model fitting, as well as balancing out some of the color issues.

@FurkanGozukara
Copy link

What is this and how to use with sdxl or sd 1.5 can you give some more info please?

u can think it is min-SNR plus version....

sadly i don't know that too
could you give little info? and how to use?

You can imagine it as an operation that automates the processing of noise, allowing for faster model fitting, as well as balancing out some of the color issues.

thank you so much

@FurkanGozukara
Copy link

FurkanGozukara commented Nov 6, 2023

doing a SDXL DreamBooth training and testing debiased_estimation_loss right now
i hope it further improves

@FurkanGozukara
Copy link

The "debiased_estimation_loss" led to overtraining.

Interestingly, the effect of this was the amplification of a constant learning rate (LR) significantly.

Also, my U-Net combined with TE1 training is producing NaNs - exactly same settings of U-NET only

@araleza
Copy link

araleza commented Nov 17, 2023

Hello, I don't know much about this new debiased_estimation_loss, but I wanted to ask if it was relevant to a training issue I've been seeing:

Any training images I have with large blocks of (e.g.) red and blue areas, such as someone wearing bright red clothing in front of a bright blue wall, almost immediately look terrible when Dreambooth training is performed. Other images without these blocks of colors look fine, so it doesn't seem to be due to overtraining.

It doesn't seem to be a fundamental issue with Stable Diffusion, as if I use the base SDXL model and prompt a picture that produces a person with bright red clothing in front of a bright blue wall, the image looks great. I just can't train any new images of this type with sd-scripts.

Does this effect sound related to your work?

@sdbds
Copy link
Contributor Author

sdbds commented Nov 18, 2023

All of the current noise-related parameters have a noticeable effect on color contrast, i dont ensure if it had affect training Here are some examples of parameters that have a significant impact:

$noise_offset $multires_noise_iterations $min_snr_gamma $ip_noise_gamma $debiased_estimation_loss

Additionally, only one of these three parameters can generally be chosen for use. $min_snr_gamma $ip_noise_gamma $debiased_estimation_loss

All of the current noise-related parameters have a noticeable effect on color contrast, i dont ensure if it had affect training
Here are some examples of parameters that have a significant impact:

$noise_offset
$multires_noise_iterations
$min_snr_gamma
$ip_noise_gamma
$debiased_estimation_loss

Additionally, only one of these three parameters can generally be chosen for use.
$min_snr_gamma
$ip_noise_gamma
$debiased_estimation_loss

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants