SD3 cannot finetunes a better model (hand and face deformation)?

[KaiWU5]

Describe the bug

I want to finetune sd3 to improve its human generation quality with 3million high-quality human datasets (which has been proven useful on sdxl and other models). But hand and face deformation doesn't improve much after two days of training.

I am using train script

What I have been done so far:

1. regular training with 3 million data with batch size 2x24(V100) for 2 epochs with lr 5e-6 and adamw optimizer
2. prodigy optimizer training with same setting
3. Add q,k RMS norm to each attention layer
4. only train several blocks

All of my training gives me nearly the same deformation results, where the hands are never normal like human.

Could you some provide more experiments about sd3 training? There seems no easy way to adapt sd3 for human generation

Reproduction

Has described in bug part

Logs

No response

System Info

V100 24GPU, batchsize 2 for each card, 3 million human data with aesthetic score > 4.5

Who can help?

No response

[DN6]

Hi @KaiWU5 I think this question would be better to ask in the Discussions section.

[mliand]

You can show me your loss training

[heart-du]

I have the same question.

[KaiWU5]

Here's my loss curve. The loss after adding qknorm is similar. Thanks for noticing and I will continue discussing the results in the discussion section.

[xduzhangjiayu]

Here's my loss curve. The loss after adding qknorm is similar. Thanks for noticing and I will continue discussing the results in the discussion section.

Your loss curve is same as I trained sd3 controlnet, have you solve this issue or do you have any suggestions? Thanks!

[appleyang123]

+1, I have the same issue.

[CodeExplode]

Have you tried training SD3.5 rather than SD3? The original model didn't ship with some RMS Norm layers which may have been important for stabilizing training.

I found that when training SD3 (the original), I got much better results just training the context path, specifically the post-attention parameters of the context path. It seems these models can 'draw' almost anything, it's just a matter of finding the correct way to speak to them to ask, and training the context path changes how it processes the prompt into a format which the rest of the model can understand and respond to.

[xduzhangjiayu]

Have you tried training SD3.5 rather than SD3? The original model didn't ship with some RMS Norm layers which may have been important for stabilizing training.

I found that when training SD3 (the original), I got much better results just training the context path, specifically the post-attention parameters of the context path. It seems these models can 'draw' almost anything, it's just a matter of finding the correct way to speak to them to ask, and training the context path changes how it processes the prompt into a format which the rest of the model can understand and respond to.

Hi, could you please explain which is the context path in SD3? Thanks!

[CodeExplode]

In SD3 and Flux, there are parts of the model which deal with understanding the prompt (the context path), and parts for understanding the image. I found that in the original SD3, just training the parts which relate to understanding the prompt was enough to teach about any concept. It's just a matter of making sure the model understands what you're asking, and not needing to train how it handles the image segments. It takes a bit of manual coding though.