add verbosity option for resize_lora.py

[mgz-dev]

• add --verbose flag to print additional statistics in resize_lora_model function
• correct device parameter reference in resize_lora_model function

The verbose print statement will give some additional metrics on the SVD reduction at each layer.

• sum(S) retained is the sum of magnitudes for the resized LoRA's singular values compared against sum of magnitudes for the original LoRA's singular values (reported as a percentage).
• max(S) ratio is the largest magnitude singular value compared against the largest magnitude singular value which was dropped during the resize.

General notes:

• Losing some "information" during resize does not always mean lower quality results, as lowering rank can have a denoising effect.
• Good target values for these metrics are domain specific, and to my knowledge, not known yet for this use case.
• Initial recommendations are to pay more attention to the second metric (max s ratios) than the first. The closer the ratio is to 1, the less accurate the resized rank representation was able to represent that layer.
• Visual inspection of final resulting outputs from LoRA is still the best way to check effectiveness of resized LoRA at this time.

Example of partial output from resizing rank 128 to rank 32:

lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn1_to_k           | sum(S) retained: 71.0%, max(S) ratio: 3.5
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn1_to_out_0       | sum(S) retained: 65.1%, max(S) ratio: 10.2
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn1_to_q           | sum(S) retained: 66.9%, max(S) ratio: 4.4
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn1_to_v           | sum(S) retained: 66.8%, max(S) ratio: 12.4
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn2_to_k           | sum(S) retained: 81.2%, max(S) ratio: 22.3
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn2_to_out_0       | sum(S) retained: 75.9%, max(S) ratio: 16.9
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn2_to_q           | sum(S) retained: 79.3%, max(S) ratio: 18.1
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_attn2_to_v           | sum(S) retained: 81.5%, max(S) ratio: 20.5
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_ff_net_0_proj        | sum(S) retained: 54.2%, max(S) ratio: 2.9
lora_unet_up_blocks_3_attentions_2_transformer_blocks_0_ff_net_2             | sum(S) retained: 66.7%, max(S) ratio: 6.6

[kohya-ss]

Thank you for this! I am not familiar with SVD, so this PR is much appreciated. It will also help many users.