Show the moving average loss

[shirayu]

This PR replaces the loss with the moving average of loss.
Current calculation of loss for logging resets at the begging of each epoch.
This causes jumping of loss.

Images

Before

After

[kohya-ss]

Thank you for this! Moving average seems to be good.
However, I think it will be better that loss/epoch is the average loss of the epoch, not the average loss of entire training.

In addition, it is a bit annoying to calculate the sum of losses every step, if the epoch has thousands of steps. TensorBoard has a 'smoothing' feature, so it will be acceptable raw loss values instead of average...

[shirayu]

loss/epoch is the same value to the moving average because window size is the number of steps of an epoch.

Surely, we can also watch the raw loss value and smoothed loss value in TensorBoard.

My first intention of this PR was the improvement of CLI loss indication.
I prefer that the loss value displayed on the screen with the progress bar be continuous.

As you point out, it is wasteful to sum every step.
I will try improving it a little.

[shirayu]

I removed the call of sum() at 8aed512.

[kohya-ss]

loss/epoch is the same value to the moving average because window size is the number of steps of an epoch.

Oh, sorry, I misunderstood.

I prefer that the loss value displayed on the screen with the progress bar be continuous.

I agree this.

The update looks good! I will merge as soon as I have time.