Finetuning trained model

[aleksas]

In reference to recommendation how should repeated training be executed for the second time?
Should checkpoint path point to checkppoints/checkpoint_step...._ema.pth or not ema checkpoint? And what does ema stands for?

[r9y9]

EMA means Exponential Moving Average (EMA). See details for Tacotron 2 paper: https://arxiv.org/abs/1712.05884. There's no clear answer which to use for finetuning, but I usually used EMA version of the checkpoint when I trained a model for sufficient time (e.g, over 2 days for MoL case).

I'll leave some commands I used for LJSpeech experiments which might help as follows:

• Finetuning trained model

python train.py --data-root=./data/ljspeech/ \
  --checkpoint-dir=./checkpoints_mixture \
  --log-event-path=./log/wavenet_mixture \
  --restore-parts=./pretrained_models/20180127_mixture_lj_checkpoint_step000410000_ema.pth

• Accidentally training aborted (my PC was rebooted), then resuming

python train.py --data-root=./data/ljspeech/ \
  --checkpoint-dir=./checkpoints_mixture \
  --log-event-path=./log/wavenet_mixture \
  --checkpoint=./checkpoints_mixture/checkpoint_step000256701.pth

The two commands above were actually used for training the pre-traind model.

[aleksas]

Thanks! Will try to fine tune my model now.

[d2sys]

Anyone can share the minimum number of iterations which is enough for fine-tuning?