restart chgnet training (finetune)

[chtchelkatchev]

Is there possibility to restart training (finetune) of chgnet?
Due to supercomputer limitations I can use A100 gpus only 24 hours, after that I must restart any calculations. 24 hours is not enough sometimes for training (finetune) chgnet.

Is there possiblity to use several gpu for training (if this speeds up)?

[chtchelkatchev]

Four finetune I tried the following procedure (is it correct?):
trainer = Trainer(
model=chgnet,
targets="efsm",
optimizer="Adam",
criterion="MSE",
learning_rate=8e-3,
epochs=1200,
use_device="cuda",
starting_epoch=803,
scheduler="CosRestartLR",
)

Here 803--is the last epoch of the previous finetuning. CosRestartLR repalces CosLR.

[BowenD-UCB]

You can try directly loading the trainer.

This way you don't have to redefine parameters.
Additionally, CosRestartLR is not related to reloading CosLR.