-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to resume training from checkpoint? #63
Comments
This is not the expected behavior. Are you placing different models such as 4.0 v2? Please post the output as there is not enough information. |
Can you please post the output of svc train? This is a bug. |
I think there is a misunderstanding. I'm not making a bug report, I'm asking if continuing training is possible? Here is the stack trace:
When training again, it overwrites the old checkpoints: Is it possible to continue training from the latest checkpoint |
Resuming the training from the last model is the expected behavior. If updating does not resolve the issue, please make sure to reopen. Thank you. |
IT doesnt resolve the issue, so ive lost 6 hours of training , you cant resume and it starts from 0 when you stop training and restart it, you should really have extra info on how to resume or extra argument for resuming , also whats the -t next to train argument ? where are the docs about it ? |
Right now if the training gets interrupted, I have to start over from scratch. Is there a way to continue training from the latest checkpoint?
The text was updated successfully, but these errors were encountered: