Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training restarts at checkpoint instead of continuing #85

Open
TheTrustedComputer opened this issue Sep 20, 2024 · 1 comment
Open

Training restarts at checkpoint instead of continuing #85

TheTrustedComputer opened this issue Sep 20, 2024 · 1 comment
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@TheTrustedComputer
Copy link

TheTrustedComputer commented Sep 20, 2024

Describe the bug
This seems to be an unintentional regression. If one stops training mid-session and resumes, it starts from the beginning instead of continuing with the saved D and G checkpoints. The initial output shows these files aren't even loaded when they exist in the file system.

To Reproduce
Create any RVC model in the training tab, stop after the first checkpoint, and repeat, using the same parameters.

Expected behavior
Training should continue from the previously saved checkpoint, as RVC always has.

Screenshots
N/A

Desktop (please complete the following information):

  • OS and version: Arch Linux
  • Python version: 3.10.13
  • Commit/Tag with the issue: main

Additional context
This is likely due to the commit 277eadc, which renamed checkpoint files from "2333333" to "latest". Resumption from the latest checkpoint worked in an older commit, but it's broken now.

@fumiama fumiama added bug Something isn't working help wanted Extra attention is needed labels Sep 20, 2024
@fumiama
Copy link
Owner

fumiama commented Sep 20, 2024

Maybe a digital epoch name is needed to resume in current code. We should modify the code and let the program check the name latest first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants