Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training Error: ZeroDivisionError: division by zero #14

Closed
Ionaut opened this issue Aug 20, 2024 · 11 comments
Closed

Training Error: ZeroDivisionError: division by zero #14

Ionaut opened this issue Aug 20, 2024 · 11 comments

Comments

@Ionaut
Copy link

Ionaut commented Aug 20, 2024

image

Traceback (most recent call last):
File "D:\pinokio\api\StyleTTS-WebUI.pinokio.git\app\venv\lib\site-packages\gradio\queueing.py", line 536, in process_events
response = await route_utils.call_process_api(
File "D:\pinokio\api\StyleTTS-WebUI.pinokio.git\app\venv\lib\site-packages\gradio\route_utils.py", line 288, in call_process_api
output = await app.get_blocks().process_api(
File "D:\pinokio\api\StyleTTS-WebUI.pinokio.git\app\venv\lib\site-packages\gradio\blocks.py", line 1931, in process_api
result = await self.call_function(
File "D:\pinokio\api\StyleTTS-WebUI.pinokio.git\app\venv\lib\site-packages\gradio\blocks.py", line 1516, in call_function
prediction = await anyio.to_thread.run_sync( # type: ignore
File "D:\pinokio\api\StyleTTS-WebUI.pinokio.git\app\venv\lib\site-packages\anyio\to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "D:\pinokio\api\StyleTTS-WebUI.pinokio.git\app\venv\lib\site-packages\anyio_backends_asyncio.py", line 2177, in run_sync_in_worker_thread
return await future
File "D:\pinokio\api\StyleTTS-WebUI.pinokio.git\app\venv\lib\site-packages\anyio_backends_asyncio.py", line 859, in run
result = context.run(func, *args)
File "D:\pinokio\api\StyleTTS-WebUI.pinokio.git\app\venv\lib\site-packages\gradio\utils.py", line 826, in wrapper
response = f(*args, **kwargs)
File "D:\pinokio\api\StyleTTS-WebUI.pinokio.git\app\venv\lib\site-packages\gradio\utils.py", line 826, in wrapper
response = f(*args, **kwargs)
File "D:\pinokio\api\StyleTTS-WebUI.pinokio.git\app\webui.py", line 716, in start_training_proxy
run_train(config_path)
File "D:\pinokio\api\StyleTTS-WebUI.pinokio.git\app\venv\lib\site-packages\styletts2\train_finetune_accelerate.py", line 725, in main
logger.info('Validation loss: %.3f, Dur loss: %.3f, F0 loss: %.3f' % (loss_test / iters_test, loss_align / iters_test, loss_f / iters_test) + '\n\n\n')
ZeroDivisionError: division by zero

Win 11 RTX3060 12GB VRAM

@JarodMica
Copy link
Owner

Was this at the beginning of training or was training running, and then you got this later into the training session? I've never seen this before on my end.

@Ionaut
Copy link
Author

Ionaut commented Aug 20, 2024

Right at the beginning.

@JarodMica
Copy link
Owner

Can you check your training and validation phonemes text files to see if they have contents in them?

Division by zero here means some issue with validation as iter_test is zero, meaning that it's either getting no data or something else is occuring. After verifying that, we can try a few things

@Ionaut
Copy link
Author

Ionaut commented Aug 21, 2024

So train.txt and train_phoneme.txt have data in them but validation.txt and validation_phoneme.txt do not and are completely blank

@JarodMica
Copy link
Owner

Ok, so looks like the issue is the dataset creation. How many lines are inside of train_phoneme? It's possible there isn't enough data to fulfill the 5% split that goes into validation_phoneme, though that should occur if you have more than 20 segments.

What you can you is cut and paste 5 or so lines from train_phoneme into validation_phoneme and that would allow you to train, but I'm very interested in how much data you are using to train based off of.

@Ionaut
Copy link
Author

Ionaut commented Aug 21, 2024

I'm preparing a larger set. I'll let you know!

@Deadstarrr
Copy link

Deadstarrr commented Aug 21, 2024

Ok, so looks like the issue is the dataset creation. How many lines are inside of train_phoneme? It's possible there isn't enough data to fulfill the 5% split that goes into validation_phoneme, though that should occur if you have more than 20 segments.

What you can you is cut and paste 5 or so lines from train_phoneme into validation_phoneme and that would allow you to train, but I'm very interested in how much data you are using to train based off of.

The wav file I'm using for training is only a minute long and I get this error. What is the recommended length in minutes for the wav file that I'm using for training? What's the best range to get good results after training?

@JarodMica
Copy link
Owner

Ok, so looks like the issue is the dataset creation. How many lines are inside of train_phoneme? It's possible there isn't enough data to fulfill the 5% split that goes into validation_phoneme, though that should occur if you have more than 20 segments.
What you can you is cut and paste 5 or so lines from train_phoneme into validation_phoneme and that would allow you to train, but I'm very interested in how much data you are using to train based off of.

The wav file I'm using for training is only a minute long and I get this error. What is the recommended length in minutes for the wav file that I'm using for training? What's the best range to get good results after training?

I'd recommend at least 10 minutes. Optimal would be around 20-30

@Ionaut
Copy link
Author

Ionaut commented Aug 21, 2024

Using approx. 25 min of audio now and it did fix this problem. Ran into another error at the end of the first epoch: RuntimeError: [enforce fail at inline_container.cc:595] . unexpected pos 1457862912 vs 1457862800
Dedicated GPU maxed out at 10.7 GB/12GB

@sleekping
Copy link

I am really having issues with setting up styleTTS. When I open gradio web UI it doesn't give me an option to train voice. I need help. When I click run training it's blank

@JarodMica
Copy link
Owner

I am really having issues with setting up styleTTS. When I open gradio web UI it doesn't give me an option to train voice. I need help. When I click run training it's blank

Hey there, I think I saw you posted in the YouTube area, so I'll just be responding here as it's easier to debug.

Are you getting zero division error? If so, please check the above answers to see what may be the issue.

If not, can you open up a new issue and post what errors you are getting, including screenshots and any error messages in the terminal window?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants