Spawn Start Method?

[woCmodsiW]

Hello, do you have time to take a look at this one below?

~/xtts$ python -m xtts_api_server -d cuda:1 --streaming-mode-improve
/home/jay/miniconda3/envs/xtts/lib/python3.10/site-packages/pydub/utils.py:170: RuntimeWarning: Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work
warn("Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work", RuntimeWarning)
2023-12-13 16:58:24.724 | WARNING | xtts_api_server.server::53 - 'Streaming Mode' has certain limitations, you can read about them here https://github.com/daswer123/xtts-api-server#about-streaming-mode
2023-12-13 16:58:24.724 | INFO | xtts_api_server.server::56 - You launched an improved version of streaming, this version features an improved tokenizer and more context when processing sentences, which can be good for complex languages like Chinese
2023-12-13 16:58:24.724 | INFO | xtts_api_server.server::58 - Load model for Streaming

Using model: xtts
Process Process-1:
Traceback (most recent call last):
File "/home/jay/miniconda3/envs/xtts/lib/python3.10/site-packages/xtts_api_server/RealtimeTTS/engines/coqui_engine.py", line 296, in _synthesize_worker
tts.to(device)
File "/home/jay/miniconda3/envs/xtts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1160, in to
return self._apply(convert)
File "/home/jay/miniconda3/envs/xtts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 810, in _apply
module._apply(fn)
File "/home/jay/miniconda3/envs/xtts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 810, in _apply
module._apply(fn)
File "/home/jay/miniconda3/envs/xtts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 810, in _apply
module._apply(fn)
File "/home/jay/miniconda3/envs/xtts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 833, in _apply
param_applied = fn(param)
File "/home/jay/miniconda3/envs/xtts/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1158, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
File "/home/jay/miniconda3/envs/xtts/lib/python3.10/site-packages/torch/cuda/init.py", line 284, in _lazy_init
raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/jay/miniconda3/envs/xtts/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/jay/miniconda3/envs/xtts/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/jay/miniconda3/envs/xtts/lib/python3.10/site-packages/xtts_api_server/RealtimeTTS/engines/coqui_engine.py", line 304, in _synthesize_worker
logging.exception(f"Error initializing main coqui engine model: {e}")
File "/home/jay/miniconda3/envs/xtts/lib/python3.10/logging/init.py", line 2113, in exception
error(msg, *args, exc_info=exc_info, **kwargs)
File "/home/jay/miniconda3/envs/xtts/lib/python3.10/logging/init.py", line 2105, in error
root.error(msg, *args, **kwargs)
File "/home/jay/miniconda3/envs/xtts/lib/python3.10/logging/init.py", line 1506, in error
self._log(ERROR, msg, args, **kwargs)
TypeError: Log._log() got an unexpected keyword argument 'exc_info'

[darkacorn]

i got the same issue with streaming

normal operation works just fine

(xtts) ➜ python -m xtts_api_server -hs 0.0.0.0 -v 2.0.3 -d cuda:1

Using model: xtts
2023-12-16 12:39:39.609 | INFO | xtts_api_server.tts_funcs:load_model:81 - Model successfully loaded
INFO: Started server process [42280]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8020 (Press CTRL+C to quit)

(xtts) ➜ python -m xtts_api_server -hs 0.0.0.0 -v 2.0.3 -d cuda:1 --streaming-mode
2023-12-16 12:41:23.183 | WARNING | xtts_api_server.server::53 - 'Streaming Mode' has certain limitations, you can read about them here https://github.com/daswer123/xtts-api-server#about-streaming-mode
2023-12-16 12:41:23.183 | INFO | xtts_api_server.server::58 - Load model for Streaming

Using model: xtts
ERROR:root:Error initializing main coqui engine model: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

the improved steaming-mode yields the same issues

[daswer123]

Guys, I honestly have no idea what the problem could be, I'm using the RealtimeTTS library, I've encountered this myself in colab, but never found a way to fix it

[lendot]

I spent some time looking into this today. In seeking out places where CUDA usage might span multiple processes, I found this in RealtimeTTS/engines/coqui_engine.py:

        self.parent_synthesize_pipe, child_synthesize_pipe = Pipe()

        ...

        self.synthesize_process = Process(target=CoquiEngine._synthesize_worker, args=(child_synthesize_pipe, model_name, cloning_reference_wav, language, self.main_synthesize_ready_event, level, self.speed, thread_count, stream_chunk_size, full_sentences, overlap_wav_len, temperature, length_penalty, repetition_penalty, top_k, top_p, enable_text_splitting, use_mps, self.local_model_path, use_deepspeed, self.voices_path))
        self.synthesize_process.start()

In other words, it's creating a pipe and forking a process to do the synthesis part. I'm guessing that's the part that CUDA's unhappy with. If so then possible solutions would be to confine all CUDA calls to synthesize_process, use the CUDA spawn stuff, or eliminate the subprocess altogether. All would require changes to the RealtimeTTS code.

[lendot]

Oh neat, I just learned that pytorch offers torch.multiprocessing as a drop-in replacement for multiprocessing. I think I have a solution, or at least a possible workaround:

In RealtimeTTS/engines/coqui_engine.py change the first line to this:

from torch.multiprocessing import Process, Pipe, Event, set_start_method

And then inside __init__ set the process start type to spawn by adding the line where shown here:

        # Start the worker process
        set_start_method('spawn')
        self.main_synthesize_ready_event = Event()
        self.parent_synthesize_pipe, child_synthesize_pipe = Pipe()
        self.voices_path = voices_path

Those two changes got realtime working for me, although I haven't tested it extensively.

[lendot]

I submitted a RealtimeTTS PR with essentially the same fix mentioned above.

[daswer123]

Great, thanks for the hint and for the PR in RealtimeTTS, since I'm using a copy of this project I've already fixed my own and released the update

[woCmodsiW]

I'm going to close this one now.

Your continued efforts and maintenance are truly appreciated.