Long transcription not maxing out GPU or CPU

[astrowonk]

After testing on my M1 Mac Mini, I moved over to my gaming PC where I believe pytorch can see CUDA and it should use the GPU to transcode faster (torch.cuda.is_available() returns true) A short 30 second clip I recorded myself converted in a few seconds, quicker than the M1 Mac.

But for a much longer test of a ~2 hour file, I am not seeing either the GPU or CPU max out. GPU is in the single digit % usage, CPU is maybe 10%.

I understand it's working through the file in sliding 30 second windows, but I'd expect to see something light up 100% on my PC when doing all this. Neither the GPU nor CPU seems very taxed at the moment during .transcribe. What is the bottleneck on performance if not the CPU/GPU?

I'm invoking in python like the example like so:

model.transcribe('file.mp3',language='en')

[drdaxxy]

GPU is in the single digit % usage

I think "GPU utilization" on Windows only measures graphics work. Looking at its power consumption etc. in a hardware monitor is much more helpful to see if it's "doing something".

I understand it's working through the file in sliding 30 second windows, but I'd expect to see something light up 100% on my PC when doing all this

Each window's output is used as context for the next, and within the windows, after the encoder "listens" to the whole segment, the decoder predicts character probabilities one at a time, based on the characters selected before.

So a lot of work can't be parallelized - without transcribing multiple recordings in parallel. model.decode() runs the model on a stack of (independent) 30s windows, but model.transcribe() which applies the fancy sliding-window processing etc. apparently only works on single recordings, so someone would have to implement that.

One thing that is parallelized within windows is keeping a list of beam_size best output sequences instead of only one path (5 by default in the CLI). You can set that parameter and hope to get better results at the expense of speed.

[astrowonk]

Hmm, well next I tried to just do multiple files at once using ProcessPoolExecutor but that didn't work; it just crashes silently. Running multiple terminal windows using my own command line script seems to do fine though.

[astrowonk]

Well, running in two terminals took twice as long so whatever it's doing / whatever the bottleneck (VRAM?) is for inference, it's maxing out what it can do with CUDA.