codec can't encode character '\u266a'

[turnkit]

When I run Whisper in a batch model on the large model it produces a musical symbol and fails.

Seems to work fine though in a normal command line just not in a shell as below. Odd?

Any suggestions?

UnicodeEncodeError: 'charmap' codec can't encode character '\u266a' in position 27: character maps to

here's the script:
for f in *.mp4
do echo 'Start processing: ' "${f}" $(date)

whisper --language en --model large --device cuda --verbose True --condition_on_previous_text False --temperature_increment_on_fallback None --output_dir whisp_out_large "${f}"

whisper --model large --language en --device cuda --verbose True --output_dir whisp_out_large "${f}"
mv "${f}" "./_transcription_done_large/${f}"
echo 'Stop processing: ' "${f}" $(date)
done

Here's the full output:

processing: 2021-1212-CCF-LiveStream-WorshipService.mp4 Mon Dec 19 17:20:52 CST 2022
Traceback (most recent call last):
File "C:\Users\turnk\anaconda3\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\turnk\anaconda3\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\turnk\anaconda3\Scripts\whisper.exe_main.py", line 7, in
File "C:\Users\turnk\anaconda3\lib\site-packages\whisper\transcribe.py", line 307, in cli
result = transcribe(model, audio_path, temperature=temperature, **args)
File "C:\Users\turnk\anaconda3\lib\site-packages\whisper\transcribe.py", line 207, in transcribe
add_segment(
File "C:\Users\turnk\anaconda3\lib\site-packages\whisper\transcribe.py", line 168, in add_segment
print(f"[{format_timestamp(start)} --> {format_timestamp(end)}] {text}")
File "C:\Users\turnk\anaconda3\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u266a' in position 27: character maps to
<Managed Device 0>
Traceback (most recent call last):

[turnkit]

Ah - weird. Fixed the issue.

I was running on Windows in Anaconda shell. Somehow when it launches a new window for a shell it doesn't handle the extended uni characters well. I got around the problem by just typing 'bash' in the Anaconda shell and then launching the little script with bash so that all the whisper run stayed in that same environment. This is the "fun" of running Whisper under WSL2 in Windows 11 I guess.

[mihadoo]

The problem is the following line (under Windows 11):

c:\Python311\Lib\site-packages\whisper\transcribe.py line 339:

                print(make_safe(line))

If I replace it to the following then it does not crash anymore but it also does not produce the correct output:

                print(make_safe(line).encode('utf8'))

Maybe someone with enough python knowledge can correct this. A possible solution is described here:

https://stackoverflow.com/questions/10569438/how-to-print-unicode-character-in-python

In any case the output must be UTF-8 for TXT and JSON VTT and other files so that all languages will work correctly.

[magnumquest]

This solves the issue:
Edit the file /modules/utils.py

And change the function add_to_textfile to:

def add_to_textfile(text_file, text):
  text = text.encode('utf-8')
  with open(text_file, 'a', encoding='utf-8') as f:
    f.write(text.decode('utf-8'))

[Sparviero-Sughero]

add_to_textfile() function no longer exists in utils.py