regrtest: "Cannot read process stdout: 'utf-8' codec can't decode byte 0xdd" error

[vstinner]

On a buildbot, a job failed because a test worker stdout cannot be decoded from UTF-8:

Cannot read process stdout: 'utf-8' codec can't decode byte 0xdd in position 2360: invalid continuation byte

In February, @gpshead reported a similar issue: issue #101634. I fixed it by adding a try/except: commit 2ac3eec.

The try/except stops regrtest in a more clean fashion, but it has a big issue: the worker stdout is lost and cannot be displayed.

IMO regrtest should use any mean to display the stdout, especially if it's corrupted. It's too important, since some bugs only occur on some platforms. For example, Windows can log assertion errors in UTF-16 whereas regrtest uses a 8-bit encoding (issue #108989 closed as "WONTFIX").

(...)
0:02:59 load avg: 9.85 [200/463] test_tempfile passed -- running (1): test_pickle (42.6 sec)
0:03:02 load avg: 10.18 [201/463] test_codecs passed -- running (2): test_pickle (45.5 sec), test.test_multiprocessing_spawn.test_processes (32.2 sec)
0:03:02 load avg: 10.18 [202/463] test_linecache passed -- running (2): test_pickle (46.1 sec), test.test_multiprocessing_spawn.test_processes (32.9 sec)
0:03:03 load avg: 10.18 [203/463] test_set passed -- running (2): test_pickle (47.3 sec), test.test_multiprocessing_spawn.test_processes (34.1 sec)
0:03:04 load avg: 10.18 [204/463] test_shlex passed -- running (2): test_pickle (47.8 sec), test.test_multiprocessing_spawn.test_processes (34.6 sec)

0:03:05 load avg: 10.18 [205/463/1] test_interpreters process crashed (Cannot read process stdout: 'utf-8' codec can't decode byte 0xdd in position 2360: invalid continuation byte) -- running (2): test_pickle (48.7 sec), test.test_multiprocessing_spawn.test_processes (35.5 sec)

Kill <WorkerThread #1 running test=test_sys pid=1206013 time=2.6 sec> process group
Kill <WorkerThread #2 running test=test.test_asyncio.test_events pid=1205711 time=9.2 sec> process group
Kill <WorkerThread #3 running test=test_tools pid=1205910 time=6.2 sec> process group
Kill <WorkerThread #4 running test=test_float pid=1206093 time=885 ms> process group
Kill <WorkerThread #5 running test=test.test_multiprocessing_spawn.test_threads pid=1205413 time=16.3 sec> process group
(...)

build: https://buildbot.python.org/all/#/builders/435/builds/3612

Linked PRs

• gh-109425: regrtest decodes worker stdout with backslashreplace #109428