doc: force UTF-8 on windows #35234
doc: force UTF-8 on windows #35234
Conversation
Use '-X utf8' to force open() in text mode to use that encoding. Unfortunately this fix is not available on Python 3.6, but this should work on 3.7 or later. Fixes: zephyrproject-rtos#34076 Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
mbolivar-nordic
added
area: Documentation
bug
| @@ -264,7 +280,7 @@ add_custom_target( | |||
| PYTHONPATH=${ZEPHYR_BASE}/scripts/dts/python-devicetree/src${SEP}$ENV{PYTHONPATH} | |||
| ZEPHYR_BASE=${ZEPHYR_BASE} | |||
| GEN_DEVICETREE_REST_ZEPHYR_DOCSET=${GEN_DEVICETREE_REST_ZEPHYR_DOCSET} | |||
| ${PYTHON_EXECUTABLE} ${GEN_DEVICETREE_REST_SCRIPT} | |||
| ${PYTHON_EXECUTABLE} ${PYTHONOPTS} ${GEN_DEVICETREE_REST_SCRIPT} | |||
There was a problem hiding this comment.
#34076 may be fixed by just adjusting the encoding used by the DTS generator script.
There was a problem hiding this comment.
#34076 may be fixed by just adjusting the encoding used by the DTS generator script.
If we start that game of whackamole then we'll never stop; I'm trying to find a solution that doesn't require us to take special care everywhere.
There was a problem hiding this comment.
I think it's the script the responsible to handle the encoding and not CMake via Python option. As mentioned, Sphinx takes care of doing that when dealing with project sources, we should do the same. I think current CMake is convoluted enough to not add more flags that, by the way, are not available on all Python versions we support.
There was a problem hiding this comment.
Another point: if any of the current scripts ends up being proper Sphinx extensions, we can't rely on -X utf8.
There was a problem hiding this comment.
I think it's fair for Zephyr scripts to expect to be run in a UTF-8 compatible way. A future question about converting to sphinx extensions seems like a hypothetical.
There was a problem hiding this comment.
Since the script knows that the files going to be handled are UTF-8 here, it's the responsible to read/write with the appropriate encoding. It's annoying that Windows doesn't default to utf-8...
There's another solution (not tested) that may not depend on having Python >= 3.7: set PYTHONUTF8=1 on CMake
edit: PYTHONUTF8 is also >= 3.7, https://vstinner.github.io/python37-new-utf8-mode.html
There was a problem hiding this comment.
To me this boils down to 2 things:
- the reporter confirms this is a fix
- we already have a precedent set in the Zephyr build system's python.cmake file that we are willing to add in Windows-specific workarounds
So this is the fix I would suggest. Feel free to file your own patch of course!
What do you mean by "touch PYTHONOPTS"? That's just a random variable name; it's not special to FindPython3 or anything. |
|
@gmarull ping |
|
obsoleted by #35470 |
Use '-X utf8' to force open() in text mode to use that encoding.
Unfortunately this fix is not available on Python 3.6, but this should
work on 3.7 or later.
Fixes: #34076
Signed-off-by: Martí Bolívar marti.bolivar@nordicsemi.no