-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Regressions with non-ASCII characters in path names #17817
Comments
The most recent regression trace we have received reads as follows.
The root issue in this callstack may well be of my own making, since the function Though when attempting to get to repro by cloning emsdk under a path |
Good find, yeah, looks like we've just not been testing this. Once this is fixed, sgtm to add testing for this, perhaps by changing the CircleCI script for windows to create and use such a path. |
I've got the same error with Russian major letter И in the COMMENT SECTION. After "//" |
@AndrewAlexArt can you share more about the error, like maybe the backtrace? I guess this is a JS or C/C++ source file? Do you know what encoding the file is in? |
Sure. I used .jslib in unity with the comment inside started with major "И"
End the error I've got:
Building Library\Bee\artifacts\WebGL\build\debug_WebGL_wasm\build.js failed
with output:
Traceback (most recent call last):
File "C:\Program
Files\Unity\Hub\Editor\2022.2.12f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten\\emcc.py",
line 3982, in <module>
sys.exit(main(sys.argv))
File
"D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\contextlib.py",
line 74, in inner
File "C:\Program
Files\Unity\Hub\Editor\2022.2.12f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten\\emcc.py",
line 3975, in main
ret = run(args)
File "C:\Program
Files\Unity\Hub\Editor\2022.2.12f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten\\emcc.py",
line 1184, in run
phase_post_link(options, state, wasm_target, wasm_target, target)
File
"D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\contextlib.py",
line 74, in inner
File "C:\Program
Files\Unity\Hub\Editor\2022.2.12f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten\\emcc.py",
line 2745, in phase_post_link
phase_emscript(options, in_wasm, wasm_target, memfile)
File
"D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\contextlib.py",
line 74, in inner
File "C:\Program
Files\Unity\Hub\Editor\2022.2.12f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten\\emcc.py",
line 2773, in phase_emscript
emscripten.run(in_wasm, wasm_target, final_js, memfile)
File "C:\Program
Files\Unity\Hub\Editor\2022.2.12f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten\emscripten.py",
line 937, in run
emscript(in_wasm, out_wasm, outfile_js, memfile)
File "C:\Program
Files\Unity\Hub\Editor\2022.2.12f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten\emscripten.py",
line 327, in emscript
glue, forwarded_data = compile_settings()
File "C:\Program
Files\Unity\Hub\Editor\2022.2.12f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten\emscripten.py",
line 183, in compile_settings
cwd=path_from_root('src'), env=env)
File "C:\Program
Files\Unity\Hub\Editor\2022.2.12f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten\tools\shared.py",
line 236, in run_js_tool
return check_call(command, **kw).stdout
File "C:\Program
Files\Unity\Hub\Editor\2022.2.12f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten\tools\shared.py",
line 222, in check_call
return run_process(cmd, *args, **kw)
File "C:\Program
Files\Unity\Hub\Editor\2022.2.12f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten\tools\shared.py",
line 106, in run_process
ret = subprocess.run(cmd, check=check, input=input, *args, **kw)
File
"D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\subprocess.py",
line 474, in run
File
"D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\subprocess.py",
line 926, in communicate
File
"D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\cp1251.py",
line 23, in decode
UnicodeDecodeError: 'charmap' codec can't decode byte 0x98 in position
149599: character maps to <undefined>
UnityEngine.GUIUtility:ProcessEvent (int,intptr,bool&)
сб, 1 апр. 2023 г. в 01:08, Sam Clegg ***@***.***>:
… @AndrewAlexArt <https://github.com/AndrewAlexArt> can you share more
about the error, like maybe the backtrace? I guess this is a JS or C/C++
source file? Do you know what encoding the file is in?
—
Reply to this email directly, view it on GitHub
<#17817 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AU44STU7QGKEWIJUXSNSEK3W65IVPANCNFSM6AAAAAAQHTKTNY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Do you know what version of emscripten you are using? The line The fact that the error is coming from cp1251.py makes me think that |
I'm not really sure what exactly version of Emscripten Unity using. On
their website it calls Emscripten 3.1.8-unity
I don't know how to check run_process function because have no access to
Emscripten source code in Unity
вс, 2 апр. 2023 г. в 04:40, Sam Clegg ***@***.***>:
… Do you know what version of emscripten you are using? The line kw.setdefault('encoding',
'utf-8') was added to run_process in tools/shared.py back in #16736
<#16736>. That change
was released in 3.1.14. Can you check if you have that line in your
run_process function, or if you are using 3.1.14 or above?
The fact that the error is coming from cp1251.py makes me think that
encoding=utf-8 is not being passed to subprocess.run.
—
Reply to this email directly, view it on GitHub
<#17817 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AU44STWRX3GFGMNVGL35ZADW7DKKJANCNFSM6AAAAAAQHTKTNY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Oh wait. I found. There are different arguments in the function:
[image: image.png]
вс, 2 апр. 2023 г. в 16:51, Andrew Alexandrov ***@***.***>:
… I'm not really sure what exactly version of Emscripten Unity using. On
their website it calls Emscripten 3.1.8-unity
I don't know how to check run_process function because have no access to
Emscripten source code in Unity
вс, 2 апр. 2023 г. в 04:40, Sam Clegg ***@***.***>:
> Do you know what version of emscripten you are using? The line kw.setdefault('encoding',
> 'utf-8') was added to run_process in tools/shared.py back in #16736
> <#16736>. That change
> was released in 3.1.14. Can you check if you have that line in your
> run_process function, or if you are using 3.1.14 or above?
>
> The fact that the error is coming from cp1251.py makes me think that
> encoding=utf-8 is not being passed to subprocess.run.
>
> —
> Reply to this email directly, view it on GitHub
> <#17817 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AU44STWRX3GFGMNVGL35ZADW7DKKJANCNFSM6AAAAAAQHTKTNY>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
If you are using |
yea, when Unity updates Emscripten in its engine, I will check this point
again
вс, 2 апр. 2023 г. в 19:11, Sam Clegg ***@***.***>:
… If you are using 3.1.8 then I think this issue may already have been
fixed in more recent versions of emscripten (3.1.14 and above).
—
Reply to this email directly, view it on GitHub
<#17817 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AU44STSJEPMXEJFP762REWLW7GQLFANCNFSM6AAAAAAQHTKTNY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
It looks like Emscripten has regressed to no longer work if there are difficult characters in the path names - now most recently we have gotten reports over the character ç on Windows, (Unicode U+00E7, UTF-8 encoding C3 A7).
This character is present as 0xE7 encoding in ISO-8859-1 (Latin 1), ISO-8859-2, *-3, *-14, *-15 and *-16 charsets.
That made me wonder what the current status of testing this kind of behavior is? Back long time ago when Mozilla was maintaining the test suites, the test CIs ran inside a path that contained all kinds of difficult characters (spaces, single quotes, dollar, hash signs, upper 8-bit Latin-1 chars, BMP Unicode chars and non-BMP Unicode chars).
Reading the circleci script, that no longer seems to be the case?
I wonder if this kind of hardening should be redeveloped to the circleci scripts? It would be good to have Emsdk, Emscripten compiler, the .emscripten config and build temp directories each reside inside a difficult path name, so they exercise all the different sources of path name difficulties that may arise.
The text was updated successfully, but these errors were encountered: