Investigate non-ASCII build file path on POSIX

[TimothyGu]

• Version: master
• Platform: macOS
• Subsystem: build

At Code & Learn Shanghai we encountered some issues with non-ASCII (specifically, Chinese) file paths when building. Don't have the Python stack trace, but could someone look into that?

[gibfahn]

Stack trace:

➜  弄得 git:(master) ❯ make                                                                                                                                                                  ~/wrk/com/弄得
/Library/Developer/CommandLineTools/usr/bin/make -C out BUILDTYPE=Release V=1
  LD_LIBRARY_PATH=/Users/gib/wrk/com/弄得/out/Release/lib.host:/Users/gib/wrk/com/弄得/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../.; mkdir -p /Users/gib/wrk/com/弄得/out/Release/obj/gen; python tools/js2c.py "/Users/gib/wrk/com/弄得/out/Release/obj/gen/node_javascript.cc" lib/internal/bootstrap_node.js lib/async_hooks.js lib/assert.js lib/buffer.js lib/child_process.js lib/console.js lib/constants.js lib/crypto.js lib/cluster.js lib/dgram.js lib/dns.js lib/domain.js lib/events.js lib/fs.js lib/http.js lib/_http_agent.js lib/_http_client.js lib/_http_common.js lib/_http_incoming.js lib/_http_outgoing.js lib/_http_server.js lib/https.js lib/inspector.js lib/module.js lib/net.js lib/os.js lib/path.js lib/process.js lib/punycode.js lib/querystring.js lib/readline.js lib/repl.js lib/stream.js lib/_stream_readable.js lib/_stream_writable.js lib/_stream_duplex.js lib/_stream_transform.js lib/_stream_passthrough.js lib/_stream_wrap.js lib/string_decoder.js lib/sys.js lib/timers.js lib/tls.js lib/_tls_common.js lib/_tls_legacy.js lib/_tls_wrap.js lib/tty.js lib/url.js lib/util.js lib/v8.js lib/vm.js lib/zlib.js lib/internal/buffer.js lib/internal/child_process.js lib/internal/cluster/child.js lib/internal/cluster/master.js lib/internal/cluster/round_robin_handle.js lib/internal/cluster/shared_handle.js lib/internal/cluster/utils.js lib/internal/cluster/worker.js lib/internal/errors.js lib/internal/freelist.js lib/internal/fs.js lib/internal/http.js lib/internal/linkedlist.js lib/internal/net.js lib/internal/module.js lib/internal/process/next_tick.js lib/internal/process/promises.js lib/internal/process/stdio.js lib/internal/process/warning.js lib/internal/process.js lib/internal/querystring.js lib/internal/process/write-coverage.js lib/internal/readline.js lib/internal/repl.js lib/internal/socket_list.js lib/internal/test/unicode.js lib/internal/url.js lib/internal/util.js lib/internal/v8_prof_polyfill.js lib/internal/v8_prof_processor.js lib/internal/streams/lazy_transform.js lib/internal/streams/BufferList.js lib/internal/streams/legacy.js lib/internal/streams/destroy.js deps/v8/tools/splaytree.js deps/v8/tools/codemap.js deps/v8/tools/consarray.js deps/v8/tools/csvparser.js deps/v8/tools/profile.js deps/v8/tools/profile_view.js deps/v8/tools/logreader.js deps/v8/tools/tickprocessor.js deps/v8/tools/SourceMap.js deps/v8/tools/tickprocessor-driver.js deps/node-inspect/lib/_inspect.js deps/node-inspect/lib/internal/inspect_client.js deps/node-inspect/lib/internal/inspect_repl.js ./config.gypi src/nolttng_macros.py src/perfctr_macros.py
  touch /Users/gib/wrk/com/弄得/out/Release/obj.host/node_js2c.stamp
  LD_LIBRARY_PATH=/Users/gib/wrk/com/弄得/out/Release/lib.host:/Users/gib/wrk/com/弄得/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../deps/v8/src/inspector; mkdir -p /Users/gib/wrk/com/弄得/out/Release/obj/gen/src; python ../../third_party/inspector_protocol/CheckProtocolCompatibility.py --stamp "/Users/gib/wrk/com/弄得/out/Release/obj/gen/src/js_protocol.stamp" js_protocol.json
  touch /Users/gib/wrk/com/弄得/out/Release/obj.target/deps/v8/src/inspector/protocol_compatibility.stamp
  touch cc205296965aac8499af8078773aa6a0978c1a60.intermediate
  LD_LIBRARY_PATH=/Users/gib/wrk/com/弄得/out/Release/lib.host:/Users/gib/wrk/com/弄得/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../deps/v8/src/inspector; mkdir -p /Users/gib/wrk/com/弄得/out/Release/obj/gen/src/inspector/protocol /Users/gib/wrk/com/弄得/out/Release/obj/gen/include/inspector; python ../../third_party/inspector_protocol/CodeGenerator.py --jinja_dir ../../third_party --output_base "/Users/gib/wrk/com/弄得/out/Release/obj/gen/src/inspector" --config inspector_protocol_config.json
Failed to parse config file: 'ascii' codec can't decode byte 0xe5 in position 19: ordinal not in range(128)

make[1]: *** [cc205296965aac8499af8078773aa6a0978c1a60.intermediate] Error 1
rm cc205296965aac8499af8078773aa6a0978c1a60.intermediate
make: *** [node] Error 2

[bnoordhuis]

cc @nodejs/v8-inspector

[TimothyGu]

I can reproduce this on Linux as well. This is because Python 2's sys.argv is an array of values of type str, which is basically an array of 8-bit values, instead of unicode. The file system functions see a str value and interpret it as ASCII, which fails when in Unix sys.argv is in fact encoded with UTF-8.

[refack]

nodejs/node-gyp#1203 might solve this
(trying to upstream https://chromium-review.googlesource.com/c/509632/)

[TimothyGu]

@refack It's unrelated.

[refack]

@refack It's unrelated.

Hmmmm it's make not configure... so maybe #12786 (comment)

[Trott]

Editing title to say POSIX since this is on Linux as well as macOS.

[SgtPooki]

I was able to reproduce this and am looking into it.

[lundibundi]

I will close this as it seems to be resolved (this seems to be the relevant comment #16735 (comment)). I've also built it locally with this path node/弄得/node just fine (Arch Linux).

[refack]

@lundibundi were you able to run make test in that path?

[lundibundi]

@refack I've done a full ./configure && make -j4 test command successfully. Though I'm on Arch and it has python3 by default. I'll now try to do that in python2 env as I forgot about that.

[lundibundi]

Upd: I've successfully built it via ./configure && make -j4 test in dev/node/弄得 folder using python2 env.