-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
java_launcher.cc parsing of JVM flags doesn't use shell tokenization #7072
Comments
Thanks for the bug report! Your culprit analysis is absolutely correct. The bad news is, on Linux/macOS the quoting support is implemented by simply embedding the argument in a shell script (the Java stub script) and letting Bash interpret it. On Windows we have to implement it ourselves. That means implementing Bash de-qouting to get the same string the The good news is, I have a prototype implementation of the de-quoting + escaping code (which needs a lot of testing before I can upstream it) and it seems to work as expected, even with nasty inputs:
produces on both Linux and Windows (with my patch):
|
Bazel on Windows now correctly forwards java_binary.jvm_flags to the binary. Bazel Bash-tokenizes java_binary.jvm_flags. On Linux/macOS, this is done by embedding the flags into the Java stub script, which is a Bash script, so Bash itself does the tokenization. On Windows, java_binary is launched by a native C++ binary. Therefore nothing could Bash-tokenize the flags until now. From now on, Bazel itself Bash-tokenizes the flags before embedding them in the launcher. The launcher then escapes these flags for CreateProcessW. This PR also adds a new, CreateProcessW-aware escaping method called CreateProcessEscapeArg, to the launcher. The pre-existing GetEscapedArgument escaped only with Bash semantics in mind. This method is now renamed to BashEscapeArg, and used only for the Bash launcher. For the Python and Java launcher, the new CreateProcessEscapeArg method is used. Fixes bazelbuild#7072 Change-Id: I48d675fcce39e4f6c95e3bad3e8569f0274f662a
Bazel on Windows now correctly forwards java_binary.jvm_flags to the binary. Bazel Bash-tokenizes java_binary.jvm_flags. On Linux/macOS, this is done by embedding the flags into the Java stub script, which is a Bash script, so Bash itself does the tokenization. On Windows, java_binary is launched by a native C++ binary. Therefore nothing could Bash-tokenize the flags until now. From now on, Bazel itself Bash-tokenizes the flags before embedding them in the launcher. The launcher then escapes these flags for CreateProcessW. This PR also adds a new, CreateProcessW-aware escaping method called CreateProcessEscapeArg, to the launcher. The pre-existing GetEscapedArgument escaped only with Bash semantics in mind. This method is now renamed to BashEscapeArg, and used only for the Bash launcher. For the Python and Java launcher, the new CreateProcessEscapeArg method is used. Fixes bazelbuild#7072
Bazel on Windows now correctly forwards java_binary.jvm_flags to the binary. Bazel Bash-tokenizes java_binary.jvm_flags. On Linux/macOS, this is done by embedding the flags into the Java stub script, which is a Bash script, so Bash itself does the tokenization. On Windows, java_binary is launched by a native C++ binary. Therefore nothing could Bash-tokenize the flags until now. From now on, Bazel itself Bash-tokenizes the flags before embedding them in the launcher. The launcher then escapes these flags for CreateProcessW. This PR also adds a new, CreateProcessW-aware escaping method called CreateProcessEscapeArg, to the launcher. The pre-existing GetEscapedArgument escaped only with Bash semantics in mind. This method is now renamed to BashEscapeArg, and used only for the Bash launcher. For the Python and Java launcher, the new CreateProcessEscapeArg method is used. Fixes bazelbuild#7072
Bazel on Windows now correctly forwards java_binary.jvm_flags to the binary. Bazel Bash-tokenizes java_binary.jvm_flags. On Linux/macOS, this is done by embedding the flags into the Java stub script, which is a Bash script, so Bash itself does the tokenization. On Windows, java_binary is launched by a native C++ binary. Therefore nothing could Bash-tokenize the flags until now. From now on, Bazel itself Bash-tokenizes the flags before embedding them in the launcher. The launcher then escapes these flags for CreateProcessW. This PR also adds a new, CreateProcessW-aware escaping method called CreateProcessEscapeArg, to the launcher. The pre-existing GetEscapedArgument escaped only with Bash semantics in mind. This method is now renamed to BashEscapeArg, and used only for the Bash launcher. For the Python and Java launcher, the new CreateProcessEscapeArg method is used. Fixes bazelbuild#7072
Next step: implement correct escaping semantics for subprocesses created with CreateProcessW. See bazelbuild#7072
Next step: implement correct escaping semantics for subprocesses created with CreateProcessW. See bazelbuild#7072
Add a unit test to assert that: - WindowsEscapeArg escapes flags as expected - When we pass a WindowsEscapeArg-escaped flag to CreateProcessW, the subprocess receives the original flag. This is a follow-up to bazelbuild#7395 Next I'll fix WindowsEscapeArg to correctly escape arguments. See bazelbuild#7072
Introduce BashEscapeArg and WindowsEscapeArg that just wrap GetEscapedArgument for now. The Bash launcher needs to escape the arguments Bash style (using BashEscapeArg) while the Java and Python launchers need to escape them Windows style (using WindowsEscapeArg). (The code is now incorrectly escaping everything with Bash syntax, i.e. GetEscapedArgument.) Next step: implement correct escaping semantics for subprocesses created with CreateProcessW. See #7072 Closes #7395. PiperOrigin-RevId: 233584339
Add a unit test to assert that: - WindowsEscapeArg escapes flags as expected - When we pass a WindowsEscapeArg-escaped flag to CreateProcessW, the subprocess receives the original flag. This is a follow-up to bazelbuild#7395 Next I'll fix WindowsEscapeArg to correctly escape arguments. See bazelbuild#7072
Add a unit test to assert that: - WindowsEscapeArg escapes flags as expected - When we pass a WindowsEscapeArg-escaped flag to CreateProcessW, the subprocess receives the original flag. This is a follow-up to #7395 Next I'll fix WindowsEscapeArg to correctly escape arguments. See #7072 Closes #7402. PiperOrigin-RevId: 233703690
Add a correct implementation of WindowsEscapeArg. This is a follow-up to bazelbuild#7402 Next steps: - update Bazel to separate jvm_flags that are embedded in the native lauancher by some other character than space, so that arguments with spaces can be embedded - replace the current WindowsEscapeArg implementation in the launcher with the new, correct one See bazelbuild#7072
Add a correct implementation of WindowsEscapeArg. This is a follow-up to #7402 Next steps: - update Bazel to separate jvm_flags that are embedded in the native lauancher by some other character than space, so that arguments with spaces can be embedded - replace the current WindowsEscapeArg implementation in the launcher with the new, correct one See #7072 Closes #7411. PiperOrigin-RevId: 233733871
When Bazel embeds the jvm_flags data into the java_binary launcher, it will now join the flags on TAB instead of on space. While TAB is just as much a legal character in flags as space is, it is probably rarer, and as such its a slightly safer choice than joining the flags on space (and thus losing the ability to support flags with spaces in them). This is a follow-up to bazelbuild#7411 Next steps: - Update Bazel to Bash-tokenize the jvm_flags before embedding them in the launcher. This is required because the jvm_flags attribute should support Bash-tokenization (according to the Build Encyclopedia). - Update the launcher to escape the jvm_flags with WindowsEscapeArg2 instead of WindowsEscapeArg. See bazelbuild#7072
When Bazel embeds the jvm_flags data into the java_binary launcher, it will now join the flags on TAB instead of on space. While TAB is just as much a legal character in flags as space is, it is probably rarer, and as such it's a slightly safer choice than joining the flags on space (and thus losing the ability to support flags with spaces in them). This is a follow-up to #7411 Next steps: - Update Bazel to Bash-tokenize the jvm_flags before embedding them in the launcher. This is required because the jvm_flags attribute should support Bash-tokenization (according to the Build Encyclopedia). - Update the launcher to escape the jvm_flags with WindowsEscapeArg2 instead of WindowsEscapeArg. See #7072 Closes #7421. PiperOrigin-RevId: 233900484
Add --incompatible_windows_escape_jvm_flags flag (default: false). This flag has no effect on platforms other than Windows. This flag affects how Bazel builds the launcher of java_binary and java_test targets. (The launcher is the .exe file that sets up the environment for the Java program and launches the JVM.) In particular, the flag controls whether or not Bazel will Bash-tokenize the jvm_flags that were declared in the BUILD file, and whether or not these jvm_flags are escaped by the launcher when it invokes the JVM. When the flag is enabled: - Bazel will Bash-tokenize java_binary.jvm_flags and java_test.jvm_flags (as documented by the Build Encyclopedia) before embedding them into the launcher. - The launcher will properly escape these flags when it runs the JVM as a subprocess (using launcher_util::WindowsEscapeArg2). - The result is that the jvm_flags declared in the BUILD file will arrive to the Java program as intended. When the flag is disabled: - Bazel does not Bash-tokenize the jvm_flags before embedding them in the launcher. - The launcher escapes the flags with a Bash-like escaping logic (launcher_util::WindowsEscapeArg) which cannot properly quote and escape everything. - The result is that the jvm_flags declared in the BUILD file might get messed up as they are passed to the JVM, or the launcher may not even be able to run the JVM. Incompatible flag: bazelbuild#7486 Related: bazelbuild#7072 RELNOTES[NEW]: Added --incompatible_windows_escape_jvm_flags flag: enables correct java_binary.jvm_flags and java_test.jvm_flags tokenization and escaping on Windows. (No-op on other platforms.)
Add --incompatible_windows_escape_jvm_flags flag (default: false). This flag has no effect on platforms other than Windows. This flag affects how Bazel builds the launcher of java_binary and java_test targets. (The launcher is the .exe file that sets up the environment for the Java program and launches the JVM.) In particular, the flag controls whether or not Bazel will Bash-tokenize the jvm_flags that were declared in the BUILD file, and whether or not these jvm_flags are escaped by the launcher when it invokes the JVM. When the flag is enabled: - Bazel will Bash-tokenize java_binary.jvm_flags and java_test.jvm_flags (as documented by the Build Encyclopedia) before embedding them into the launcher. - The launcher will properly escape these flags when it runs the JVM as a subprocess (using launcher_util::WindowsEscapeArg2). - The result is that the jvm_flags declared in the BUILD file will arrive to the Java program as intended. When the flag is disabled: - Bazel does not Bash-tokenize the jvm_flags before embedding them in the launcher. - The launcher escapes the flags with a Bash-like escaping logic (launcher_util::WindowsEscapeArg) which cannot properly quote and escape everything. - The result is that the jvm_flags declared in the BUILD file might get messed up as they are passed to the JVM, or the launcher may not even be able to run the JVM. Incompatible flag: bazelbuild#7486 Related: bazelbuild#7072 RELNOTES[NEW]: Added --incompatible_windows_escape_jvm_flags flag: enables correct java_binary.jvm_flags and java_test.jvm_flags tokenization and escaping on Windows. (No-op on other platforms.)
Add --incompatible_windows_escape_jvm_flags flag (default: false). This flag has no effect on platforms other than Windows. This flag affects how Bazel builds the launcher of java_binary and java_test targets. (The launcher is the .exe file that sets up the environment for the Java program and launches the JVM.) In particular, the flag controls whether or not Bazel will Bash-tokenize the jvm_flags that were declared in the BUILD file, and whether or not these jvm_flags are escaped by the launcher when it invokes the JVM. When the flag is enabled: - Bazel will Bash-tokenize java_binary.jvm_flags and java_test.jvm_flags (as documented by the Build Encyclopedia) before embedding them into the launcher. - The launcher will properly escape these flags when it runs the JVM as a subprocess (using launcher_util::WindowsEscapeArg2). - The result is that the jvm_flags declared in the BUILD file will arrive to the Java program as intended. When the flag is disabled: - Bazel does not Bash-tokenize the jvm_flags before embedding them in the launcher. - The launcher escapes the flags with a Bash-like escaping logic (launcher_util::WindowsEscapeArg) which cannot properly quote and escape everything. - The result is that the jvm_flags declared in the BUILD file might get messed up as they are passed to the JVM, or the launcher may not even be able to run the JVM. Incompatible flag: bazelbuild#7486 Related: bazelbuild#7072 RELNOTES[NEW]: Added --incompatible_windows_escape_jvm_flags flag: enables correct java_binary.jvm_flags and java_test.jvm_flags tokenization and escaping on Windows. (No-op on other platforms.) Change-Id: I531cc63cdfeccbe4b6d48876cb82870c1726a723
Add --incompatible_windows_escape_jvm_flags flag (default: false). This flag has no effect on platforms other than Windows. This flag affects how Bazel builds the launcher of java_binary and java_test targets. (The launcher is the .exe file that sets up the environment for the Java program and launches the JVM.) In particular, the flag controls whether or not Bazel will Bash-tokenize the jvm_flags that were declared in the BUILD file, and whether or not these jvm_flags are escaped by the launcher when it invokes the JVM. When the flag is enabled: - Bazel will Bash-tokenize java_binary.jvm_flags and java_test.jvm_flags (as documented by the Build Encyclopedia) before embedding them into the launcher. The build fails if an entry cannot be Bash-tokenized, e.g. if it has unterminated quotes. - The launcher will properly escape these flags when it runs the JVM as a subprocess (using launcher_util::WindowsEscapeArg2). - The result is that the jvm_flags declared in the BUILD file will arrive to the Java program as intended. However, due to the extra Bash-tokenization step, some ill-formed flags are no longer accepted, e.g. `jvm_flags=["-Dfoo='a"]` now results in a build error. When the flag is disabled: - Bazel does not Bash-tokenize the jvm_flags before embedding them in the launcher. This preserves quoting meant to be stripped away, and it also means Bazel won't check whether the argument is properly quoted. - The launcher escapes the flags with a Bash-like escaping logic (launcher_util::WindowsEscapeArg) which cannot properly quote and escape everything. - The result: the jvm_flags declared in the BUILD file might get messed up as they are passed to the JVM, or the launcher may not even be able to run the JVM. However, due to the lack of Bash-tokenization, Bazel propagates some flags to the Java binary that it would no longer accept if the new `--incompatible_windows_escape_jvm_flags` were enabled, e.g. `jvm_flags=["'a"]` is fine. Incompatible flag: bazelbuild#7486 Related: bazelbuild#7072 RELNOTES[NEW]: Added --incompatible_windows_escape_jvm_flags flag: enables correct java_binary.jvm_flags and java_test.jvm_flags tokenization and escaping on Windows. (No-op on other platforms.) Change-Id: I531cc63cdfeccbe4b6d48876cb82870c1726a723
Add --incompatible_windows_escape_jvm_flags flag (default: false). This flag has no effect on platforms other than Windows. This flag affects how Bazel builds the launcher of java_binary and java_test targets. (The launcher is the .exe file that sets up the environment for the Java program and launches the JVM.) In particular, the flag controls whether or not Bazel will Bash-tokenize the jvm_flags that were declared in the BUILD file, and whether or not these jvm_flags are escaped by the launcher when it invokes the JVM. When the flag is enabled: - Bazel will Bash-tokenize java_binary.jvm_flags and java_test.jvm_flags (as documented by the Build Encyclopedia) before embedding them into the launcher. The build fails if an entry cannot be Bash-tokenized, e.g. if it has unterminated quotes. - The launcher will properly escape these flags when it runs the JVM as a subprocess (using launcher_util::WindowsEscapeArg2). - Result: the jvm_flags declared in the BUILD file will arrive to the Java program as intended. However, due to the extra Bash-tokenization step, some ill-formed flags are no longer accepted, e.g. `jvm_flags=["-Dfoo='a"]` now results in a build error. When the flag is disabled: - Bazel does not Bash-tokenize the jvm_flags before embedding them in the launcher. This preserves quoting meant to be stripped away, and it also means Bazel won't check whether the argument is properly quoted. - The launcher escapes the flags with a Bash-like escaping logic (launcher_util::WindowsEscapeArg) which cannot properly quote and escape everything. - Result: the jvm_flags declared in the BUILD file might get messed up as they are passed to the JVM, or the launcher may not even be able to run the JVM. However, due to the lack of Bash-tokenization, Bazel propagates some flags to the Java binary that it would no longer accept if the new `--incompatible_windows_escape_jvm_flags` were enabled, e.g. `jvm_flags=["'a"]` is fine. Incompatible flag: #7486 Related: #7072 RELNOTES[NEW]: Added --incompatible_windows_escape_jvm_flags flag: enables correct java_binary.jvm_flags and java_test.jvm_flags tokenization and escaping on Windows. (No-op on other platforms.) Change-Id: I531cc63cdfeccbe4b6d48876cb82870c1726a723 Closes #7490. PiperOrigin-RevId: 235700143
Closing this, because it's fixed at HEAD, see #7486. That bug will be closed when the flag's default value is flipped to true at HEAD. |
Thank you for fixing this! |
The java_binary() rule's jvm_flags argument says that it will be subject to "Bourne shell tokenization" but this doesn't happen on Windows, when apparently java_launcher.cc is used as the launcher instead of the normal stub shell script from java_stub_template.txt.
As a consequence, jvm_flags values that use quoting or backslash-escaping to include values with spaces don't work as expected.
I confirmed this on Windows 10 with bazel 0.21.0. Here's a minimal repro case:
File structure:
/WORKSPACE (empty)
/java/repro/BUILD
/java/repro/Repro.java
contents of BUILD:
contents of Repro.java
Command to reproduce:
bazel run //java/repro:Repro
Expected output:
Hello Ada Lovelace
Actual output:
Error: Could not find or load main class Lovelace'
This appears to occur because java_launcher.h splits the jvm_flags argument list up by spaces, without protecting spaces within quotes (and note that it also leaves the quotes behind). It produces a cryptic error because evidently the "extra" jvm_flags argument is just tacked on as an extra "flag" but since it's not formatted like a flag
java
interprets it as the name of the main class.bazel/src/tools/launcher/java_launcher.cc
Lines 84 to 89 in c10b108
The text was updated successfully, but these errors were encountered: