Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

java_launcher.cc parsing of JVM flags doesn't use shell tokenization #7072

Closed
nfelt opened this issue Jan 10, 2019 · 3 comments
Closed

java_launcher.cc parsing of JVM flags doesn't use shell tokenization #7072

nfelt opened this issue Jan 10, 2019 · 3 comments
Assignees
Labels
area-Windows Windows-specific issues and feature requests P2 We'll consider working on this in future. (Assignee optional) team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website type: bug

Comments

@nfelt
Copy link

nfelt commented Jan 10, 2019

The java_binary() rule's jvm_flags argument says that it will be subject to "Bourne shell tokenization" but this doesn't happen on Windows, when apparently java_launcher.cc is used as the launcher instead of the normal stub shell script from java_stub_template.txt.

As a consequence, jvm_flags values that use quoting or backslash-escaping to include values with spaces don't work as expected.

I confirmed this on Windows 10 with bazel 0.21.0. Here's a minimal repro case:

File structure:
/WORKSPACE (empty)
/java/repro/BUILD
/java/repro/Repro.java

contents of BUILD:

java_binary(
  name = "Repro",
  srcs = ["Repro.java",
  jvm_flags = ["-Duser.name='Ada Lovelace'"],
)

contents of Repro.java

package repro;

public class Repro {
  public static void main(String[] args) {
    System.out.printf("Hello %s\n", System.getProperty("user.name"));
  }
}

Command to reproduce: bazel run //java/repro:Repro
Expected output: Hello Ada Lovelace
Actual output: Error: Could not find or load main class Lovelace'

This appears to occur because java_launcher.h splits the jvm_flags argument list up by spaces, without protecting spaces within quotes (and note that it also leaves the quotes behind). It produces a cryptic error because evidently the "extra" jvm_flags argument is just tacked on as an extra "flag" but since it's not formatted like a flag java interprets it as the name of the main class.

} else if (GetFlagValue(argument, L"--jvm_flags=", &flag_value)) {
wstringstream flag_value_ss(flag_value);
wstring item;
while (getline(flag_value_ss, item, L' ')) {
this->jvm_flags_cmdline.push_back(item);
}

@iirina iirina added platform: windows untriaged area-Windows Windows-specific issues and feature requests and removed platform: windows labels Jan 10, 2019
@laszlocsomor laszlocsomor self-assigned this Jan 11, 2019
@laszlocsomor laszlocsomor added type: bug P2 We'll consider working on this in future. (Assignee optional) and removed untriaged labels Jan 11, 2019
@laszlocsomor
Copy link
Contributor

laszlocsomor commented Jan 11, 2019

Thanks for the bug report! Your culprit analysis is absolutely correct.

The bad news is, on Linux/macOS the quoting support is implemented by simply embedding the argument in a shell script (the Java stub script) and letting Bash interpret it. On Windows we have to implement it ourselves. That means implementing Bash de-qouting to get the same string the java process would (on Linux), and proper escaping for CreateProcessW that runs java.exe.

The good news is, I have a prototype implementation of the de-quoting + escaping code (which needs a lot of testing before I can upstream it) and it seems to work as expected, even with nasty inputs:

    jvm_flags = ["-Duser.name=\"$(COMPILATION_MODE)\\\\\"\ '\\\\\"Ada Lovelace'"],

produces on both Linux and Windows (with my patch):

Hello fastbuild\ \\"Ada Lovelace

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Jan 31, 2019
Bazel on Windows now correctly forwards
java_binary.jvm_flags to the binary.

Bazel Bash-tokenizes java_binary.jvm_flags.
On Linux/macOS, this is done by embedding the
flags into the Java stub script, which is a Bash
script, so Bash itself does the tokenization.

On Windows, java_binary is launched by a native
C++ binary. Therefore nothing could Bash-tokenize
the flags until now. From now on, Bazel itself
Bash-tokenizes the flags before embedding them in
the launcher. The launcher then escapes these
flags for CreateProcessW.

This PR also adds a new, CreateProcessW-aware
escaping method called CreateProcessEscapeArg, to
the launcher.

The pre-existing GetEscapedArgument escaped only
with Bash semantics in mind. This method is now
renamed to BashEscapeArg, and used only for the
Bash launcher. For the Python and Java launcher,
the new CreateProcessEscapeArg method is used.

Fixes bazelbuild#7072

Change-Id: I48d675fcce39e4f6c95e3bad3e8569f0274f662a
laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Feb 5, 2019
Bazel on Windows now correctly forwards
java_binary.jvm_flags to the binary.

Bazel Bash-tokenizes java_binary.jvm_flags.
On Linux/macOS, this is done by embedding the
flags into the Java stub script, which is a Bash
script, so Bash itself does the tokenization.

On Windows, java_binary is launched by a native
C++ binary. Therefore nothing could Bash-tokenize
the flags until now. From now on, Bazel itself
Bash-tokenizes the flags before embedding them in
the launcher. The launcher then escapes these
flags for CreateProcessW.

This PR also adds a new, CreateProcessW-aware
escaping method called CreateProcessEscapeArg, to
the launcher.

The pre-existing GetEscapedArgument escaped only
with Bash semantics in mind. This method is now
renamed to BashEscapeArg, and used only for the
Bash launcher. For the Python and Java launcher,
the new CreateProcessEscapeArg method is used.

Fixes bazelbuild#7072
laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Feb 5, 2019
Bazel on Windows now correctly forwards
java_binary.jvm_flags to the binary.

Bazel Bash-tokenizes java_binary.jvm_flags.
On Linux/macOS, this is done by embedding the
flags into the Java stub script, which is a Bash
script, so Bash itself does the tokenization.

On Windows, java_binary is launched by a native
C++ binary. Therefore nothing could Bash-tokenize
the flags until now. From now on, Bazel itself
Bash-tokenizes the flags before embedding them in
the launcher. The launcher then escapes these
flags for CreateProcessW.

This PR also adds a new, CreateProcessW-aware
escaping method called CreateProcessEscapeArg, to
the launcher.

The pre-existing GetEscapedArgument escaped only
with Bash semantics in mind. This method is now
renamed to BashEscapeArg, and used only for the
Bash launcher. For the Python and Java launcher,
the new CreateProcessEscapeArg method is used.

Fixes bazelbuild#7072
laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Feb 5, 2019
Bazel on Windows now correctly forwards
java_binary.jvm_flags to the binary.

Bazel Bash-tokenizes java_binary.jvm_flags.
On Linux/macOS, this is done by embedding the
flags into the Java stub script, which is a Bash
script, so Bash itself does the tokenization.

On Windows, java_binary is launched by a native
C++ binary. Therefore nothing could Bash-tokenize
the flags until now. From now on, Bazel itself
Bash-tokenizes the flags before embedding them in
the launcher. The launcher then escapes these
flags for CreateProcessW.

This PR also adds a new, CreateProcessW-aware
escaping method called CreateProcessEscapeArg, to
the launcher.

The pre-existing GetEscapedArgument escaped only
with Bash semantics in mind. This method is now
renamed to BashEscapeArg, and used only for the
Bash launcher. For the Python and Java launcher,
the new CreateProcessEscapeArg method is used.

Fixes bazelbuild#7072
laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Feb 11, 2019
Next step: implement correct escaping semantics
for subprocesses created with CreateProcessW.

See bazelbuild#7072
laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Feb 12, 2019
Next step: implement correct escaping semantics
for subprocesses created with CreateProcessW.

See bazelbuild#7072
laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Feb 12, 2019
Add a unit test to assert that:
- WindowsEscapeArg escapes flags as expected
- When we pass a WindowsEscapeArg-escaped flag to
  CreateProcessW, the subprocess receives the
  original flag.

This is a follow-up to bazelbuild#7395

Next I'll fix WindowsEscapeArg to correctly escape
arguments.

See bazelbuild#7072
bazel-io pushed a commit that referenced this issue Feb 12, 2019
Introduce BashEscapeArg and WindowsEscapeArg that
just wrap GetEscapedArgument for now.

The Bash launcher needs to escape the arguments
Bash style (using BashEscapeArg) while the Java
and Python launchers need to escape them Windows
style (using WindowsEscapeArg).

(The code is now incorrectly escaping everything
with Bash syntax, i.e. GetEscapedArgument.)

Next step: implement correct escaping semantics
for subprocesses created with CreateProcessW.

See #7072

Closes #7395.

PiperOrigin-RevId: 233584339
laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Feb 13, 2019
Add a unit test to assert that:
- WindowsEscapeArg escapes flags as expected
- When we pass a WindowsEscapeArg-escaped flag to
  CreateProcessW, the subprocess receives the
  original flag.

This is a follow-up to bazelbuild#7395

Next I'll fix WindowsEscapeArg to correctly escape
arguments.

See bazelbuild#7072
bazel-io pushed a commit that referenced this issue Feb 13, 2019
Add a unit test to assert that:
- WindowsEscapeArg escapes flags as expected
- When we pass a WindowsEscapeArg-escaped flag to
CreateProcessW, the subprocess receives the
original flag.

This is a follow-up to #7395

Next I'll fix WindowsEscapeArg to correctly escape
arguments.

See #7072

Closes #7402.

PiperOrigin-RevId: 233703690
laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Feb 13, 2019
Add a correct implementation of WindowsEscapeArg.

This is a follow-up to bazelbuild#7402

Next steps:
- update Bazel to separate jvm_flags that are
  embedded in the native lauancher by some other
  character than space, so that arguments with
  spaces can be embedded

- replace the current WindowsEscapeArg
  implementation in the launcher with the new,
  correct one

See bazelbuild#7072
bazel-io pushed a commit that referenced this issue Feb 13, 2019
Add a correct implementation of WindowsEscapeArg.

This is a follow-up to #7402

Next steps:
- update Bazel to separate jvm_flags that are
  embedded in the native lauancher by some other
  character than space, so that arguments with
  spaces can be embedded

- replace the current WindowsEscapeArg
  implementation in the launcher with the new,
  correct one

See #7072

Closes #7411.

PiperOrigin-RevId: 233733871
laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Feb 13, 2019
When Bazel embeds the jvm_flags data into the
java_binary launcher, it will now join the flags
on TAB instead of on space.

While TAB is just as much a legal character in
flags as space is, it is probably rarer, and as
such its a slightly safer choice than joining the
flags on space (and thus losing the ability to
support flags with spaces in them).

This is a follow-up to bazelbuild#7411

Next steps:

- Update Bazel to Bash-tokenize the jvm_flags
  before embedding them in the launcher. This is
  required because the jvm_flags attribute should
  support Bash-tokenization (according to the
  Build Encyclopedia).

- Update the launcher to escape the jvm_flags with
  WindowsEscapeArg2 instead of WindowsEscapeArg.

See bazelbuild#7072
bazel-io pushed a commit that referenced this issue Feb 14, 2019
When Bazel embeds the jvm_flags data into the
java_binary launcher, it will now join the flags
on TAB instead of on space.

While TAB is just as much a legal character in
flags as space is, it is probably rarer, and as
such it's a slightly safer choice than joining the
flags on space (and thus losing the ability to
support flags with spaces in them).

This is a follow-up to #7411

Next steps:

- Update Bazel to Bash-tokenize the jvm_flags
  before embedding them in the launcher. This is
  required because the jvm_flags attribute should
  support Bash-tokenization (according to the
  Build Encyclopedia).

- Update the launcher to escape the jvm_flags with
  WindowsEscapeArg2 instead of WindowsEscapeArg.

See #7072

Closes #7421.

PiperOrigin-RevId: 233900484
laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Feb 21, 2019
Add --incompatible_windows_escape_jvm_flags flag
(default: false). This flag has no effect on
platforms other than Windows.

This flag affects how Bazel builds the launcher of
java_binary and java_test targets. (The launcher
is the .exe file that sets up the environment for
the Java program and launches the JVM.)

In particular, the flag controls whether or not
Bazel will Bash-tokenize the jvm_flags that were
declared in the BUILD file, and whether or not
these jvm_flags are escaped by the launcher when
it invokes the JVM.

When the flag is enabled:

- Bazel will Bash-tokenize java_binary.jvm_flags
  and java_test.jvm_flags (as documented by the
  Build Encyclopedia) before embedding them into
  the launcher.

- The launcher will properly escape these flags
  when it runs the JVM as a subprocess (using
  launcher_util::WindowsEscapeArg2).

- The result is that the jvm_flags declared in the
  BUILD file will arrive to the Java program as
  intended.

When the flag is disabled:

- Bazel does not Bash-tokenize the jvm_flags
  before embedding them in the launcher.

- The launcher escapes the flags with a Bash-like
  escaping logic (launcher_util::WindowsEscapeArg)
  which cannot properly quote and escape
  everything.

- The result is that the jvm_flags declared in the
  BUILD file might get messed up as they are
  passed to the JVM, or the launcher may not even
  be able to run the JVM.

Incompatible flag: bazelbuild#7486

Related: bazelbuild#7072

RELNOTES[NEW]: Added --incompatible_windows_escape_jvm_flags flag: enables correct java_binary.jvm_flags and java_test.jvm_flags tokenization and escaping on Windows. (No-op on other platforms.)
laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Feb 21, 2019
Add --incompatible_windows_escape_jvm_flags flag
(default: false). This flag has no effect on
platforms other than Windows.

This flag affects how Bazel builds the launcher of
java_binary and java_test targets. (The launcher
is the .exe file that sets up the environment for
the Java program and launches the JVM.)

In particular, the flag controls whether or not
Bazel will Bash-tokenize the jvm_flags that were
declared in the BUILD file, and whether or not
these jvm_flags are escaped by the launcher when
it invokes the JVM.

When the flag is enabled:

- Bazel will Bash-tokenize java_binary.jvm_flags
  and java_test.jvm_flags (as documented by the
  Build Encyclopedia) before embedding them into
  the launcher.

- The launcher will properly escape these flags
  when it runs the JVM as a subprocess (using
  launcher_util::WindowsEscapeArg2).

- The result is that the jvm_flags declared in the
  BUILD file will arrive to the Java program as
  intended.

When the flag is disabled:

- Bazel does not Bash-tokenize the jvm_flags
  before embedding them in the launcher.

- The launcher escapes the flags with a Bash-like
  escaping logic (launcher_util::WindowsEscapeArg)
  which cannot properly quote and escape
  everything.

- The result is that the jvm_flags declared in the
  BUILD file might get messed up as they are
  passed to the JVM, or the launcher may not even
  be able to run the JVM.

Incompatible flag: bazelbuild#7486

Related: bazelbuild#7072

RELNOTES[NEW]: Added --incompatible_windows_escape_jvm_flags flag: enables correct java_binary.jvm_flags and java_test.jvm_flags tokenization and escaping on Windows. (No-op on other platforms.)
laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Feb 21, 2019
Add --incompatible_windows_escape_jvm_flags flag
(default: false). This flag has no effect on
platforms other than Windows.

This flag affects how Bazel builds the launcher of
java_binary and java_test targets. (The launcher
is the .exe file that sets up the environment for
the Java program and launches the JVM.)

In particular, the flag controls whether or not
Bazel will Bash-tokenize the jvm_flags that were
declared in the BUILD file, and whether or not
these jvm_flags are escaped by the launcher when
it invokes the JVM.

When the flag is enabled:

- Bazel will Bash-tokenize java_binary.jvm_flags
  and java_test.jvm_flags (as documented by the
  Build Encyclopedia) before embedding them into
  the launcher.

- The launcher will properly escape these flags
  when it runs the JVM as a subprocess (using
  launcher_util::WindowsEscapeArg2).

- The result is that the jvm_flags declared in the
  BUILD file will arrive to the Java program as
  intended.

When the flag is disabled:

- Bazel does not Bash-tokenize the jvm_flags
  before embedding them in the launcher.

- The launcher escapes the flags with a Bash-like
  escaping logic (launcher_util::WindowsEscapeArg)
  which cannot properly quote and escape
  everything.

- The result is that the jvm_flags declared in the
  BUILD file might get messed up as they are
  passed to the JVM, or the launcher may not even
  be able to run the JVM.

Incompatible flag: bazelbuild#7486

Related: bazelbuild#7072

RELNOTES[NEW]: Added --incompatible_windows_escape_jvm_flags flag: enables correct java_binary.jvm_flags and java_test.jvm_flags tokenization and escaping on Windows. (No-op on other platforms.)

Change-Id: I531cc63cdfeccbe4b6d48876cb82870c1726a723
laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Feb 21, 2019
Add --incompatible_windows_escape_jvm_flags flag
(default: false). This flag has no effect on
platforms other than Windows.

This flag affects how Bazel builds the launcher of
java_binary and java_test targets. (The launcher
is the .exe file that sets up the environment for
the Java program and launches the JVM.)

In particular, the flag controls whether or not
Bazel will Bash-tokenize the jvm_flags that were
declared in the BUILD file, and whether or not
these jvm_flags are escaped by the launcher when
it invokes the JVM.

When the flag is enabled:

- Bazel will Bash-tokenize java_binary.jvm_flags
  and java_test.jvm_flags (as documented by the
  Build Encyclopedia) before embedding them into
  the launcher. The build fails if an entry cannot
  be Bash-tokenized, e.g. if it has unterminated
  quotes.

- The launcher will properly escape these flags
  when it runs the JVM as a subprocess (using
  launcher_util::WindowsEscapeArg2).

- The result is that the jvm_flags declared in the
  BUILD file will arrive to the Java program as
  intended. However, due to the extra
  Bash-tokenization step, some ill-formed flags
  are no longer accepted, e.g.
  `jvm_flags=["-Dfoo='a"]` now results in a build
  error.

When the flag is disabled:

- Bazel does not Bash-tokenize the jvm_flags
  before embedding them in the launcher. This
  preserves quoting meant to be stripped away, and
  it also means Bazel won't check whether the
  argument is properly quoted.

- The launcher escapes the flags with a Bash-like
  escaping logic (launcher_util::WindowsEscapeArg)
  which cannot properly quote and escape
  everything.

- The result: the jvm_flags declared in the BUILD
  file might get messed up as they are passed to
  the JVM, or the launcher may not even be able to
  run the JVM. However, due to the lack of
  Bash-tokenization, Bazel propagates some flags
  to the Java binary that it would no longer
  accept if the new
  `--incompatible_windows_escape_jvm_flags` were
  enabled, e.g. `jvm_flags=["'a"]` is fine.

Incompatible flag: bazelbuild#7486

Related: bazelbuild#7072

RELNOTES[NEW]: Added --incompatible_windows_escape_jvm_flags flag: enables correct java_binary.jvm_flags and java_test.jvm_flags tokenization and escaping on Windows. (No-op on other platforms.)

Change-Id: I531cc63cdfeccbe4b6d48876cb82870c1726a723
bazel-io pushed a commit that referenced this issue Feb 26, 2019
Add --incompatible_windows_escape_jvm_flags flag
(default: false). This flag has no effect on
platforms other than Windows.

This flag affects how Bazel builds the launcher of
java_binary and java_test targets. (The launcher
is the .exe file that sets up the environment for
the Java program and launches the JVM.)

In particular, the flag controls whether or not
Bazel will Bash-tokenize the jvm_flags that were
declared in the BUILD file, and whether or not
these jvm_flags are escaped by the launcher when
it invokes the JVM.

When the flag is enabled:

- Bazel will Bash-tokenize java_binary.jvm_flags
  and java_test.jvm_flags (as documented by the
  Build Encyclopedia) before embedding them into
  the launcher. The build fails if an entry cannot
  be Bash-tokenized, e.g. if it has unterminated
  quotes.

- The launcher will properly escape these flags
  when it runs the JVM as a subprocess (using
  launcher_util::WindowsEscapeArg2).

- Result: the jvm_flags declared in the BUILD file
  will arrive to the Java program as intended.
  However, due to the extra Bash-tokenization
  step, some ill-formed flags are no longer
  accepted, e.g.  `jvm_flags=["-Dfoo='a"]` now
  results in a build error.

When the flag is disabled:

- Bazel does not Bash-tokenize the jvm_flags
  before embedding them in the launcher. This
  preserves quoting meant to be stripped away, and
  it also means Bazel won't check whether the
  argument is properly quoted.

- The launcher escapes the flags with a Bash-like
  escaping logic (launcher_util::WindowsEscapeArg)
  which cannot properly quote and escape
  everything.

- Result: the jvm_flags declared in the BUILD file
  might get messed up as they are passed to the
  JVM, or the launcher may not even be able to run
  the JVM. However, due to the lack of
  Bash-tokenization, Bazel propagates some flags
  to the Java binary that it would no longer
  accept if the new
  `--incompatible_windows_escape_jvm_flags` were
  enabled, e.g. `jvm_flags=["'a"]` is fine.

Incompatible flag: #7486

Related: #7072

RELNOTES[NEW]: Added --incompatible_windows_escape_jvm_flags flag: enables correct java_binary.jvm_flags and java_test.jvm_flags tokenization and escaping on Windows. (No-op on other platforms.)

Change-Id: I531cc63cdfeccbe4b6d48876cb82870c1726a723

Closes #7490.

PiperOrigin-RevId: 235700143
@laszlocsomor
Copy link
Contributor

Closing this, because it's fixed at HEAD, see #7486. That bug will be closed when the flag's default value is flipped to true at HEAD.

@nfelt
Copy link
Author

nfelt commented Mar 1, 2019

Thank you for fixing this!

@philwo philwo added the team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website label Jun 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-Windows Windows-specific issues and feature requests P2 We'll consider working on this in future. (Assignee optional) team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website type: bug
Projects
None yet
Development

No branches or pull requests

4 participants