Bazel's shell tests on Windows are noisy unless you use "bazel shutdown"

[philsc]

Description of the problem / feature request:

When running a shell test, you get lots of noise about not being able to remove directories at the end of a test.
I was able to fix this by running bazel shutdown in the tear_down() helper.

@katre suggested that there's likely a good way to fix this for all tests.

Here's the output without `bazel shutdown':

$ bazel test src/test/shell/integration:target_compatible_with_external_repo_test --test_output=streamed
WARNING: Streamed test output requested. All tests will be run locally, without sharding, one at a time
Loading:
Loading: 0 packages loaded
DEBUG: C:/msys64/home/gamer/_bazel_gamer/sawo4taj/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:68:14:
Current running Bazel is ahead of bazel-toolchains repo. Please update your pin to bazel-toolchains repo in your WORKSPACE file.
DEBUG: C:/msys64/home/gamer/_bazel_gamer/sawo4taj/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:103:14: rbe_ubuntu1804_java11 not using checked in configs as detect_java_home was set to True
DEBUG: C:/msys64/home/gamer/_bazel_gamer/sawo4taj/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:68:14:
Current running Bazel is ahead of bazel-toolchains repo. Please update your pin to bazel-toolchains repo in your WORKSPACE file.
DEBUG: C:/msys64/home/gamer/_bazel_gamer/sawo4taj/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:103:14: rbe_ubuntu1604_java8 not using checked in configs as detect_java_home was set to True
Analyzing: target //src/test/shell/integration:target_compatible_with_external_repo_test (0 packages loaded, 0 targets configured)
INFO: Analyzed target //src/test/shell/integration:target_compatible_with_external_repo_test (0 packages loaded, 2 targets configured).
INFO: Found 1 test target...
[0 / 4] [Prepa] BazelWorkspaceStatusAction stable-status.txt
[5 / 6] Testing //src/test/shell/integration:target_compatible_with_external_repo_test; 1s local
INFO[target_compatible_with_external_repo_test 2020-11-02 01:50:17 (+0000)] bazel binary is at /home/gamer/bazel/src/test/shell/bin
INFO[target_compatible_with_external_repo_test 2020-11-02 01:50:18 (+0000)] setting up client in C:/msys64/home/gamer/_bazel_gamer/sawo4taj/execroot/io_bazel/_tmp/db8fcff708fc08e7272a54cb3a2462cd/workspace

target_compatible_with_external_repo tests

Mon Nov  2 01:50:20 UTC 2020
** test_failure_on_incompatible_top_level_target_in_external_repo **************
INFO[target_compatible_with_external_repo_test 2020-11-02 01:50:28 (+0000)] Cleaning up workspace
rm: cannot remove 'target_skipping': Device or resource busy
rm: cannot remove 'target_skipping': Device or resource busy
rm: cannot remove 'target_skipping': Device or resource busy
rm: cannot remove 'target_skipping': Device or resource busy
rm: cannot remove 'target_skipping': Device or resource busy
rm: cannot remove 'target_skipping': Device or resource busy
rm: cannot remove 'target_skipping': Device or resource busy
[5 / 6] Testing //src/test/shell/integration:target_compatible_with_external_repo_test; 22s local
rm: cannot remove 'target_skipping': Device or resource busy
rm: cannot remove 'target_skipping': Device or resource busy
rm: cannot remove 'target_skipping': Device or resource busy
INFO[target_compatible_with_external_repo_test 2020-11-02 01:50:40 (+0000)] try_with_timeout(rm -fr target_skipping): no success after 10 seconds (timeout in 110 seconds)
rm: cannot remove 'target_skipping': Device or resource busy
rm: cannot remove 'target_skipping': Device or resource busy
rm: cannot remove 'target_skipping': Device or resource busy

PASSED: test_failure_on_incompatible_top_level_target_in_external_repo

Mon Nov  2 01:50:45 UTC 2020
** test_repository_defines_target_compatible_with ******************************
INFO[target_compatible_with_external_repo_test 2020-11-02 01:50:49 (+0000)] Cleaning up workspace

PASSED: test_repository_defines_target_compatible_with

** 2 / 2 tests passed. *********************************************************
$TEST_TMPDIR defined: output root default is 'c:\msys64\home\gamer\_bazel_gamer\sawo4taj\execroot\io_bazel\_tmp\db8fcff708fc08e7272a54cb3a2462cd' and max_idle_secs default is '15'.
[5 / 6] 1 / 1 tests; Testing //src/test/shell/integration:target_compatible_with_external_repo_test; 36s local
Target //src/test/shell/integration:target_compatible_with_external_repo_test up-to-date:
  bazel-bin/src/test/shell/integration/target_compatible_with_external_repo_test
  bazel-bin/src/test/shell/integration/target_compatible_with_external_repo_test.exe
INFO: Elapsed time: 37.655s, Critical Path: 36.10s
INFO: 6 processes: 5 internal, 1 local.
INFO: Build completed successfully, 6 total actions
//src/test/shell/integration:target_compatible_with_external_repo_test   PASSED in 36.0s

Executed 1 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
INFO: Build completed successfully, 6 total actions

Here's the output with bazel shutdown:

$ bazel test src/test/shell/integration:target_compatible_with_external_repo_test --test_output=streamed
WARNING: Streamed test output requested. All tests will be run locally, without sharding, one at a time
Loading:
Loading: 0 packages loaded
DEBUG: C:/msys64/home/gamer/_bazel_gamer/sawo4taj/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:68:14:
Current running Bazel is ahead of bazel-toolchains repo. Please update your pin to bazel-toolchains repo in your WORKSPACE file.
DEBUG: C:/msys64/home/gamer/_bazel_gamer/sawo4taj/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:103:14: rbe_ubuntu1804_java11 not using checked in configs as detect_java_home was set to True
DEBUG: C:/msys64/home/gamer/_bazel_gamer/sawo4taj/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:68:14:
Current running Bazel is ahead of bazel-toolchains repo. Please update your pin to bazel-toolchains repo in your WORKSPACE file.
DEBUG: C:/msys64/home/gamer/_bazel_gamer/sawo4taj/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:103:14: rbe_ubuntu1604_java8 not using checked in configs as detect_java_home was set to True
Analyzing: target //src/test/shell/integration:target_compatible_with_external_repo_test (0 packages loaded, 0 targets configured)
INFO: Analyzed target //src/test/shell/integration:target_compatible_with_external_repo_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
[0 / 3] [Prepa] BazelWorkspaceStatusAction stable-status.txt
[2 / 3] Testing //src/test/shell/integration:target_compatible_with_external_repo_test; 1s local
INFO[target_compatible_with_external_repo_test 2020-11-02 01:51:09 (+0000)] bazel binary is at /home/gamer/bazel/src/test/shell/bin
INFO[target_compatible_with_external_repo_test 2020-11-02 01:51:09 (+0000)] setting up client in C:/msys64/home/gamer/_bazel_gamer/sawo4taj/execroot/io_bazel/_tmp/db8fcff708fc08e7272a54cb3a2462cd/workspace

target_compatible_with_external_repo tests

Mon Nov  2 01:51:11 UTC 2020
** test_failure_on_incompatible_top_level_target_in_external_repo **************
$TEST_TMPDIR defined: output root default is 'c:\msys64\home\gamer\_bazel_gamer\sawo4taj\execroot\io_bazel\_tmp\db8fcff708fc08e7272a54cb3a2462cd' and max_idle_secs default is '15'.
INFO[target_compatible_with_external_repo_test 2020-11-02 01:51:21 (+0000)] Cleaning up workspace

PASSED: test_failure_on_incompatible_top_level_target_in_external_repo

Mon Nov  2 01:51:23 UTC 2020
** test_repository_defines_target_compatible_with ******************************
$TEST_TMPDIR defined: output root default is 'c:\msys64\home\gamer\_bazel_gamer\sawo4taj\execroot\io_bazel\_tmp\db8fcff708fc08e7272a54cb3a2462cd' and max_idle_secs default is '15'.
INFO[target_compatible_with_external_repo_test 2020-11-02 01:51:25 (+0000)] Cleaning up workspace

PASSED: test_repository_defines_target_compatible_with

** 2 / 2 tests passed. *********************************************************
$TEST_TMPDIR defined: output root default is 'c:\msys64\home\gamer\_bazel_gamer\sawo4taj\execroot\io_bazel\_tmp\db8fcff708fc08e7272a54cb3a2462cd' and max_idle_secs default is '15'.
[2 / 3] Testing //src/test/shell/integration:target_compatible_with_external_repo_test; 21s local
Target //src/test/shell/integration:target_compatible_with_external_repo_test up-to-date:
  bazel-bin/src/test/shell/integration/target_compatible_with_external_repo_test
  bazel-bin/src/test/shell/integration/target_compatible_with_external_repo_test.exe
INFO: Elapsed time: 23.235s, Critical Path: 22.14s
INFO: 3 processes: 2 internal, 1 local.
INFO: Build completed successfully, 3 total actions
//src/test/shell/integration:target_compatible_with_external_repo_test   PASSED in 22.0s

Executed 1 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
INFO: Build completed successfully, 3 total actions

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

See the snippets above.

What operating system are you running Bazel on?

This is Windows 10.

What's the output of bazel info release?

$ bazel info release
release 3.7.0

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

$ git remote get-url origin ; git rev-parse master ; git rev-parse HEAD
https://github.com/bazelbuild/bazel
08a7179e982f981bf9555d25964855a5507571e9
27264d4639feeac65a2d304783737448336cbb59

Any other information, logs, or outputs that you want to share?

This came up during code review for #10945.

[katre]

We should probably just update test teardown to always shut down the bazel server, on all platforms.

[katre]

Looks like the appropriate hook would be testenv.sh#testenv_tear_down, except that we're already shutting down bazel in testenv.sh#cleanup, which is run when the entire test file ends. I'm not sure what the effect of doing shutdown all the time would be, so I'll defer to @philwo, who appears to have updated that in the past.