Bazel CI failure: Bazel test command is too long on Windows

[meteorcloudy]

http://ci.bazel.io/blue/rest/organizations/jenkins/pipelines/CR/pipelines/bazel-tests/runs/383/nodes/36/steps/468/log/?start=0

All tests passed, but we are getting an error:

ERROR: Skipping '//src/test/java/com/google/devtools/build/android/desugar:desugar_testdaa_by_disabling_lambda_desugaring_test': no such target '//src/test/java/com/google/devtools/build/android/desugar:desugar_testdaa_by_disabling_lambda_desugaring_test': target 'desugar_testdaa_by_disabling_lambda_desugaring_test' not declared in package 'src/test/java/com/google/devtools/build/android/desugar' (did you mean 'desugar_testdata_by_disabling_lambda_desugaring_test'?) defined by C:/jenkins/workspace/cr/bazel-tests-node=windows-x86_64,toolchain=msvc/src/test/java/com/google/devtools/build/android/desugar/BUILD.
ERROR: Skipping '//src/test/java/com/google/devtools/build/android/desugar:desugar_testdaa_by_disabling_lambda_desugaring_test': no such target '//src/test/java/com/google/devtools/build/android/desugar:desugar_testdaa_by_disabling_lambda_desugaring_test': target 'desugar_testdaa_by_disabling_lambda_desugaring_test' not declared in package 'src/test/java/com/google/devtools/build/android/desugar' (did you mean 'desugar_testdata_by_disabling_lambda_desugaring_test'?) defined by C:/jenkins/workspace/cr/bazel-tests-node=windows-x86_64,toolchain=msvc/src/test/java/com/google/devtools/build/android/desugar/BUILD.

I suspect this is because the bazel test command exceeds the max length (8K) of command line on Windows.

c:\bazel_ci\installs\latest\bazel.exe --bazelrc=c:\jenkins\workspace\CR\bazel-tests-node=windows-x86_64,toolchain=msvc/bazel.bazelrc test //third_party/def_parser:def_parser_test //src/tools/launcher/util:util_test //src/tools/launcher/util:data_parser_test //src/test/shell/bazel:bazel_windows_example_test //src/test/shell/bazel:bazel_bootstrap_distfile_test //src/test/py/bazel:launcher_test //src/test/py/bazel:bazel_windows_test //src/test/py/bazel:bazel_windows_dynamic_link_test //src/test/py/bazel:bazel_server_mode_test //src/test/py/bazel:bazel_external_repository_test //src/test/py/bazel:bazel_clean_test //src/test/native:windows_jni_test //src/test/java/com/google/devtools/common/options/testing:OptionsTesterTest //src/test/java/com/google/devtools/common/options:options_test //src/test/java/com/google/devtools/build/skyframe:skyframe_base_test //src/test/java/com/google/devtools/build/lib/skylark:SkylarkTests //src/test/java/com/google/devtools/build/lib/skyframe/serialization:SerializationTests //src/test/java/com/google/devtools/build/lib/skyframe/packages:BazelPackageLoaderTest //src/test/java/com/google/devtools/build/lib/skyframe:SkyframeTests //src/test/java/com/google/devtools/build/lib/rules/repository:RepositoryTests //src/test/java/com/google/devtools/build/lib/rules/platform:PlatformRulesTests //src/test/java/com/google/devtools/build/lib/rules/objc:ObjcRulesTests //src/test/java/com/google/devtools/build/lib/rules/config:ConfigRulesTests //src/test/java/com/google/devtools/build/lib/rules/apple:AppleRulesTests //src/test/java/com/google/devtools/build/lib/rules/apple:AppleConfigurationSerializationTest //src/test/java/com/google/devtools/build/lib/rules/android:ResourceFilterTest //src/test/java/com/google/devtools/build/lib/rules/android:LocalResourceContainerTest //src/test/java/com/google/devtools/build/lib/rules/android:ApkManifestActionTest //src/test/java/com/google/devtools/build/lib/rules/android:AndroidSkylarkSplitTransitionTest //src/test/java/com/google/devtools/build/lib/rules/android:AndroidLibraryTest //src/test/java/com/google/devtools/build/lib/rules/android:AndroidInstrumentationTestTest //src/test/java/com/google/devtools/build/lib/rules/android:AndroidInstrumentationRuleImplTest //src/test/java/com/google/devtools/build/lib/rules/android:AndroidHostServiceFixtureTest //src/test/java/com/google/devtools/build/lib/rules/android:AndroidDeviceTest //src/test/java/com/google/devtools/build/lib/rules/android:AndroidDeviceScriptFixtureTest //src/test/java/com/google/devtools/build/lib/rules/android:AndroidDataBindingTest //src/test/java/com/google/devtools/build/lib/rules/android:AndroidCommonTest //src/test/java/com/google/devtools/build/lib/rules/android:AndroidBinaryTest //src/test/java/com/google/devtools/build/lib/rules/android:AndroidBinaryMultidexTest //src/test/java/com/google/devtools/build/lib/buildtool:BuildtoolTests //src/test/java/com/google/devtools/build/lib/buildeventstream/transports:BuildEventTransportTest //src/test/java/com/google/devtools/build/lib/buildeventservice:BuildEventServiceTest //src/test/java/com/google/devtools/build/lib/bazel/repository/downloader:DownloaderTestSuite //src/test/java/com/google/devtools/build/lib/bazel/repository/cache:RepositoryCacheTests //src/test/java/com/google/devtools/build/lib/bazel/repository:RepositoryTests //src/test/java/com/google/devtools/build/lib/analysis/whitelisting:WhitelistingTests //src/test/java/com/google/devtools/build/lib/analysis/platform:PlatformAnalysisTests //src/test/java/com/google/devtools/build/lib:worker-tests //src/test/java/com/google/devtools/build/lib:windows_test //src/test/java/com/google/devtools/build/lib:windows-tests //src/test/java/com/google/devtools/build/lib:test-rules-tests //src/test/java/com/google/devtools/build/lib:syntax_test //src/test/java/com/google/devtools/build/lib:standalone-tests //src/test/java/com/google/devtools/build/lib:shell_test //src/test/java/com/google/devtools/build/lib:server_test //src/test/java/com/google/devtools/build/lib:sandbox-tests //src/test/java/com/google/devtools/build/lib:runtime-tests //src/test/java/com/google/devtools/build/lib:rules-tests //src/test/java/com/google/devtools/build/lib:remote-tests //src/test/java/com/google/devtools/build/lib:profiler-tests //src/test/java/com/google/devtools/build/lib:pkgcache_test //src/test/java/com/google/devtools/build/lib:packages_test //src/test/java/com/google/devtools/build/lib:java-rules-tests //src/test/java/com/google/devtools/build/lib:graph_test //src/test/java/com/google/devtools/build/lib:foundations_test //src/test/java/com/google/devtools/build/lib:filegroup-rules-tests //src/test/java/com/google/devtools/build/lib:exec-tests //src/test/java/com/google/devtools/build/lib:exec-local-tests //src/test/java/com/google/devtools/build/lib:cpp-rules-tests //src/test/java/com/google/devtools/build/lib:bazel-rules-tests //src/test/java/com/google/devtools/build/lib:analysis_test //src/test/java/com/google/devtools/build/lib:analysis_select_test //src/test/java/com/google/devtools/build/lib:analysis_constraints_test //src/test/java/com/google/devtools/build/lib:analysis_config_test //src/test/java/com/google/devtools/build/lib:analysis_actions_test //src/test/java/com/google/devtools/build/lib:actions_test //src/test/java/com/google/devtools/build/lib:StrictDepsUtilsTest //src/test/java/com/google/devtools/build/lib:ProtoLangToolchainTest //src/test/java/com/google/devtools/build/lib:ProtoCompileActionBuilderTest //src/test/java/com/google/devtools/build/lib:CcProtoLibraryTest //src/test/java/com/google/devtools/build/lib:BazelProtoLibraryTest //src/test/java/com/google/devtools/build/lib:BazelDocumentationTests //src/test/java/com/google/devtools/build/docgen:DocumentationTests //src/test/java/com/google/devtools/build/android/ziputils:ziputils-tests //src/test/java/com/google/devtools/build/android/resources:RClassGeneratorTest //src/test/java/com/google/devtools/build/android/junctions:JunctionCreatorTest //src/test/java/com/google/devtools/build/android/idlclass:IdlClassTest //src/test/java/com/google/devtools/build/android/dexer:AllTests //src/test/java/com/google/devtools/build/android/desugar/runtime:ThrowableExtensionTestWithReuseDesugaringStrategy //src/test/java/com/google/devtools/build/android/desugar/runtime:ThrowableExtensionTestWithNullDesugaringStrategy //src/test/java/com/google/devtools/build/android/desugar/runtime:ThrowableExtensionTestWithMimicDesugaringStrategy //src/test/java/com/google/devtools/build/android/desugar/runtime:ConcurrentWeakIdentityHashMapTest //src/test/java/com/google/devtools/build/android/desugar:testdata_sanity_test //src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_twice_jar_test //src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_java8_jar_test //src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_jar_test //src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_for_try_with_resources_test_twice //src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_for_try_with_resources_test //src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_for_disabling_try_with_resources_with_large_minsdkversion_test //src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_core_library_jar_test //src/test/java/com/google/devtools/build/android/desugar:stateless_lambda_has_no_factory_method_test //src/test/java/com/google/devtools/build/android/desugar:simple_method_reference_has_no_factory_method_test //src/test/java/com/google/devtools/build/android/desugar:no_stubs_for_lambdas_in_implemented_interface //src/test/java/com/google/devtools/build/android/desugar:jacoco_legacy_default_method_regression_test //src/test/java/com/google/devtools/build/android/desugar:interface_has_method_bodies_removed //src/test/java/com/google/devtools/build/android/desugar:initializer_of_functional_interface_should_not_execute //src/test/java/com/google/devtools/build/android/desugar:inherited_abstract_method_gets_no_default_method_stub //src/test/java/com/google/devtools/build/android/desugar:desugar_testda

[meteorcloudy]

@damienmg , the tests are returned by bazel query command, can we just run the tests instead of doing a query first?

[damienmg]

That would create other issues but we could if we replace all the place we use queries to select tests (I have tried to do so for some of them, query are complex to write).

…

On Fri, Sep 15, 2017, 3:27 PM Yun Peng ***@***.***> wrote: @damienmg <https://github.com/damienmg> , the tests are returned by bazel query command, can we just run the test instead of do a query first? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3742 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADjHf4vCaQ-d9Bq1sX29xQy8h3YP1e7Bks5sintFgaJpZM4PY9xJ> .

[davido]

Why not just to use the standard approach on windows and provide respone file as input instead of too long command line? This is what LibreOffice build tool chain is doing in all places.

See for example: [1].

• [1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa367156(v=vs.85).aspx

[damienmg]

This is what all our intermediate tool support, we could certainly add that support to Bazel.

…

On Mon, Sep 18, 2017, 9:09 AM David Ostrovsky ***@***.***> wrote: Why not just to use the satndard approach on windows and providde respone file as input instead of too long command line? This is what LibreOffice build tool chain is doing in all places. https://msdn.microsoft.com/en-us/library/windows/desktop/aa367156(v=vs.85).aspx — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3742 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADjHf8-CQdItqF_U0FQVzb3o-KDrX2ttks5sjhcNgaJpZM4PY9xJ> .

[meteorcloudy]

FYI, @laszlocsomor . This is why Windows is still red while all tests passes.
http://ci.bazel.io/blue/rest/organizations/jenkins/pipelines/bazel-tests/runs/1030/nodes/31/steps/463/log/?start=0

[laszlocsomor]

Thanks @meteorcloudy !

I also see this in the log:

ERROR: Skipping '//src/test/java/om/google/devtools/build/android/desugar:desugar_idempotency_test': no such package 'src/test/java/om/google/devtools/build/android/desugar': BUILD file not found on package path.
ERROR: Skipping '//src/test/java/om/google/devtools/build/android/desugar:desugar_idempotency_test': no such package 'src/test/java/om/google/devtools/build/android/desugar': BUILD file not found on package path.

Notice it says "java/om/google/..." instead of "java/com/google/..."

[meteorcloudy]

Yes, I noticed the same thing, I guess it's somehow caused by the command line being too long?

[laszlocsomor]

I think we can work around this problem if we create test_suite rules.

[laszlocsomor]

New occurrence:

http://ci.bazel.io/view/Dashboard/job/bazel-tests/1036/consoleFull

[node=windows-x86_64,toolchain=msvc] ERROR: Skipping '//src/test/java/com/google/devtools/build/android/desugar:desugar_testdata_y_disabling_lambda_desugaring_test': no such target '//src/test/java/com/google/devtools/build/android/desugar:desugar_testdata_y_disabling_lambda_desugaring_test': target 'desugar_testdata_y_disabling_lambda_desugaring_test' not declared in package 'src/test/java/com/google/devtools/build/android/desugar' (did you mean 'desugar_testdata_by_disabling_lambda_desugaring_test'?) defined by C:/jenkins/workspace/bazel-tests-node=windows-x86_64,toolchain=msvc/src/test/java/com/google/devtools/build/android/desugar/BUILD.
[node=windows-x86_64,toolchain=msvc] ERROR: Skipping '//src/test/java/com/google/devtools/build/android/desugar:desugar_testdata_y_disabling_lambda_desugaring_test': no such target '//src/test/java/com/google/devtools/build/android/desugar:desugar_testdata_y_disabling_lambda_desugaring_test': target 'desugar_testdata_y_disabling_lambda_desugaring_test' not declared in package 'src/test/java/com/google/devtools/build/android/desugar' (did you mean 'desugar_testdata_by_disabling_lambda_desugaring_test'?) defined by C:/jenkins/workspace/bazel-tests-node=windows-x86_64,toolchain=msvc/src/test/java/com/google/devtools/build/android/desugar/BUILD.
...
[node=windows-x86_64,toolchain=msvc] ERROR: command succeeded, but there were errors parsing the target pattern.
...
[node=windows-x86_64,toolchain=msvc] ERROR: command succeeded, but there were errors parsing the target pattern.

[aj-michael]

Am I understanding this correctly that Windows is removing characters from the middle of the command line when the length is too long? testdata --> testdaa, com --> om, by --> y

[laszlocsomor]

I don't know what mangled the pattern names in the error message... But I have a fix for the long command problem: https://bazel-review.googlesource.com/c/bazel/+/16910

[laszlocsomor]

related commit: bazelbuild/continuous-integration@61b5986

[damienmg]

This one is fixed, once and for all :)