Windows, testing: make all tests work on Windows

[laszlocsomor]

As of today, there are 282 tests that Bazel doesn't run on Windows.

EDIT(2017-12-13 15:52 +0100): we need to consider tests that check the current platform too, e.g. in Java do things like OS.getCurrent() == OS.WINDOWS.

C:\work\bazel>bazel --output_user_root=c:\tmp2 query "tests(//...)-tests(//src:all_windows_tests)"
//tools/test:LcovMergerTest
//tools/jdk:proguard_whitelister_test
//tools/build_defs/pkg:path_test
//tools/build_defs/pkg:make_rpm_test
//tools/build_defs/pkg:build_test
//tools/build_defs/pkg:archive_test
//tools/build_defs/docker:rewrite_json_test
//tools/build_defs/docker:create_image_config_test
//tools/android:stubify_manifest_test
//tools/android:strip_resources_test
//tools/android:resource_extractor_test
//tools/android:merge_manifests_test
//tools/android:junction_test
//tools/android:incremental_install_test
//tools/android:build_split_manifest_test
//tools/android:build_incremental_dexmanifest_test
//tools/android:aar_resources_extractor_test
//tools/android:aar_native_libs_zip_creator_test
//tools/android:aar_embedded_jars_extractor_test
//third_party/protobuf/3.4.0:win32_test
//third_party/protobuf/3.4.0:py_wire_format_test
//third_party/protobuf/3.4.0:py_unknown_fields_test
//third_party/protobuf/3.4.0:py_text_format_test
//third_party/protobuf/3.4.0:py_text_encoding_test
//third_party/protobuf/3.4.0:py_symbol_database_test
//third_party/protobuf/3.4.0:py_service_reflection_test
//third_party/protobuf/3.4.0:py_reflection_test
//third_party/protobuf/3.4.0:py_proto_builder_test
//third_party/protobuf/3.4.0:py_message_test
//third_party/protobuf/3.4.0:py_message_factory_test
//third_party/protobuf/3.4.0:py_json_format_test
//third_party/protobuf/3.4.0:py_generator_test
//third_party/protobuf/3.4.0:py_descriptor_test
//third_party/protobuf/3.4.0:py_descriptor_pool_test
//third_party/protobuf/3.4.0:py_descriptor_database_test
//third_party/protobuf/3.4.0:protobuf_test
//third_party/ijar/test:zip_test
//third_party/ijar/test:ijar_test
//third_party/ijar/test:IjarTests
//src/tools/skylark/javatests/com/google/devtools/skylark/skylint:SkylintTests
//src/tools/singlejar:zlib_interface_test
//src/tools/singlejar:zip_headers_test
//src/tools/singlejar:zip64_test
//src/tools/singlejar:transient_bytes_test
//src/tools/singlejar:token_stream_test
//src/tools/singlejar:output_jar_simple_test
//src/tools/singlejar:output_jar_bash_test
//src/tools/singlejar:options_test
//src/tools/singlejar:input_jar_scan_ziptool_test
//src/tools/singlejar:input_jar_scan_jartool_test
//src/tools/singlejar:input_jar_preambled_test
//src/tools/singlejar:input_jar_empty_jar_test
//src/tools/singlejar:input_jar_bad_jar_test
//src/tools/singlejar:combiners_test
//src/test/skylark/skylint:skylint_test
//src/test/skylark:skylark_test_testdata/string_test_characters_sky
//src/test/skylark:skylark_test_testdata/string_splitlines_sky
//src/test/skylark:skylark_test_testdata/string_split_sky
//src/test/skylark:skylark_test_testdata/string_partition_sky
//src/test/skylark:skylark_test_testdata/string_format_sky
//src/test/skylark:skylark_test_testdata/min_max_sky
//src/test/skylark:skylark_test_testdata/list_slices_sky
//src/test/skylark:skylark_test_testdata/int_sky
//src/test/skylark:skylark_test_testdata/int_constructor_sky
//src/test/skylark:skylark_test_testdata/equality_sky
//src/test/skylark:skylark_test_testdata/and_or_not_sky
//src/test/skylark:skylark_test_testdata/all_any_sky
//src/test/skylark:question_regexp_1
//src/test/skylark:question_regexp
//src/test/skylark:question_content
//src/test/skylark:assert_tone_below_1
//src/test/shell/integration:ui_test
//src/test/shell/integration:test_test
//src/test/shell/integration:stub_finds_runfiles_test
//src/test/shell/integration:startup_options_test
//src/test/shell/integration:stamping_test
//src/test/shell/integration:skylark_flag_test
//src/test/shell/integration:runfiles_test
//src/test/shell/integration:run_test
//src/test/shell/integration:rc_options_test
//src/test/shell/integration:python_test
//src/test/shell/integration:progress_reporting_test
//src/test/shell/integration:process_wrapper_test
//src/test/shell/integration:outputs_test
//src/test/shell/integration:output_filter_test
//src/test/shell/integration:logging_test
//src/test/shell/integration:loading_phase_tests
//src/test/shell/integration:linux_sandbox_test
//src/test/shell/integration:linux_sandbox_network_test
//src/test/shell/integration:java_integration_test
//src/test/shell/integration:incompatible_changes_conflict_test
//src/test/shell/integration:force_delete_output_test
//src/test/shell/integration:experimental_ui_test
//src/test/shell/integration:execution_phase_tests
//src/test/shell/integration:discard_graph_edges_test
//src/test/shell/integration:discard_analysis_cache_test
//src/test/shell/integration:cpp_test
//src/test/shell/integration:configured_query_test
//src/test/shell/integration:client_sigint_test
//src/test/shell/integration:build_event_stream_test
//src/test/shell/integration:bazel_worker_test
//src/test/shell/integration:bazel_testjobs_test
//src/test/shell/integration:bazel_sandboxed_worker_test
//src/test/shell/integration:bazel_query_test
//src/test/shell/integration:bazel_javabase_test
//src/test/shell/integration:bazel_command_log_test
//src/test/shell/integration:action_env_test
//src/test/shell/bazel/apple:bazel_objc_test
//src/test/shell/bazel/apple:bazel_apple_test
//src/test/shell/bazel/android:proguard_integration_test
//src/test/shell/bazel/android:desugarer_integration_test
//src/test/shell/bazel/android:android_sdk_integration_test
//src/test/shell/bazel/android:android_ndk_integration_test
//src/test/shell/bazel/android:android_integration_test
//src/test/shell/bazel/android:aidl_integration_test
//src/test/shell/bazel/android:aar_integration_test
//src/test/shell/bazel:workspace_test
//src/test/shell/bazel:toolchain_test
//src/test/shell/bazel:srcs_test
//src/test/shell/bazel:skylark_repository_test
//src/test/shell/bazel:skylark_git_repository_test
//src/test/shell/bazel:runfiles_test
//src/test/shell/bazel:rule_test_test
//src/test/shell/bazel:remote_execution_test
//src/test/shell/bazel:remote_execution_sandboxing_test
//src/test/shell/bazel:remote_execution_rest_test
//src/test/shell/bazel:persistent_test_runner_test
//src/test/shell/bazel:maven_test
//src/test/shell/bazel:maven_skylark_test
//src/test/shell/bazel:location_test
//src/test/shell/bazel:local_repository_test_jdk8
//src/test/shell/bazel:local_repository_test
//src/test/shell/bazel:local_action_cache_test
//src/test/shell/bazel:java_launcher_test
//src/test/shell/bazel:help_test
//src/test/shell/bazel:git_repository_test
//src/test/shell/bazel:external_skylark_load_test
//src/test/shell/bazel:external_integration_test
//src/test/shell/bazel:external_correctness_test
//src/test/shell/bazel:execroot_test
//src/test/shell/bazel:empty_package_test
//src/test/shell/bazel:embedded_tools_deps_test
//src/test/shell/bazel:cross_repository_test
//src/test/shell/bazel:cpp_darwin_integration_test
//src/test/shell/bazel:client_test
//src/test/shell/bazel:cc_static_binary_test
//src/test/shell/bazel:cc_integration_test
//src/test/shell/bazel:build_files_test
//src/test/shell/bazel:bound_targets_test
//src/test/shell/bazel:bazel_workspace_status_test
//src/test/shell/bazel:bazel_with_jdk_test
//src/test/shell/bazel:bazel_tools_test
//src/test/shell/bazel:bazel_toolchain_test
//src/test/shell/bazel:bazel_test_test
//src/test/shell/bazel:bazel_sandboxing_test
//src/test/shell/bazel:bazel_sandboxing_cpp_test
//src/test/shell/bazel:bazel_rules_test
//src/test/shell/bazel:bazel_repository_cache_test
//src/test/shell/bazel:bazel_random_characters_test
//src/test/shell/bazel:bazel_localtest_test
//src/test/shell/bazel:bazel_java_test
//src/test/shell/bazel:bazel_experimental_ui_test
//src/test/shell/bazel:bazel_execute_testlog
//src/test/shell/bazel:bazel_example_test
//src/test/shell/bazel:bazel_docgen_test
//src/test/shell/bazel:bazel_determinism_test
//src/test/shell/bazel:bazel_coverage_test
//src/test/shell/bazel:bazel_build_event_stream_test
//src/test/shell:shell_utils_test
//src/test/shell:bashunit_test
//src/test/java/com/google/devtools/build/lib/rules/objc:ObjcRulesTests
//src/test/java/com/google/devtools/build/lib/rules/cpp:cpp-rules-tests
//src/test/java/com/google/devtools/build/lib/rules/android:AndroidInstrumentationTestTest
//src/test/java/com/google/devtools/build/lib/profiler/memory:AllocationTrackerTest
//src/test/java/com/google/devtools/build/lib/profiler/callcounts:CallcountsTest
//src/test/java/com/google/devtools/build/lib/bazel/repository/downloader:DownloaderTestSuite
//src/test/java/com/google/devtools/build/lib/bazel/repository:RepositoryTests
//src/test/java/com/google/devtools/build/lib:standalone-tests
//src/test/java/com/google/devtools/build/lib:shell_test
//src/test/java/com/google/devtools/build/lib:server_test
//src/test/java/com/google/devtools/build/lib:sandbox-tests
//src/test/java/com/google/devtools/build/lib:foundations_test
//src/test/java/com/google/devtools/build/lib:analysis_test
//src/test/java/com/google/devtools/build/android/desugar/runtime:ThrowableExtensionTestWithReuseDesugaringStrategy
//src/test/java/com/google/devtools/build/android/desugar/runtime:ThrowableExtensionTestWithNullDesugaringStrategy
//src/test/java/com/google/devtools/build/android/desugar/runtime:ThrowableExtensionTestWithMimicDesugaringStrategy
//src/test/java/com/google/devtools/build/android/desugar:testdata_sanity_test
//src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_twice_jar_test
//src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_java8_jar_test
//src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_jar_test
//src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_for_try_with_resources_test_twice
//src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_for_try_with_resources_test
//src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_for_disabling_try_with_resources_with_large_minsdkversion_test
//src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_core_library_jar_test
//src/test/java/com/google/devtools/build/android/desugar:stateless_lambda_has_no_factory_method_test
//src/test/java/com/google/devtools/build/android/desugar:simple_method_reference_has_no_factory_method_test
//src/test/java/com/google/devtools/build/android/desugar:no_stubs_for_lambdas_in_implemented_interface
//src/test/java/com/google/devtools/build/android/desugar:jacoco_legacy_default_method_regression_test
//src/test/java/com/google/devtools/build/android/desugar:interface_has_method_bodies_removed
//src/test/java/com/google/devtools/build/android/desugar:initializer_of_functional_interface_should_not_execute
//src/test/java/com/google/devtools/build/android/desugar:inherited_abstract_method_gets_no_default_method_stub
//src/test/java/com/google/devtools/build/android/desugar:desugar_testdata_by_disabling_lambda_desugaring_test
//src/test/java/com/google/devtools/build/android/desugar:desugar_idempotency_test
//src/test/java/com/google/devtools/build/android/desugar:capture_lambda_has_no_factory_method_test
//src/test/java/com/google/devtools/build/android/desugar:TryWithResourcesRewriterUnitTestWithReuseStrategy
//src/test/java/com/google/devtools/build/android/desugar:TryWithResourcesRewriterUnitTestWithNullStrategy
//src/test/java/com/google/devtools/build/android/desugar:TryWithResourcesRewriterUnitTestWithMimicStrategy
//src/test/java/com/google/devtools/build/android/desugar:StackMapBugTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarTryWithResourcesFunctionalTestWithReuseStrategyTwice
//src/test/java/com/google/devtools/build/android/desugar:DesugarTryWithResourcesFunctionalTestWithReuseStrategy
//src/test/java/com/google/devtools/build/android/desugar:DesugarTryWithResourcesFunctionalTestWithNullStrategyTwice
//src/test/java/com/google/devtools/build/android/desugar:DesugarTryWithResourcesFunctionalTestWithNullStrategy
//src/test/java/com/google/devtools/build/android/desugar:DesugarTryWithResourcesFunctionalTestWithMimicStrategyTwice
//src/test/java/com/google/devtools/build/android/desugar:DesugarTryWithResourcesFunctionalTestWithMimicStrategy
//src/test/java/com/google/devtools/build/android/desugar:DesugarObjectsRequireNonNullTestForAndroidLintMode
//src/test/java/com/google/devtools/build/android/desugar:DesugarObjectsRequireNonNullTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarMainClassTestLambdaDirectoryIncorrectlySet
//src/test/java/com/google/devtools/build/android/desugar:DesugarMainClassTestLambdaDirectoryCorrectlySet
//src/test/java/com/google/devtools/build/android/desugar:DesugarLongCompareTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarLikeAndroidStudioFunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarJava8LikeAndroidStudioFunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarJava8FunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarIdempotencyFunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarFunctionalTestWithMultipleInputs
//src/test/java/com/google/devtools/build/android/desugar:DesugarFunctionalTestWithClasspathDirectory
//src/test/java/com/google/devtools/build/android/desugar:DesugarFunctionalTestFromDirectoryToJar
//src/test/java/com/google/devtools/build/android/desugar:DesugarFunctionalTestFromDirectoryToDirectory
//src/test/java/com/google/devtools/build/android/desugar:DesugarFunctionalTestForSyntheticMethodsWithLambdaNames
//src/test/java/com/google/devtools/build/android/desugar:DesugarFunctionalTestForConstantArgumentsInLambdas
//src/test/java/com/google/devtools/build/android/desugar:DesugarFunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarDefaultMethodsIdempotencyFunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarDefaultMethodsFunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarCoreLibraryFunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DefaultMethodClassFixerTest
//src/test/java/com/google/devtools/build/android/desugar:BytecodeTypeInferenceTest
//src/test/docker:test_cc_configure-ubuntu-16.04-opt
//src/test/docker:test_cc_configure-ubuntu-16.04-fastbuild
//src/test/docker:test_cc_configure-ubuntu-16.04-dbg
//src/test/docker:test_cc_configure-ubuntu-15.04-opt
//src/test/docker:test_cc_configure-ubuntu-15.04-fastbuild
//src/test/docker:test_cc_configure-ubuntu-15.04-dbg
//src/test/docker:test_cc_configure-fedora23-opt
//src/test/docker:test_cc_configure-fedora23-fastbuild
//src/test/docker:test_cc_configure-fedora23-dbg
//src/test/docker:test_cc_configure-debian-stretch-opt
//src/test/docker:test_cc_configure-debian-stretch-fastbuild
//src/test/docker:test_cc_configure-debian-stretch-dbg
//src/test/docker:test_cc_configure-centos6.7-opt
//src/test/docker:test_cc_configure-centos6.7-fastbuild
//src/test/docker:test_cc_configure-centos6.7-dbg
//src/java_tools/singlejar/javatests/com/google/devtools/build/zip:ZipTests
//src/java_tools/singlejar/javatests/com/google/devtools/build/singlejar:SingleJarTests
//src/java_tools/junitrunner/javatests/com/google/testing/junit/runner:utf8_test_log_test
//src/java_tools/junitrunner/javatests/com/google/testing/junit/runner:junit4_testbridge_integration_test
//src/java_tools/junitrunner/javatests/com/google/testing/junit/runner:deploy_jar_integration_test
//src/java_tools/junitrunner/javatests/com/google/testing/junit/runner:antxmlresultwriter_integration_test
//src/java_tools/junitrunner/javatests/com/google/testing/junit/runner:AllTests
//src/java_tools/junitrunner/javatests/com/google/testing/junit/junit4/runner:RegExTestCaseFilterTest
//src/java_tools/buildjar/javatests/com/google/devtools/build/java/turbine/javac:TreePrunerTest
//src/java_tools/buildjar/javatests/com/google/devtools/build/java/turbine/javac:ProcessorClasspathTest_bootclasspath
//src/java_tools/buildjar/javatests/com/google/devtools/build/java/turbine/javac:JavacTurbineTest
//src/java_tools/buildjar/javatests/com/google/devtools/build/java/bazel:BazelJavaCompilerTest
//src/java_tools/buildjar/javatests/com/google/devtools/build/buildjar:VanillaJavaBuilderTest
//scripts/release:relnotes_test
//scripts/release:release_test
//scripts:bash_completion_test
//examples/shell:test
//examples/py_native:test
//examples/py_native:fail
//examples/java-skylark/src/test/java/com/example/myproject:pass
//examples/java-skylark/src/test/java/com/example/myproject:fail
//examples/java-native/src/test/java/com/example/myproject:sleep3
//examples/java-native/src/test/java/com/example/myproject:sleep2
//examples/java-native/src/test/java/com/example/myproject:sleep1
//examples/java-native/src/test/java/com/example/myproject:sleep0
//examples/java-native/src/test/java/com/example/myproject:resource-fail
//examples/java-native/src/test/java/com/example/myproject:hello
//examples/java-native/src/test/java/com/example/myproject:fail
//examples/java-native/src/test/java/com/example/myproject:custom_with_test_class
//examples/java-native/src/test/java/com/example/myproject:custom
//examples/cpp:hello-success_test
//examples/cpp:hello-fail_test

We need to make them all run on Windows and turn them on on CI, in order to have full test coverage on Windows.

[laszlocsomor]

I'm tempted to make this bug P1.
Since we're about to plan the work for next 3 months, I'm marking this bug P1 and P2 at the same time, indicating that I'm not decided on priority yet, and making this bug easily discoverable.

[davido]

I'm, the most concerned about disabled buildjar tests:

//src/java_tools/buildjar/javatests/com/google/devtools/build/java/turbine/javac:TreePrunerTest
//src/java_tools/buildjar/javatests/com/google/devtools/build/java/turbine/javac:ProcessorClasspathTest_bootclasspath
//src/java_tools/buildjar/javatests/com/google/devtools/build/java/turbine/javac:JavacTurbineTest
//src/java_tools/buildjar/javatests/com/google/devtools/build/java/bazel:BazelJavaCompilerTest
//src/java_tools/buildjar/javatests/com/google/devtools/build/buildjar:VanillaJavaBuilderTest

TBH with you: I stopped working on gerrit build on Windows, untill these tests are up and running. I was not amused to find out, that transitive classpath calculation in buildjar was inherently broken on Windows, and that the existing test, that would probably catch this was disabled (and is still broken) on Windows.

[davido]

Strictly speaking, removing the warning from Bazel FAQ, that Windows support is bleeding edge, experimental and may produce unexpected results, was a mistake and this CL should be reverted. Nothing document this better as this issue.

[laszlocsomor]

David, I'm not sure what reaction you expect.

[pcj]

I'm seeing a problem that would likely be addressed by fixing these two tests:

//src/java_tools/singlejar/javatests/com/google/devtools/build/zip:ZipTests
//src/java_tools/singlejar/javatests/com/google/devtools/build/singlejar:SingleJarTests

ERROR: C:/src/jenova/test/baseline/BUILD.bazel:9:1: Building deploy jar test/baseline/ATE-369_xlsx_deploy.jar failed (Exit 1)
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 79
        at com.google.devtools.build.zip.ZipUtil.get16(ZipUtil.java:87)
        at com.google.devtools.build.zip.ZipUtil.getUnsignedShort(ZipUtil.java:119)
        at com.google.devtools.build.zip.ZipReader.findEndOfCentralDirectoryRecord(ZipReader.java:277)
        at com.google.devtools.build.zip.ZipReader.readCentralDirectory(ZipReader.java:194)
        at com.google.devtools.build.zip.ZipReader.<init>(ZipReader.java:94)
        at com.google.devtools.build.zip.ZipReader.<init>(ZipReader.java:74)
        at com.google.devtools.build.zip.ZipReader.<init>(ZipReader.java:62)
        at com.google.devtools.build.singlejar.ZipCombiner.addZip(ZipCombiner.java:431)
        at com.google.devtools.build.singlejar.SingleJar.run(SingleJar.java:240)
        at com.google.devtools.build.singlejar.SingleJar.singleRun(SingleJar.java:394)
        at com.google.devtools.build.singlejar.SingleJar.main(SingleJar.java:413)
INFO: Elapsed time: 2.851s, Critical Path: 2.17s

Seems to occur when building multiple java_test jars concurrently where there are shared deps or resources (not sure which one) between the multiple targets.

[michajlo]

I'm assuming this is why we have //src:all_windows_tests instead of running all the tests and opting out those that are broken? I'm worried that this abstraction may become (or has been becoming) brittle or broken as we restructure our BUILD files.

[laszlocsomor]

I'm assuming this is why we have //src:all_windows_tests instead of running all the tests and opting out those that are broken?

IIRC yes.

Today we can skip incompatible targets (yay!) so filtering out incompatible tests is easier.

sh_test(
    name = "only_on_linux_arm64",
    srcs = ["a.sh"],
    target_compatible_with = [
        "@platforms//cpu:aarch64",
        "@platforms//os:linux",
    ],
)

sh_test(
    name = "except_on_windows",
    srcs = ["a.sh"],
    target_compatible_with = select({
        "@platforms//os:windows": ["@platforms//:incompatible"],
        "//conditions:default": [],
    }),
)

[meteorcloudy]

Since #3742 is no longer an issue on BuildKite (because we don't run bazel query first), I think we can switch to the bazel test --test_tag_filters=-no_windows //... approach to run Windows tests.

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.