Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bazel CI: Projects are failing on Windows with remote caching #9072

Closed
meteorcloudy opened this issue Aug 5, 2019 · 5 comments
Closed

Bazel CI: Projects are failing on Windows with remote caching #9072

meteorcloudy opened this issue Aug 5, 2019 · 5 comments
Labels
breakage P1 I'll work on this now. (Assignee required) team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website type: bug

Comments

@meteorcloudy
Copy link
Member

https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1122#4891fa47-6f07-47c0-98d0-fe3b16ec23dc

02:05:30) WARNING: Reading from Remote Cache:
--
  | D:/b/pwvxvp3w/execroot/io_bazel/bazel-out/x64_windows-fastbuild/testlogs/src/test/java/com/google/devtools/build/lib/profiler/memory/AllocationTrackerTest/test.log (Permission denied)
  | (02:05:30) INFO: From Executing genrule //src:java_tools_java9_build_zip:
  | adding: BUILD (deflated 86%)
  | (02:05:30) ERROR: D:/b/bk-windows-java8-3111/bazel-downstream-projects/bazel/src/test/java/com/google/devtools/build/lib/profiler/memory/BUILD:9:1: Couldn't build file src/test/java/com/google/devtools/build/lib/profiler/memory/AllocationTrackerTest/test.log:  failed (Exit -1). Note: Remote connection/protocol failed with: execution failed

Happens to Android, Bazel, rules_nodejs, rules_jvm_external

Bisecting didn't give any reliable result, the build failure seems to be flaky.
@meteorcloudy meteorcloudy added breakage P1 I'll work on this now. (Assignee required) type: bug under investigation labels Aug 5, 2019
@meteorcloudy
Copy link
Member Author

/cc @buchgr

@meteorcloudy
Copy link
Member Author

meteorcloudy commented Aug 5, 2019 •
edited
Loading

https://buildkite.com/bazel/culprit-finder/builds/194#8eb8f7cd-36b1-4dc1-b7eb-676566d21768

A bisect with REPEAT_TIMES=10 gives a6b5b05 as the culprit, @buchgr is looking into it.

@meteorcloudy
Copy link
Member Author

Tried with Bazel 0.28.1's baseline with rules_nodejs, I can reproduce the same failure:
https://buildkite.com/bazel/culprit-finder/builds/200#bce84699-40d5-4ba3-b478-9f9b3ebd6335

So a6b5b05 is probably not the culprit, the failure seems to be caused by some infrastructure change.

bazel-io pushed a commit that referenced this issue Aug 5, 2019
@meteorcloudy @copybara-github
Automated rollback of commit a6b5b05.
be26ba0 
*** Reason for rollback ***

Probably the culprit that broke Windows builds with remote caching: #9072

*** Original change description ***

Streamline uploading of stdout and stderr

This change removes extra upload logic for stdout/stderr. Besides
the streamlined code this also saves two round trips uploading an
action that produced both stdout and stderr.

Closes #9025.

PiperOrigin-RevId: 261700869
bazel-io pushed a commit that referenced this issue Aug 6, 2019
@buchgr @copybara-github
Automated rollback of commit be26ba0.
59b989e 
*** Reason for rollback ***

It was not the culprit

*** Original change description ***

Automated rollback of commit a6b5b05.

*** Reason for rollback ***

Probably the culprit that broke Windows builds with remote caching: #9072

*** Original change description ***

Streamline uploading of stdout and stderr

This change removes extra upload logic for stdout/stderr. Besides
the streamlined code this also saves two round trips uploading an
action that produced both stdout and s...

***

ROLLBACK_OF=261700869

RELNOTES: None
PiperOrigin-RevId: 261864282
@meteorcloudy
Copy link
Member Author

meteorcloudy commented Aug 6, 2019 •
edited
Loading

It turned out a6b5b05 poisoned the remote cache, that's why it sometimes fail with Bazel built at previous commit as well.

@meteorcloudy
Copy link
Member Author

@buchgr is working on a fix and we need to update the remote cache key to invalidate previous caching.

@meteorcloudy meteorcloudy added team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website and removed under investigation labels Aug 6, 2019
buchgr added a commit to buchgr/bazel that referenced this issue Aug 6, 2019
@buchgr
Fix incorrect stdout/stderr in remote action cache. Fixes bazelbuild#…
ee31742 
…9072

Commit a6b5b05 introduced a bug where the stdout/test.log of
an action was incorrectly added as an output file to the
ActionResult protobuf. Fortunately, we found the bug on Windows
because one cannot delete a file while it's still being open.

PAIR=pcloudy
@meteorcloudy meteorcloudy mentioned this issue Aug 6, 2019
Merged
@buchgr buchgr mentioned this issue Aug 6, 2019
Closed
katre pushed a commit that referenced this issue Aug 6, 2019
@buchgr @katre
Fix incorrect stdout/stderr in remote action cache. Fixes #9072
0f7b793 
Commit a6b5b05 introduced a bug where the stdout/test.log of
an action was incorrectly added as an output file to the
ActionResult protobuf. Fortunately, we found the bug on Windows
because one cannot delete a file while it's still being open.

PAIR=pcloudy

Closes #9087.

PiperOrigin-RevId: 261885751
katre pushed a commit that referenced this issue Aug 7, 2019
@buchgr @katre
Fix incorrect stdout/stderr in remote action cache. Fixes #9072
70a3e02 
Commit a6b5b05 introduced a bug where the stdout/test.log of
an action was incorrectly added as an output file to the
ActionResult protobuf. Fortunately, we found the bug on Windows
because one cannot delete a file while it's still being open.

PAIR=pcloudy

Closes #9087.

PiperOrigin-RevId: 261885751
katre pushed a commit that referenced this issue Aug 12, 2019
@buchgr @katre
Fix incorrect stdout/stderr in remote action cache. Fixes #9072
78c3fff 
Commit a6b5b05 introduced a bug where the stdout/test.log of
an action was incorrectly added as an output file to the
ActionResult protobuf. Fortunately, we found the bug on Windows
because one cannot delete a file while it's still being open.

PAIR=pcloudy

Closes #9087.

PiperOrigin-RevId: 261885751
bazel-io pushed a commit that referenced this issue Aug 28, 2019
@katre
Release 0.29.0 (2019-08-28)
f791df0 
Baseline: 6c5ef53

Cherry picks:

   + 338829f:
     Fix retrying of SocketTimeoutExceptions in HttpConnector
   + 14651cd:
     Fallback to next urls if download fails in HttpDownloader
   + b7d300c:
     Fix incorrect stdout/stderr in remote action cache. Fixes #9072
   + 9602176:
     Automated rollback of commit
     0f0a0d5.
   + da557f9:
     Windows: fix "bazel run" argument quoting
   + ef8b6f6:
     Return JavaInfo from java proto aspects.
   + 209175f:
     Revert back to the old behavior of not creating a proto source
     root for generated .proto files.
   + 644060b:
     Fix PatchUtil for parsing special patch format
   + 067040d:
     Put the removal of the legacy repository-relative proto path
     behind the --incompatible_generated_protos_in_virtual_imports
     flag.
   + 76ed014:
     repository mapping lookup: convert to canonical name first

Important changes:

  - rule_test: fix Bazel 0.27 regression ("tags" attribute was
    ingored, #8723
  - Adds --incompatible_enable_execution_transition, which enables
    incremental migration of host attributes to exec attributes.
  - objc_proto_library rule has been deleted from Bazel.
  - repository_ctx.read is no longer restricted to files
        in the repository contructed.
  - tags 'no-remote', 'no-cache', 'no-remote-cache',
    'no-remote-exec', 'no-sandbox' are propagated now to the actions
    from targets when '--ncompatible_allow_tags_propagation' flag set
    to true. See #8830.
  - Adds flag
    --//tools/build_defs/pkg:incompatible_no_build_defs_pkg. This
    flag turns off the rules //tools/build_defs/pkg:{pkg_deb,
    pkg_rpm, pkg_tar}.
  - The Android NDK is now integrated with toolchains. To use them,
    pass the `--extra_toolchains=@androidndk//:all` flag or register
    them in your WORKSPACE with
    `register_toolchains("@androidndk//:all")`.
  - Stdout and stderr are checked to determine if output is going to a
    terminal. `--is_stderr_atty` is deprecated and `--isatty` is
    undeprecated.
  - --incompatible_load_proto_rules_from_bzl was added to forbid
    loading the native proto rules directly. See more on tracking
    issue #8922
  - Docker Sandbox now respects remote_default_platform_properties
  - pkg_deb, pkg_rpm & pkg_tar deprecation plan announced in the
    documentation.
  - The new java_tools release:
    * fixes #8614
    * exposes a new toolchain `@java_tools//:prebuilt_toolchain`
    which is using all the pre-built tools, including singlejar and
    ijar, even on remote execution. This toolchain should be used
    only when host and execution platform are the same, otherwise the
    binaries will not work on the execution platform.
  - java_common.compile supports specifying
    annotation_processor_additional_inputs and
    annotation_processor_additional_outputs for the Java compilation
    action for supporting annotation processors that consume or
    produce artifacts. Fixes #6415
  - There is now documentation on optimizing Android app build
    performance. Read it at
    https://docs.bazel.build/versions/0.29.0/android-build-performance
    .html
  - Execution log now respects --remote_default_platform_properties
  - Include a link to the relevant documenation on transitive Python
    version errors.
  - New incompatible flag
    --incompatible_disable_target_provider_fields removes the ability
    (in Starlark) to access a target's providers via the field syntax
    (for example, `ctx.attr.dep.my_provider`). The provider-key
    syntax should be used instead (for example,
    `ctx.attr.dep[MyProvider]`). See
    #9014 for details.
  - A new platform exec_properties is added to replace
    remote_execution_properties.
  - Added --incompatible_load_python_rules_from_bzl, which will be
    flipped in Bazel 1.0. See
    #9006.
  - add --break_build_on_parallel_dex2oat_failure to shortcut tests
    on dex2oat errors

This release contains contributions from many people at Google, as well as Alexander Ilyin, Arek Sredzki, Artem Zinnatullin, Benjamin Peterson, Fan Wu, John Millikin, Loo Rong Jie, Marwan Tammam, Oscar Bonilla, Peter Mounce, Sergio Rodriguez Orellana, Takeo Sawada, and Yannic Bonenberger.
buchgr pushed a commit to buchgr/bazel that referenced this issue Aug 30, 2019
@katre
Release 0.29.0 (2019-08-28)
5a9fa49 
Baseline: 6c5ef53

Cherry picks:

   + 338829f:
     Fix retrying of SocketTimeoutExceptions in HttpConnector
   + 14651cd:
     Fallback to next urls if download fails in HttpDownloader
   + b7d300c:
     Fix incorrect stdout/stderr in remote action cache. Fixes bazelbuild#9072
   + 9602176:
     Automated rollback of commit
     0f0a0d5.
   + da557f9:
     Windows: fix "bazel run" argument quoting
   + ef8b6f6:
     Return JavaInfo from java proto aspects.
   + 209175f:
     Revert back to the old behavior of not creating a proto source
     root for generated .proto files.
   + 644060b:
     Fix PatchUtil for parsing special patch format
   + 067040d:
     Put the removal of the legacy repository-relative proto path
     behind the --incompatible_generated_protos_in_virtual_imports
     flag.
   + 76ed014:
     repository mapping lookup: convert to canonical name first

Important changes:

  - rule_test: fix Bazel 0.27 regression ("tags" attribute was
    ingored, bazelbuild#8723
  - Adds --incompatible_enable_execution_transition, which enables
    incremental migration of host attributes to exec attributes.
  - objc_proto_library rule has been deleted from Bazel.
  - repository_ctx.read is no longer restricted to files
        in the repository contructed.
  - tags 'no-remote', 'no-cache', 'no-remote-cache',
    'no-remote-exec', 'no-sandbox' are propagated now to the actions
    from targets when '--ncompatible_allow_tags_propagation' flag set
    to true. See bazelbuild#8830.
  - Adds flag
    --//tools/build_defs/pkg:incompatible_no_build_defs_pkg. This
    flag turns off the rules //tools/build_defs/pkg:{pkg_deb,
    pkg_rpm, pkg_tar}.
  - The Android NDK is now integrated with toolchains. To use them,
    pass the `--extra_toolchains=@androidndk//:all` flag or register
    them in your WORKSPACE with
    `register_toolchains("@androidndk//:all")`.
  - Stdout and stderr are checked to determine if output is going to a
    terminal. `--is_stderr_atty` is deprecated and `--isatty` is
    undeprecated.
  - --incompatible_load_proto_rules_from_bzl was added to forbid
    loading the native proto rules directly. See more on tracking
    issue bazelbuild#8922
  - Docker Sandbox now respects remote_default_platform_properties
  - pkg_deb, pkg_rpm & pkg_tar deprecation plan announced in the
    documentation.
  - The new java_tools release:
    * fixes bazelbuild#8614
    * exposes a new toolchain `@java_tools//:prebuilt_toolchain`
    which is using all the pre-built tools, including singlejar and
    ijar, even on remote execution. This toolchain should be used
    only when host and execution platform are the same, otherwise the
    binaries will not work on the execution platform.
  - java_common.compile supports specifying
    annotation_processor_additional_inputs and
    annotation_processor_additional_outputs for the Java compilation
    action for supporting annotation processors that consume or
    produce artifacts. Fixes bazelbuild#6415
  - There is now documentation on optimizing Android app build
    performance. Read it at
    https://docs.bazel.build/versions/0.29.0/android-build-performance
    .html
  - Execution log now respects --remote_default_platform_properties
  - Include a link to the relevant documenation on transitive Python
    version errors.
  - New incompatible flag
    --incompatible_disable_target_provider_fields removes the ability
    (in Starlark) to access a target's providers via the field syntax
    (for example, `ctx.attr.dep.my_provider`). The provider-key
    syntax should be used instead (for example,
    `ctx.attr.dep[MyProvider]`). See
    bazelbuild#9014 for details.
  - A new platform exec_properties is added to replace
    remote_execution_properties.
  - Added --incompatible_load_python_rules_from_bzl, which will be
    flipped in Bazel 1.0. See
    bazelbuild#9006.
  - add --break_build_on_parallel_dex2oat_failure to shortcut tests
    on dex2oat errors

This release contains contributions from many people at Google, as well as Alexander Ilyin, Arek Sredzki, Artem Zinnatullin, Benjamin Peterson, Fan Wu, John Millikin, Loo Rong Jie, Marwan Tammam, Oscar Bonilla, Peter Mounce, Sergio Rodriguez Orellana, Takeo Sawada, and Yannic Bonenberger.
bazel-io pushed a commit that referenced this issue Sep 10, 2019
@katre
Release 0.29.1 (2019-09-10)
4f9cd08 
Baseline: 6c5ef53

Cherry picks:

   + 338829f:
     Fix retrying of SocketTimeoutExceptions in HttpConnector
   + 14651cd:
     Fallback to next urls if download fails in HttpDownloader
   + b7d300c:
     Fix incorrect stdout/stderr in remote action cache. Fixes #9072
   + 9602176:
     Automated rollback of commit
     0f0a0d5.
   + da557f9:
     Windows: fix "bazel run" argument quoting
   + ef8b6f6:
     Return JavaInfo from java proto aspects.
   + 209175f:
     Revert back to the old behavior of not creating a proto source
     root for generated .proto files.
   + 644060b:
     Fix PatchUtil for parsing special patch format
   + 067040d:
     Put the removal of the legacy repository-relative proto path
     behind the --incompatible_generated_protos_in_virtual_imports
     flag.
   + 76ed014:
     repository mapping lookup: convert to canonical name first
   + f791df0:
     Release 0.29.0 (2019-08-28)
   + 2c04648:
     Fix git_repository rule to support fetching a commit on a tag
   + 9e1d65a:
     Fix a serious regression in remote execution. Fixes #9284
   + 8b0bfaf:
     Include cc configure headers in the cache key
   + 5c02b92:
     Make --workspace_status_command work with "Builds without the
     Bytes".
   + a0e3bb2:
     Remove support for authentication and .netrc

This release contains contributions from many people at Google, as well as Artem Zinnatullin.
exoson pushed a commit to exoson/bazel that referenced this issue Sep 25, 2019
@katre
Release 0.29.1 (2019-09-10)
1b771ec 
Baseline: 6c5ef53

Cherry picks:

   + 338829f:
     Fix retrying of SocketTimeoutExceptions in HttpConnector
   + 14651cd:
     Fallback to next urls if download fails in HttpDownloader
   + b7d300c:
     Fix incorrect stdout/stderr in remote action cache. Fixes bazelbuild#9072
   + 9602176:
     Automated rollback of commit
     0f0a0d5.
   + da557f9:
     Windows: fix "bazel run" argument quoting
   + ef8b6f6:
     Return JavaInfo from java proto aspects.
   + 209175f:
     Revert back to the old behavior of not creating a proto source
     root for generated .proto files.
   + 644060b:
     Fix PatchUtil for parsing special patch format
   + 067040d:
     Put the removal of the legacy repository-relative proto path
     behind the --incompatible_generated_protos_in_virtual_imports
     flag.
   + 76ed014:
     repository mapping lookup: convert to canonical name first
   + f791df0:
     Release 0.29.0 (2019-08-28)
   + 2c04648:
     Fix git_repository rule to support fetching a commit on a tag
   + 9e1d65a:
     Fix a serious regression in remote execution. Fixes bazelbuild#9284
   + 8b0bfaf:
     Include cc configure headers in the cache key
   + 5c02b92:
     Make --workspace_status_command work with "Builds without the
     Bytes".
   + a0e3bb2:
     Remove support for authentication and .netrc

This release contains contributions from many people at Google, as well as Artem Zinnatullin.
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
Automated rollback of commit be26ba030319cf1cda44649d94d7beee4ccc86c4.
574f8d6 
    *** Reason for rollback ***

    It was not the culprit

    *** Original change description ***

    Automated rollback of commit a6b5b0540a0167f84437127d3dd6cef804286d6f.

    *** Reason for rollback ***

    Probably the culprit that broke Windows builds with remote caching: bazelbuild/bazel#9072

    *** Original change description ***

    Streamline uploading of stdout and stderr

    This change removes extra upload logic for stdout/stderr. Besides
    the streamlined code this also saves two round trips uploading an
    action that produced both stdout and s...

    ***

    ROLLBACK_OF=261700869

    RELNOTES: None
    PiperOrigin-RevId: 261864282
luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
Automated rollback of commit a6b5b0540a0167f84437127d3dd6cef804286d6f.
254d5f9 
    *** Reason for rollback ***

    Probably the culprit that broke Windows builds with remote caching: bazelbuild/bazel#9072

    *** Original change description ***

    Streamline uploading of stdout and stderr

    This change removes extra upload logic for stdout/stderr. Besides
    the streamlined code this also saves two round trips uploading an
    action that produced both stdout and stderr.

    Closes #9025.

    PiperOrigin-RevId: 261700869
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
breakage P1 I'll work on this now. (Assignee required) team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@meteorcloudy