Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upload corrupted downloads to Artifacts in Buildkite #1175

Closed
3 tasks done
coeuvre opened this issue Jun 10, 2021 · 1 comment
Closed
3 tasks done

Upload corrupted downloads to Artifacts in Buildkite #1175

coeuvre opened this issue Jun 10, 2021 · 1 comment
Assignees
@coeuvre

Comments

@coeuvre
Copy link
Member

coeuvre commented Jun 10, 2021 •
edited
Loading

In the past, we've seen a couple of cache poisoning cases when we use remote cache/execution and we usually only get error messages like: java.io.IOException: Output download failed: Expected digest '...' does not match received digest '...'. It would be helpful if we can get the corrupted blob to debug with.

Steps:

  • Add more details describing the error in Bazel. e.g. including file name, using a dedicated exception class.
  • Don't delete (partial) downloaded outputs upon errors.
  • Let Bazel CI catch the exception and upload blobs referenced in the error message to Buildkite.

#1174
@coeuvre coeuvre self-assigned this Jun 10, 2021
@coeuvre coeuvre mentioned this issue Jun 10, 2021
Closed
bazel-io pushed a commit to bazelbuild/bazel that referenced this issue Jun 10, 2021
@coeuvre @copybara-github
Remote: Add --experimental_capture_corrupted_outputs flag.
4ca8946 
Which when set, Bazel will save outputs whose digest does not match the expected value to the target directories.

Also use OutputDigestMismatchException to indicate the error and include output path in the error message. The message for such an error will become e.g.:

```
com.google.devtools.build.lib.remote.common.OutputDigestMismatchException: Output bazel-out/darwin-fastbuild/bin/output.txt download failed: Expected digest '872af2fe77729717832d0a020ae87a93b8b944146a2af6b3490491e1eaf1dc74/29229' does not match received digest '872af2fe77729717832d0a020ae87a93b8b944146a2af6b3490491e1eaf1dc74/29229'.
```

Used to support bazelbuild/continuous-integration#1175.

Closes #13568.

PiperOrigin-RevId: 378624823
@coeuvre
Copy link
Member Author

coeuvre commented Jun 11, 2021

Added capture_corrupted_outputs option with #1179.

@coeuvre coeuvre closed this as completed Jun 11, 2021
coeuvre added a commit to coeuvre/bazel that referenced this issue Jul 15, 2021
@coeuvre
Remote: Add --experimental_capture_corrupted_outputs flag.
a922ff9 
Which when set, Bazel will save outputs whose digest does not match the expected value to the target directories.

Also use OutputDigestMismatchException to indicate the error and include output path in the error message. The message for such an error will become e.g.:

```
com.google.devtools.build.lib.remote.common.OutputDigestMismatchException: Output bazel-out/darwin-fastbuild/bin/output.txt download failed: Expected digest '872af2fe77729717832d0a020ae87a93b8b944146a2af6b3490491e1eaf1dc74/29229' does not match received digest '872af2fe77729717832d0a020ae87a93b8b944146a2af6b3490491e1eaf1dc74/29229'.
```

Used to support bazelbuild/continuous-integration#1175.

Closes bazelbuild#13568.

PiperOrigin-RevId: 378624823
coeuvre added a commit to coeuvre/bazel that referenced this issue Jul 15, 2021
@coeuvre
Remote: Add --experimental_capture_corrupted_outputs flag.
2d81c81 
Which when set, Bazel will save outputs whose digest does not match the expected value to the target directories.

Also use OutputDigestMismatchException to indicate the error and include output path in the error message. The message for such an error will become e.g.:

```
com.google.devtools.build.lib.remote.common.OutputDigestMismatchException: Output bazel-out/darwin-fastbuild/bin/output.txt download failed: Expected digest '872af2fe77729717832d0a020ae87a93b8b944146a2af6b3490491e1eaf1dc74/29229' does not match received digest '872af2fe77729717832d0a020ae87a93b8b944146a2af6b3490491e1eaf1dc74/29229'.
```

Used to support bazelbuild/continuous-integration#1175.

Closes bazelbuild#13568.

PiperOrigin-RevId: 378624823
coeuvre added a commit to coeuvre/bazel that referenced this issue Jul 15, 2021
@coeuvre
Remote: Add --experimental_capture_corrupted_outputs flag.
b054d51 
Which when set, Bazel will save outputs whose digest does not match the expected value to the target directories.

Also use OutputDigestMismatchException to indicate the error and include output path in the error message. The message for such an error will become e.g.:

```
com.google.devtools.build.lib.remote.common.OutputDigestMismatchException: Output bazel-out/darwin-fastbuild/bin/output.txt download failed: Expected digest '872af2fe77729717832d0a020ae87a93b8b944146a2af6b3490491e1eaf1dc74/29229' does not match received digest '872af2fe77729717832d0a020ae87a93b8b944146a2af6b3490491e1eaf1dc74/29229'.
```

Used to support bazelbuild/continuous-integration#1175.

Closes bazelbuild#13568.

PiperOrigin-RevId: 378624823
coeuvre added a commit to coeuvre/bazel that referenced this issue Jul 15, 2021
@coeuvre
Remote: Add --experimental_capture_corrupted_outputs flag.
f4a33b0 
Which when set, Bazel will save outputs whose digest does not match the expected value to the target directories.

Also use OutputDigestMismatchException to indicate the error and include output path in the error message. The message for such an error will become e.g.:

```
com.google.devtools.build.lib.remote.common.OutputDigestMismatchException: Output bazel-out/darwin-fastbuild/bin/output.txt download failed: Expected digest '872af2fe77729717832d0a020ae87a93b8b944146a2af6b3490491e1eaf1dc74/29229' does not match received digest '872af2fe77729717832d0a020ae87a93b8b944146a2af6b3490491e1eaf1dc74/29229'.
```

Used to support bazelbuild/continuous-integration#1175.

Closes bazelbuild#13568.

PiperOrigin-RevId: 378624823
coeuvre added a commit to coeuvre/bazel that referenced this issue Jul 16, 2021
@coeuvre
Remote: Add --experimental_capture_corrupted_outputs flag.
00efa84 
Which when set, Bazel will save outputs whose digest does not match the expected value to the target directories.

Also use OutputDigestMismatchException to indicate the error and include output path in the error message. The message for such an error will become e.g.:

```
com.google.devtools.build.lib.remote.common.OutputDigestMismatchException: Output bazel-out/darwin-fastbuild/bin/output.txt download failed: Expected digest '872af2fe77729717832d0a020ae87a93b8b944146a2af6b3490491e1eaf1dc74/29229' does not match received digest '872af2fe77729717832d0a020ae87a93b8b944146a2af6b3490491e1eaf1dc74/29229'.
```

Used to support bazelbuild/continuous-integration#1175.

Closes bazelbuild#13568.

PiperOrigin-RevId: 378624823
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@coeuvre coeuvre

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@coeuvre