-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tracking issue for "Remote Builds without the Bytes" #6862
Comments
Please assign a priority (see the maintainers guide). |
See bazelbuild#6862 for details.
See bazelbuild#6862 for details.
@buchgr Do I understand correctly that |
See bazelbuild#6862 for details.
Also, allow it to respond to interrupts. Progress towards bazelbuild#6862
Some native actions run both remotely and locally. For example, the CppCompileAction runs compilation remotely but does .d file pruning locally. This change adds the necessary infrastructure for an action to tell a Spawn which of its outputs it needs on the local filesytem. Progress towards bazelbuild#6862
Adds a MetadataInjector type that can be accessed via the SpawnExecutionContext. Progress towards bazelbuild#6862
The FilesystemValueChecker is used by Bazel to detect modified outputs before a command. This change teaches it about remote outputs that don't exist in the output base. That is if SkyFrame has metadata about a remote output file which does not exist in the output base it will not invalidate the output. However, if the file exists in the output base it will be taken as the source of truth. Progress towards bazelbuild#6862
@buchgr does this mean the change is only one commit? (github says |
Edit: sorry, disregard, my bad, it does compile with 0.22.0! Which bazel version should we use to compile this with? For me, bazel build //src:bazel on this repository didn't work with any bazel releases in {0.16.1, 0.17.1, 0.19.2, 0.20.0, 0.21.0, 0.22.0}. I tried bootstrapping it, but that didn't work either. Thank you! |
Hey, I was trying this branch on a very simple project today and it seems to fail to fetch binaries in some cases. I have a very simple
where I’m now running the following commands:
The first invocation works completely fine and outputs "Hello World". However, after cleaning the cache this fails and I get the following output:
So the binary was never downloaded. The problem appears to be in |
The commits should be part of 7.0.0-pre.20230628.2 that we released yesterday. |
I encountered issue with the latest pre release where I had |
@purkhusid Not that I'm aware of - do you have a repro? |
I don't have a minimal repro, this is part of a larger repo. Things worked again after I added:
to our bazelrc If it works with toplevel, it should also work with minimal, right? I'll see if I can narrow this down a bit and extract a minimal repro. |
@tjgq I've looked a bit into this and my build breaks as soon as I add I'll see if I can narrow it down a bit more and see in what build it started to fail. |
@purkhusid I find that surprising, because until 24ba4fa (which is not included in the latest pre-release, 7.0.0-pre.20230823.4), Also - are you by any chance running on Windows? (Asking because this might be somehow related to #19333.) Anyway, I've just tried |
Just to be clear I explicitly set:
And am not using This worked in 6.3.2 but not in the latest prerelease. If I remove |
If the file you're trying to access is indeed included in the runfiles for the target you're trying to |
Alrighty, I managed to narrow this down a bit. You actually have to run If I run |
Another update. Maybe I should have looked into it in the beginning but if I use |
Ok, this seems to be an issue in the latest pre release and on last_green. The step I have to reproduce it is still:
This is with |
When I tried this a few years ago I ran into issues where bazel would crash if the remote cache said it had a file but didn't (which could happen due to cache eviction, original file upload failed, or any other reason). I'm wondering if these issues have been resolved -- suppose I created a remote cache called "lying cache" which randomly lies and says it has AC/CAS files that it doesn't. Will bazel gracefully handle this and perform "action rewinding" in all cases? |
@bjacklyn See #8250 (comment). |
Activate option --remote_download_minimal to make remote build even more faster. See this upstream issue for more details: [1]. [1] bazelbuild/bazel#6862 Release-Notes: skip Change-Id: Ibcf1c95b38e2194444f32cec9eab95aa9ad82039
@purkhusid Do you mind giving last_green a try to see if it fixes the issue you were describing above? The recently submitted 84d1a72 improved the incremental correctness of |
@tjgq The problem still exists on |
@tjgq I have managed to do a pretty minimal reproduction: https://github.com/purkhusid/bazel_runfiles_repro I managed to narrow it down to the interaction between these two flags:
I've got |
@purkhusid Thanks for the repro. My initial reaction is that it's a bug (I don't see why those two flags would be incompatible) but we're going to need some more time to figure out where the issue is. |
Let's move the discussion to the separately filed #20843. |
Activate option --remote_download_minimal to make remote build even more faster. See this upstream issue for more details: [1]. [1] bazelbuild/bazel#6862 Release-Notes: skip Change-Id: Ibcf1c95b38e2194444f32cec9eab95aa9ad82039
Activate option --remote_download_minimal to make remote build even more faster. See this upstream issue for more details: [1]. [1] bazelbuild/bazel#6862 Release-Notes: skip Change-Id: Ibcf1c95b38e2194444f32cec9eab95aa9ad82039
Activate option --remote_download_minimal to make remote build even more faster. See this upstream issue for more details: [1]. [1] bazelbuild/bazel#6862 Release-Notes: skip Change-Id: Ibcf1c95b38e2194444f32cec9eab95aa9ad82039
Activate option --remote_download_minimal to make remote build even more faster. See this upstream issue for more details: [1]. [1] bazelbuild/bazel#6862 Release-Notes: skip Change-Id: Ibcf1c95b38e2194444f32cec9eab95aa9ad82039
Activate option --remote_download_minimal to make remote build even more faster. See this upstream issue for more details: [1]. [1] bazelbuild/bazel#6862 Release-Notes: skip Change-Id: Ibcf1c95b38e2194444f32cec9eab95aa9ad82039
Activate option --remote_download_minimal to make remote build even more faster. See this upstream issue for more details: [1]. [1] bazelbuild/bazel#6862 Release-Notes: skip Change-Id: Ibcf1c95b38e2194444f32cec9eab95aa9ad82039
Activate option --remote_download_minimal to make remote build even more faster. See this upstream issue for more details: [1]. [1] bazelbuild/bazel#6862 Release-Notes: skip Change-Id: Ibcf1c95b38e2194444f32cec9eab95aa9ad82039
- move the statement from the service documentation to the call. - add a section about extending TTLs related discussions: - https://groups.google.com/d/msg/remote-execution-apis/zmcEV8kubRc/5PPcHOVZAgAJ - bazelbuild/bazel#6862
This issue tracks the progress towards Remote Builds without the Bytes.
Milestones
The feature can be enabled by adding the flag
--experimental_remote_download_outputs=minimal
to the remote caching / execution invocation. The feature mostly works, but comes with the limitations stated in the subsequent tasks.While Bazel can run most actions remotely there are a few dozen or so action implementations that currently can't. This can be problematic because such actions often require Bazel to download remote files which this change tries to avoid. An example of such an action is the
SymlinkAction
.Implement
--experimental_remote_download_outputs=toplevel
as described in the design document.See --remote_download_toplevel --nozip_undeclared_test_outputs does not download any undeclared test outputs for passing tests #17884 for details
The persistent action cache stores the state of Bazel's output base. It does not know about remotely stored files. We need to extend it to do so. See Store remote metadata in the on disk action cache #8248
See Builds without the Bytes fails on missing AC result #10880 for details
As detailed in the design document it's possible for Bazel to wrongly think that an output file exists on the remote system because it has been evicted ("garbage collected") in the meantime. If a Bazel action then depends on an evicted file there is currently no other way but to fail the build/test. See Remote Builds without the Bytes should support remote output eviction #8250
A user should have insight into what is being up/downloaded and why.
This change currently doesn't interact well with the BEP and BES. See Remote Builds without the Bytes should work with BES #8249
For a test that runs remotely we currently create a runfiles tree in the output base with dangling symlinks that's unused. We need to fix the
SymlinkTreeStrategy
.See --remote_download_toplevel does not download symlink targets of runfiles #18249 for details
See Action input symlink into directory doesn't work with --remote_download_minimal #15678 for details
See FR?: ctx.actions.symlink action time scales w/ size of input; should be constant #14125 for details
See Symlinks in action outputs are not yet supported by --experimental_remote_download_outputs=minimal #13355 for details
See Do not create runfile trees unnecessarily when building without the bytes #18580 for details
See Bazel query silently thrashes the analysis cache #10902 for details
See Build event file creation fails with symlinks when using --remote_download_minimal #11942 for details
See mobile-install fails with BWOB #14523 for details
Related Issues
#1922 #5505
The text was updated successfully, but these errors were encountered: