Provide action integration with Buildbarn Browser

[Qinusty]

This would be incredibly useful and definitely open up the project to interest from users of the Buildbarn project, bb-browser (buildbarn browser) is an incredible utility for viewing information stored in the CAS and AC through a web page and the links to the pages for this can be assembled from a few key components ${bb-browser-url}/action/${instance}/${hash}/${size}/.

I propose adding some configuration options for the browser url and instance (remote_instance_name from bazel), and allowing these actions to be accessed via links from targets. Currently targets cannot be broken down into their various actions but this should be straight forward from the BES data.

Problem The main blocker here is the limitation on the current implementation of BES which doesn't seem to provide ANY route to action digests / action sizes through the BEP.

[siggisim]

The lack of information in the BEP to map information back to the cache is one of my biggest frustrations with the protocol.

Some of this info can be reverse engineered from the bytestream urls for build artifacts - but only for "important outputs", not for all actions associated with the target.

I haven't had a chance to play around with bb-browser. Do you have a screenshot of what information is displayed on that page?

[Qinusty]

I've attached a PDF of the browser page, there's quite a bit of useful information provided,
Buildbarn Browser.pdf

Some of the data is quite specific to a buildbarn execution platform such as the Execution metadata and resource usage.

[ulfjack]

Hi! I'm one of the original designers of the BEP. We were given per-event, per-stream, and company-wide size restrictions for the amount of data that could be reported in the BEP. One of our biggest concerns was that reporting all executed actions would be prohibitively expensive on both the second and third metrics (based on experiments), at least for Google. (Note that this can also affect build latency: uploading a lot of data can take a lot of time.)

The --build_event_publish_all_actions option enables reporting of all actions in the BEP, but it'd caution to closely watch the amount of data generated for this reason.

[Qinusty]

The commit message here buildbarn/bb-browser@6b5437f seems to suggest this issue could see traction following the integration of bazelbuild/remote-apis#186

This makes it possible to see the build tool's invocation ID and version
number on action and uncached action result pages. This feature will
become a lot more interesting once the following API for remote-apis
lands:

bazelbuild/remote-apis#186

Once we have that, we can display Bazel target names, configuration IDs,
etc., meaning we can finally correlate Build Event Stream events with
individual actions in Buildbarn.

[brentleyjones]

Bazel now sends that additional metadata.

[siggisim]

Did some more digging here.

The additional metadata that Bazel now sends is with GRPC requests to the cache / RBE server (Buildbarn in this case), and are still not included in the BEP (which is what BuildBuddy receives). This makes is it difficult / impossible to show this data for the configuration described (using BuildBuddy for BES, but Buildbarn for caching / RBE) without further changes to the BES (since BuildBuddy servers never receive this additional metadata).

In the scenario where you're using BuildBuddy for both BES and caching / RBE - we have much more flexibility. We've made huge improvements since this issue was opened with the addition of the executions tab, which now shows all of the important information from the Buildbarn Browser screenshot including input files, arguments, environment variables, output files, exit code, worker, timeline, stderr/stdout and more!

We'd like to continue to support the BuildBuddy + Buildbarn configuration, and are open to pull requests once the BEP contains enough information to make this possible.

Going to close this issue since we've covered the use-case (viewing action information via web), and because the suggested integration seems to be infeasible given the current state of the BEP.