Use remote.buildbuddy.io for backend and cache GRPCS URLs

[avdv]

CI tests have been failing (at least for PRs) with:

WARNING: BES was not properly closed
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.util.concurrent.RejectedExecutionException: Task com.google.common.util.concurrent.TrustedListenableFutureTask@71ed4a73[status=PENDING, info=[task=[running=[NOT STARTED YET], com.google.devtools.build.lib.remote.ByteStreamBuildEventArtifactUploader$$Lambda$703/0x0000000100797440@6a5c7015]]] rejected from java.util.concurrent.ThreadPoolExecutor@56302f63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
	at com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:586)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:66)
	at com.google.devtools.build.lib.remote.ByteStreamBuildEventArtifactUploader.upload(ByteStreamBuildEventArtifactUploader.java:220)
	at com.google.devtools.build.lib.buildeventstream.BuildEventArtifactUploader.uploadReferencedLocalFiles(BuildEventArtifactUploader.java:100)
	at com.google.devtools.build.lib.buildeventservice.BuildEventServiceUploader.enqueueEvent(BuildEventServiceUploader.java:196)
	at com.google.devtools.build.lib.buildeventservice.BuildEventServiceTransport.sendBuildEvent(BuildEventServiceTransport.java:95)
	at com.google.devtools.build.lib.runtime.BuildEventStreamer.post(BuildEventStreamer.java:268)
	at com.google.devtools.build.lib.runtime.BuildEventStreamer.buildEvent(BuildEventStreamer.java:472)
	at com.google.devtools.build.lib.runtime.BuildEventStreamer.buildEvent(BuildEventStreamer.java:481)
	at com.google.devtools.build.lib.runtime.BuildEventStreamer.clearPendingEvents(BuildEventStreamer.java:307)
	at com.google.devtools.build.lib.runtime.BuildEventStreamer.clearEventsAndPostFinalProgress(BuildEventStreamer.java:634)
	at com.google.devtools.build.lib.runtime.BuildEventStreamer.close(BuildEventStreamer.java:354)
	at com.google.devtools.build.lib.runtime.BuildEventStreamer.closeOnAbort(BuildEventStreamer.java:336)
	at com.google.devtools.build.lib.buildeventservice.BuildEventServiceModule.forceShutdownBuildEventStreamer(BuildEventServiceModule.java:409)
	at com.google.devtools.build.lib.buildeventservice.BuildEventServiceModule.afterCommand(BuildEventServiceModule.java:578)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.afterCommand(BlazeRuntime.java:626)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:603)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:231)
	at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:543)
	at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:606)
	at io.grpc.Context$1.run(Context.java:579)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Error: Process completed with exit code 37.

Related: buildbuddy-io/buildbuddy#992

Change the GRPCS URLs to the ones Buildbuddy generates in the UI now - I think the cloud.buildbuddy.io ones are old and the newer ones are distributed globally by some sort of CDN.

[aherrmann]

Good catch, thank you!

I guess rules_haskell and rules_sh will have the same issue. @avdv want to fix those, too?

[avdv]

Good catch, thank you!

I guess rules_haskell and rules_sh will have the same issue. @avdv want to fix those, too?

Seems it was a bit early to declare victory... The tests in PR #231 just failed again :-(

Furthermore, I haven't seen this problem in rules_haskell or rules_sh yet...

[avdv]

Hey @siggisim, could you chime in and help us out? We are using Bazel 4.2.1 currently, and --config=ci is setting in .bazelrc:

rules_nixpkgs/core/.bazelrc

Lines 21 to 28 in 178ad0e

	# Use a remote cache during CI
	build:ci --bes_results_url=https://app.buildbuddy.io/invocation/
	build:ci --bes_backend=grpcs://remote.buildbuddy.io
	build:ci --remote_cache=grpcs://remote.buildbuddy.io
	build:ci --remote_timeout=3600
	# Avoid failures of the form `deadline exceeded after 14999958197ns DEADLINE_EXCEEDED`.
	# See https://github.com/tweag/rules_haskell/issues/1498.
	build:ci --keep_backend_build_event_connections_alive=false

Seems like the invocation URL that is printed when running the tests is never valid if we see this error, e.g. https://github.com/tweag/rules_nixpkgs/runs/6657544824?check_suite_focus=true#step:5:336

INFO: Streaming build results to: https://app.buildbuddy.io/invocation/e6321e48-7bea-4cef-b268-7974c9e71450

and also rules_nixpkgs does not even appear under "repos" in the Buildbuddy UI, but rules_haskell and rules_sh is there.

TIA

[siggisim]

Hey folks - definitely strongly recommend the newer remote.buildbuddy.io endpoint which are geographically distributed, but I don't think that's what's causing this stack trace that you're seeing.

Based on buildbuddy-io/buildbuddy#992 it seems like this issue may have been fixed in Bazel 5.0 by this change: bazelbuild/bazel@e855a26

If it's not fixed by that upgrade, then then capturing a --experimental_remote_grpc_log would be super helpful.

[avdv]

Thank you @siggisim !

definitely strongly recommend the newer remote.buildbuddy.io endpoint which are geographically distributed, but I don't think that's what's causing this stack trace that you're seeing.
Yeah, it was kind of a long shot to see whether it would fix the problem. And it just worked the first time...

If it's not fixed by that upgrade, then then capturing a --experimental_remote_grpc_log would be super helpful.

OK, we will upgrade to Bazel 5 in the near future. Until then probably just disable buildbuddy usage in this repo...