cquery --output=proto crashes in ProtoOutputFormatterCallback.writeData: IllegalArgumentException: bufferSize must be >= 0

[keithl-stripe]

Description of the bug:

In a large monorepo with 45,000 Bazel packages, 300,000 targets, and 800,000 configured targets, bazel cquery --output=proto crashes because the field of CqueryResult proto is larger than the 2GB inherent limit in the proto wire format.

bazel cquery --output=proto --keep_going 'deps(...)'
INFO: Analysis succeeded for only 325040 of 325813 top-level targets
INFO: Analyzed 325813 targets (55049 packages loaded, 859577 targets configured).
INFO: Found 325040 targets...

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.IllegalArgumentException: bufferSize must be >= 0
        at com.google.protobuf.CodedOutputStream$AbstractBufferedEncoder.<init>(CodedOutputStream.java:2242)
        at com.google.protobuf.CodedOutputStream$OutputStreamEncoder.<init>(CodedOutputStream.java:2712)
        at com.google.protobuf.CodedOutputStream.newInstance(CodedOutputStream.java:105)
        at com.google.protobuf.AbstractMessageLite.writeTo(AbstractMessageLite.java:82)
        at com.google.devtools.build.lib.query2.cquery.ProtoOutputFormatterCallback.writeData(ProtoOutputFormatterCallback.java:109)
        at com.google.devtools.build.lib.query2.cquery.ProtoOutputFormatterCallback.close(ProtoOutputFormatterCallback.java:93)
        at com.google.devtools.build.lib.buildtool.PostAnalysisQueryBuildTool.doPostAnalysisQuery(PostAnalysisQueryBuildTool.java:188)
        at com.google.devtools.build.lib.buildtool.PostAnalysisQueryBuildTool.postProcessAnalysisResult(PostAnalysisQueryBuildTool.java:78)
        at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:227)
        at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:443)
        at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:411)
        at com.google.devtools.build.lib.runtime.commands.CqueryCommand.exec(CqueryCommand.java:152)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:585)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:231)
        at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:550)
        at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:614)
        at io.grpc.Context$1.run(Context.java:579)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

bazel cquery --output=proto --keep_going 'deps(...)'

Which operating system are you running Bazel on?

Linux Ubuntu 20.04.1

What is the output of bazel info release?

5.4.0

Have you found anything relevant by searching the web?

This may be because a field is more than 2GB - see protocolbuffers/protobuf#8075

Any other information, logs, or outputs that you want to share?

A simple fix might be to add a new output format, perhaps --output=proto-stream which is simply a stream of ConfiguredTargets. (Or if possible a stream of ConfiguredTargets and Configurations – I don't know if protobuf makes it easy to parse a mixed stream of delimited protos)

[sgowroji]

Hi @keithl-stripe, Is there a way we can reproduce the error/crash it our end. Thanks!

[keithl-stripe]

Sure, just create a Bazel workspace by creating the following files in an empty directory:

WORKSPACE

# empty workspace

BUILD

[
    genrule(name = "gen%d" % i, cmd = "echo hi", outs=["gen%d.txt" % i]) 
    for i in range(1000 * 1000 * 2)
]

Then type:

bazel --host_jvm_args=-Xmx40G cquery --output=proto :all > all-cquery.proto

You will see:

java.lang.IllegalArgumentException: bufferSize must be >= 0
        at com.google.protobuf.CodedOutputStream$AbstractBufferedEncoder.<init>(CodedOutputStream.java:2186)
        at com.google.protobuf.CodedOutputStream$OutputStreamEncoder.<init>(CodedOutputStream.java:2645)
        at com.google.protobuf.CodedOutputStream.newInstance(CodedOutputStream.java:108)
        at com.google.protobuf.AbstractMessageLite.writeTo(AbstractMessageLite.java:85)
        at com.google.devtools.build.lib.query2.ProtoOutputFormatterCallback.close(ProtoOutputFormatterCallback.java:70)
        at com.google.devtools.build.lib.query2.AbstractBlazeQueryEnvironment.evaluateQuery(AbstractBlazeQueryEnvironment.java:150)
        at com.google.devtools.build.lib.query2.ConfiguredTargetQueryEnvironment.evaluateQuery(ConfiguredTargetQueryEnvironment.java:216)
        at com.google.devtools.build.lib.buildtool.CqueryBuildTool.doConfiguredTargetQuery(CqueryBuildTool.java:152)
        at com.google.devtools.build.lib.buildtool.CqueryBuildTool.postProcessAnalysisResult(CqueryBuildTool.java:66)
        at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:237)
        at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:350)
        at com.google.devtools.build.lib.runtime.commands.CqueryCommand.exec(CqueryCommand.java:121)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:472)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:199)
        at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:855)
        at com.google.devtools.build.lib.server.GrpcServerImpl.access$2100(GrpcServerImpl.java:111)
        at com.google.devtools.build.lib.server.GrpcServerImpl$2.lambda$run$0(GrpcServerImpl.java:924)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
java.lang.IllegalArgumentException: bufferSize must be >= 0
        at com.google.protobuf.CodedOutputStream$AbstractBufferedEncoder.<init>(CodedOutputStream.java:2186)
        at com.google.protobuf.CodedOutputStream$OutputStreamEncoder.<init>(CodedOutputStream.java:2645)
        at com.google.protobuf.CodedOutputStream.newInstance(CodedOutputStream.java:108)
        at com.google.protobuf.AbstractMessageLite.writeTo(AbstractMessageLite.java:85)
        at com.google.devtools.build.lib.query2.ProtoOutputFormatterCallback.close(ProtoOutputFormatterCallback.java:70)
        at com.google.devtools.build.lib.query2.AbstractBlazeQueryEnvironment.evaluateQuery(AbstractBlazeQueryEnvironment.java:150)
        at com.google.devtools.build.lib.query2.ConfiguredTargetQueryEnvironment.evaluateQuery(ConfiguredTargetQueryEnvironment.java:216)
        at com.google.devtools.build.lib.buildtool.CqueryBuildTool.doConfiguredTargetQuery(CqueryBuildTool.java:152)
        at com.google.devtools.build.lib.buildtool.CqueryBuildTool.postProcessAnalysisResult(CqueryBuildTool.java:66)
        at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:237)
        at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:350)
        at com.google.devtools.build.lib.runtime.commands.CqueryCommand.exec(CqueryCommand.java:121)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:472)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:199)
        at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:855)
        at com.google.devtools.build.lib.server.GrpcServerImpl.access$2100(GrpcServerImpl.java:111)
        at com.google.devtools.build.lib.server.GrpcServerImpl$2.lambda$run$0(GrpcServerImpl.java:924)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
FAILED: Build did NOT complete successfully

[katre]

This is pretty unfortunate.

A similar issue was fixed for query, in 68d2181 (by @michajlo). You could do something similar here, streaming the ConfiguredTarget protos out instead of accumulating them. @michajlo's note was "if i could do it again i'd keep track of bytes written and throw when it got too big, for whatever reason i didn't do that on the first go, making it harder to fix later".

I'm marking this as low-priority, we won't be able to pick this up, but if you want to send a PR we'll review it.

[kkress]

The solution to this problem in query is --output=streamed_proto which produces 1 message per Target in a length delimited stream. This means we avoid having to treat the output a single serialized proto and can both process the individual Targets more efficiently and without bumping into inherent limits.

Ideally both aquery and cquery should inherit this behavior and support that output format for their respective primary outputs (Target for cquery and ActionGraphContainer for aquery)

[meisterT]

cc @zhengwei143 who will look into streamed proto for aquery.

[zhengwei143]

The fix has been commited in 607d0f7, which introduces --output=streamed_proto for cquery, containing a length-delimited stream of CqueryResult(s) each wrapping a single ConfiguredTarget or Configuration proto.

[zhengwei143]

We rolled this back due to a bug, will submit a roll forward with a fix soon.

[zhengwei143]

This has been reimplemented in aa88357.