Using --experimental_split_coverage_postprocessing causes a RuntimeException

[finn-ball]

Description of the problem:

With Bazel 4.0, using the --experimental_split_coverage_postprocessing flag causes errors and hard crashes. This probably shouldn't happen.

Experimenting with bazel coverage on the abseil-cpp project.

Using both flags without remote execution:

bazel coverage //absl/flags:config_test --experimental_split_coverage_postprocessing --experimental_fetch_all_coverage_outputs

ERROR: /<redacted>/abseil-cpp/absl/flags/BUILD.bazel:336:8: Testing //absl/flags:config_test failed: I/O exception during sandboxed execution:  /<redacted>/.cache/bazel/_ /<redacted>/execroot/com_google_absl/bazel-out/k8-fastbuild/testlogs/absl/flags/config_test/_coverage (File exists)
Target //absl/flags:config_test up-to-date:
  bazel-bin/absl/flags/config_test
INFO: Elapsed time: 5.745s, Critical Path: 3.71s
INFO: 153 processes: 54 internal, 98 linux-sandbox, 1 worker.
FAILED: Build did NOT complete successfully
//absl/flags:config_test                                              NO STATUS

FAILED: Build did NOT complete successfully

Using just one flag causes a stack-trace regardless of if you use this with remote execution or not:

bazel coverage //:test_hello_world --experimental_split_coverage_postprocessing 

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'UnshareableActionLookupData{actionLookupKey=ConfiguredTargetKey{label=//absl/flags:config_test, config=BuildConfigurationValue.Key[2a87b5057419310409dbaa77df700ae2acbe8a53d5e2f613593e99ea39e39754]}, actionIndex=11}' (requested by nodes 'TestCompletionKey{configuredTargetKey=ConfiguredTargetKey{label=//absl/flags:config_test, config=BuildConfigurationValue.Key[2a87b5057419310409dbaa77df700ae2acbe8a53d5e2f613593e99ea39e39754]}, topLevelArtifactContext=com.google.devtools.build.lib.analysis.TopLevelArtifactContext@524aae23, exclusiveTesting=false}')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:563)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:398)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException
        at com.google.devtools.build.lib.skyframe.ActionMetadataHandler.getMetadata(ActionMetadataHandler.java:233)
        at com.google.devtools.build.lib.exec.StandaloneTestStrategy$BazelTestAttemptContinuation.execute(StandaloneTestStrategy.java:693)
        at com.google.devtools.build.lib.analysis.test.TestRunnerAction$RunAttemptsContinuation.execute(TestRunnerAction.java:1131)
        at com.google.devtools.build.lib.analysis.test.TestRunnerAction.execute(TestRunnerAction.java:907)
        at com.google.devtools.build.lib.analysis.test.TestRunnerAction.execute(TestRunnerAction.java:896)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$5.execute(SkyframeActionExecutor.java:855)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:1016)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:975)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:129)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:81)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:472)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:834)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:307)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:477)
        ... 4 more

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Run similar test commands locally, on your own RE infrastructure or follow the commands here:
#4685 (comment)

What operating system are you running Bazel on?

Linux 5.10.16-arch1-1

What's the output of bazel info release?

release 4.0.0

Any other information, logs, or outputs that you want to share?

Perhaps somewhat related. Also contains an example in one of the recent comments you can follow to reproduce these errors.
#4685

Experimenting with bazel coverage on the abseil-cpp project:

bazel coverage --config=remote-execution //absl/flags:config_test  --experimental_split_coverage_postprocessing --experimental_fetch_all_coverage_outputs

This currently returns empty coverage.dat files. Run without RE, the files contain data. May or may not be related.

[ulfjack]

This also happens with 4.1.0 and java_test so it's not specific to cc_test.

[rainingmaster]

@oquenchil I also meet this problem in bazel 4.2.1, this is my simple test

when I run bazel coverage --test_output=all --cache_test_results=no --experimental_split_coverage_postprocessing //test:hello-test will get these error:

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'UnshareableActionLookupData{actionLookupKey=ConfiguredTargetKey{label=//test:hello-test, config=BuildConfigurationValue.Key[c4b98c948f1b119c240186091080d4e29835d92667e65833a5d582a71c83a141]}, actionIndex=12}' (requested by nodes 'TestCompletionKey{configuredTargetKey=ConfiguredTargetKey{label=//test:hello-test, config=BuildConfigurationValue.Key[c4b98c948f1b119c240186091080d4e29835d92667e65833a5d582a71c83a141]}, topLevelArtifactContext=com.google.devtools.build.lib.analysis.TopLevelArtifactContext@524aae23, exclusiveTesting=false}')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:563)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:398)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException
        at com.google.devtools.build.lib.skyframe.ActionMetadataHandler.getMetadata(ActionMetadataHandler.java:233)
        at com.google.devtools.build.lib.exec.StandaloneTestStrategy$BazelTestAttemptContinuation.execute(StandaloneTestStrategy.java:814)
        at com.google.devtools.build.lib.analysis.test.TestRunnerAction$RunAttemptsContinuation.execute(TestRunnerAction.java:1131)
        at com.google.devtools.build.lib.analysis.test.TestRunnerAction.execute(TestRunnerAction.java:907)
        at com.google.devtools.build.lib.analysis.test.TestRunnerAction.execute(TestRunnerAction.java:896)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$5.execute(SkyframeActionExecutor.java:855)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:1016)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:975)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:129)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:81)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:472)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:834)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:307)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:477)
        ... 4 more

but run coverage --test_output=all --cache_test_results=no //test:hello-test work fine.

[blindpirate]

According to a445cda , --experimental_split_coverage_postprocessing seems to depend on --experimental_fetch_all_coverage_outputs, it works for me if I enable the two flags together.

The error message should be more explicit, though.

[voxeljorge]

According to a445cda , --experimental_split_coverage_postprocessing seems to depend on --experimental_fetch_all_coverage_outputs, it works for me if I enable the two flags together.

The error message should be more explicit, though.

I just want to mention in case anyone else sees this that I've been encountering this problem with both of these flags enabled on bazel 5.2.3. Unfortunately I'm not able to try and contribute a minimal repro case, but if anyone else sees this they should chime in so that this issue could maybe be re-opened if it isn't actually resolved.