Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bazel coverage false negative #14807

Closed

Conversation

keith
Copy link
Member

@keith keith commented Feb 13, 2022

Previously this short circuit meant the tests weren't actually run, and
they would always exit 0, in the case the test rule didn't set the lcov
related attributes.

More context: #13978

Previously this short circuit meant the tests weren't actually run, and
they would always exit 0, in the case the test rule didn't set the lcov
related attributes.

More context: bazelbuild#13978
@keith
Copy link
Member Author

keith commented Feb 14, 2022

Maybe it would be safer to no-op LCOV_MERGER instead in case any of the test setup below this short circuit is expected even without this attribute

@c-mita
Copy link
Member

c-mita commented Feb 14, 2022

Maybe it would be safer to no-op LCOV_MERGER instead in case any of the test setup below this short circuit is expected even without this attribute

Part of me agrees; I didn't really like the early exit in the first place and this makes the collect_coverage script more complicated.
The downside is that a "null" LCOV_MERGER command still has to output the parameters and take all the parameters.

I think, for now, this best preserves the "old" behaviour, where effectively no coverage information is collected and none of the coverage outputs are generated (modulo the empty $COVERAGE_OUTPUT_FILE).

@keith
Copy link
Member Author

keith commented Feb 14, 2022

Missing variables like COVERAGE=1 worries me a bit, since we don't know what folks do with that. I think setting it to true which is what most folks (including ourselves) do here would be a fine no-op.

But looking at the script a bit more, even though you would have missed some variable setup I think in the past this would have just failed when LCOV_MERGER wasn't set, so maybe at least this is a bit better?

@c-mita
Copy link
Member

c-mita commented Feb 14, 2022

In the past this script wouldn't be run, it then started being run without the full setup. From my perspective, the early exit without full setup in this case isn't too bad for the time being, but this isn't a state of affairs I would be happy with for ever.

@keith
Copy link
Member Author

keith commented Feb 14, 2022

Sg, should we merge and cherry pick for 5.1?

@bazel-io bazel-io closed this in 16de035 Feb 15, 2022
@brentleyjones
Copy link
Contributor

@bazel-io fork 5.1

brentleyjones pushed a commit to brentleyjones/bazel that referenced this pull request Feb 16, 2022
Previously this short circuit meant the tests weren't actually run, and
they would always exit 0, in the case the test rule didn't set the lcov
related attributes.

More context: bazelbuild#13978

Closes bazelbuild#14807.

RELNOTES: Fixed an issue where Bazel could erroneously report a test passes in coverage mode without actually running the test.
PiperOrigin-RevId: 428756211
(cherry picked from commit 16de035)
Wyverald pushed a commit that referenced this pull request Feb 16, 2022
Previously this short circuit meant the tests weren't actually run, and
they would always exit 0, in the case the test rule didn't set the lcov
related attributes.

More context: #13978

Closes #14807.

RELNOTES: Fixed an issue where Bazel could erroneously report a test passes in coverage mode without actually running the test.
PiperOrigin-RevId: 428756211
(cherry picked from commit 16de035)

Co-authored-by: Keith Smiley <keithbsmiley@gmail.com>
@keith keith deleted the ks/fix-bazel-coverage-false-negative branch February 28, 2022 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants