Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore all #[test]-related code automatically #123

Open
Swatinem opened this issue Jan 7, 2022 · 14 comments
Open

Ignore all #[test]-related code automatically #123

Swatinem opened this issue Jan 7, 2022 · 14 comments
Labels
C-enhancement Category: A new feature or an improvement for an existing one S-blocked Status: Blocked on something else

Comments

@Swatinem
Copy link
Contributor

Swatinem commented Jan 7, 2022

UPDATE3(taiki-e):

If you want to ignore all #[test]-related code, you can use module-level #[coverage(off)] attribute:

#[cfg(test)]
#[cfg_attr(coverage, coverage(off))]
mod tests {
    // ...
}

cargo-llvm-cov excludes code contained in the directory named tests from the report by default, so you can also use it instead of #[coverage(off)] attribute.

#[coverage(off)] attribute has been stabilized in rust-lang/rust#130766 (will be included in Rust 1.85).


UPDATE2(taiki-e):

See #123 (comment)


UPDATE(taiki-e):

If you want to ignore all #[test]-related code, consider using coverage-helper crate version 0.2+.

cargo-llvm-cov excludes code contained in the directory named tests from the report by default, so you can also use it instead of coverage-helper crate.

https://github.com/taiki-e/cargo-llvm-cov#exclude-function-from-coverage


AFAIK this is not currently possible (though I would love to be proven wrong), so just filing this as a nice-to-have tracking issue.

What I would like is that only library code counts towards coverage. Having all of the #[test] and #[cfg(test)] code (fn and whole utility modules) count towards coverage metrics right now skews those numbers up quite a bit.

Right now cargo-llvm-cov automatically ignores all the examples and tests code via -ignore-filename-regex but unit test functions are still counted (and possibly doctests once those are being fixed [tentative fix in #122]).

@taiki-e taiki-e added C-enhancement Category: A new feature or an improvement for an existing one S-blocked Status: Blocked on something else labels Jan 7, 2022
@taiki-e
Copy link
Owner

taiki-e commented Mar 5, 2022

The idea that probably works here is a helper attribute that generates #[cfg_attr(coverage, no_coverage)].

#[cfg(coverage)]
use coverage_helper::test;

#[test]
fn test() {
}

generated:

#[cfg(coverage)]
use coverage_helper::test;

#[cfg_attr(coverage, no_coverage)]
#[::core::prelude::v1::test]
fn test() {
}

UPDATE: this idea is now implemented as https://github.com/taiki-e/coverage-helper.

@scouten-adobe
Copy link

@taiki-e I'm not an expert on attribute macros. Would you (or anyone) be kind enough to translate this into code that we could implement?

Adding the no_coverage attribute in front of every test seems unsustainable.

@taiki-e
Copy link
Owner

taiki-e commented May 29, 2022

bors bot added a commit that referenced this issue May 29, 2022
171: Add test for coverage-helper r=taiki-e a=taiki-e

https://github.com/taiki-e/coverage-helper

cc #123

Co-authored-by: Taiki Endo <te316e89@gmail.com>
@taiki-e taiki-e added S-blocked Status: Blocked on something else and removed S-blocked Status: Blocked on something else labels May 29, 2022
@scouten-adobe
Copy link

I appreciate the coverage-helper crate, but since the #[no_coverage] attribute is not yet stabilized, I've decided not to use this approach in my project.

For those following along, if you want to exclude test-related code from coverage and base your CI builds on stable Rust versions, there's a more drastic approach that does seem to solve the problem. See adobe/xmp-toolkit-rs#66.

@mati865

This comment was marked as off-topic.

@taiki-e

This comment was marked as off-topic.

@Zalathar

This comment was marked as off-topic.

@taiki-e
Copy link
Owner

taiki-e commented Jul 15, 2024

I'm surprised myself that I (probably) haven't documented this EDIT: #123 (comment), but a way that doesn't depend on the unstable features that have worked since the earliest versions is to put unit tests in a directory named tests.

My projects over the past few years have adopted a directory structure that obviously has this in mind.

@taiki-e
Copy link
Owner

taiki-e commented Jul 18, 2024

I'm surprised myself that I (probably) haven't documented this

Nah, It's already documented 10 months ago.

cargo-llvm-cov excludes code contained in the directory named tests from the report by default, so you can also use it instead of coverage-helper crate.

@wyfo
Copy link

wyfo commented Jul 23, 2024

Would it make sense to exclude tests.rs files also? If the test module is small, user may prefer having src/tests.rs instead of src/tests/mod.rs.

@taiki-e
Copy link
Owner

taiki-e commented Sep 1, 2024

exclude tests.rs files

Seems reasonable to me.

@taiki-e
Copy link
Owner

taiki-e commented Oct 19, 2024

In the recent rustc, it seems the module-level coverage(off) is supported:

#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))] // <---
mod tests {
   // ...
}

(Perhaps coverage-helper library should be deprecated for most cases in favor of the module-level coverage(off) ...)

@Zalathar
Copy link

Oh yes, that was rust-lang/rust#126721.

@taiki-e
Copy link
Owner

taiki-e commented Dec 18, 2024

#[coverage(off)] attribute has been stabilized in rust-lang/rust#130766 (will be included in Rust 1.85).

And I've updated documentation to recommend the pattern described in #123 (comment):

If you want to ignore all #[test]-related code, you can use module-level #[coverage(off)] attribute:

#[cfg(test)]
#[cfg_attr(coverage, coverage(off))]
mod tests {
    // ...
}

cargo-llvm-cov excludes code contained in the directory named tests from the report by default, so you can also use it instead of #[coverage(off)] attribute.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-enhancement Category: A new feature or an improvement for an existing one S-blocked Status: Blocked on something else
Projects
None yet
Development

No branches or pull requests

6 participants