Doctests aren't taken into account #13
Comments
|
Doctests are on the roadmap, mildly difficult as when you run the doctest what cargo does is create a new folder and auto generate a doctest executable. Then once the doc test is finished it deletes the directory. Now I need to after generating that executable instrument it and then collect coverage, so rustc is working counter to my aims in that respect. I can't really give a firm date, I need to inspect the cargo and rustc source (and libs derived from rustc modules) to see if any offer a good solution. |
|
This would be great, in some libraries most of the code is covered by doc tests. Having to duplicate those just to get coverage information is a pain :( |
|
I've had a look again, librustdoc isn't available as a separate crate as of yet. I may have to create and publish the fork myself. This one will be a bit more of a slow burner though. |
|
I was able to run kcov on doctests in the past. My quick and dirty solution was to use Here is the code that I used to create a .so file: #[macro_use]
extern crate redhook;
extern crate libc;
use libc::c_char;
use std::ffi::CString;
use std::slice;
use std::env;
hook! {
unsafe fn execvp(filename: *const c_char, argv: *const *const c_char) => my_execvp {
let file = slice::from_raw_parts(filename as *const u8, libc::strlen(filename));
if let Ok(run) = env::var("TEST_WRAPPER") {
if let Ok(cmd) = run.split_whitespace()
.map(|s| CString::new(s))
.collect::<Result<Vec<_>, _>>() {
if !cmd.is_empty() &&
file.starts_with(b"/tmp/rustdoctest") &&
file.ends_with(b"rust_out")
{
let mut args: Vec<_> = cmd.iter().map(|s| s.as_ptr()).collect();
let mut x = argv;
while !(*x).is_null() {
args.push(*x);
x = x.offset(1);
}
args.push(*x);
real!(execvp)(cmd[0].as_ptr(), args.as_ptr());
return;
}
}
}
real!(execvp)(filename, argv);
}
}To run doctess with kcov, execute Although this is not a beautiful solution, it works. @xd009642 Maybe you can integrate this in tarpaulin. |
|
Hey, I was thinking if this is a thing where we could raise an issue against rustc or Cargo to add a flag where the doc-tests are not removed automatically after test runs? I've seen this same issue in a few separate places and the main issue seems to be the core test framework removing the doc-tests directory before anyone can act upon the test results. |
|
Yeah that sounds like a good shout. I did have a look at this in the past and I think the code that deletes the doctests after run is in rustc. In fact looking into the cargo source I can't see any sign in cargo that it's done there. |
|
So rustc should be our target for making a plea. I wonder how this would be received there. EDIT: oh I found and issue, and you (@xd009642 ) have already chimed in on it. rust-lang/rust#37048 |
|
Ah forgot about that! Such a long time ago I feel this won't be added to the compiler in the near time unless someone does a PR or there's some more requests |
|
Currently working to try to get a PR to the main Rust repo to make this possible, I'll try to keep things updated here |
|
PR is open, hopefully this should be pulled in soon :) unfortunately will require nightly to actually generate the doctest execs once its merged in, but its progress! |
|
Awesome thank you! Please keep us updated! |
|
PR is getting very close to being merged! 👍 |
…eavus rustdoc: Add option to persist doc test executables Fixes #37048. This is the initial version of the code so the doctest executables can be used for stuff like code coverage (specifically xd009642/tarpaulin#13) the folders it goes into were just a first idea, so any better ones are welcome. Right now it creates a directory structure like: ``` given_path/ |_____ <filename>_rs_<linenum>/ |_____ ... |_____ <filename>_rs_<linenum>/ |_____ rust_out ``` I couldn't figure out where it actually outputs the file w/ the name, I suspect its somewhere deeper in the compiler. It also adds the unstable `--persist-doctests` flag to `rustdoc` that enables this behavior.
|
PR was merged 👍 |
|
yep! the pull request actually is laying out a bigger foundation for more doctest improvements, e.g. the plan is to eventually have doctests be a single executable like regular tests, but there's a fair amount of work to do on that stuff, however this should hopefully allow tarpaulin to start taking them into account! |
|
This would be awesome! Let's hope the nightly compiles today ;) |
|
It's finally here everyone! I've implemented optional coverage for doc-tests and tarpaulin now has a new My only bug-bear is that As always, changes have been merged into develop so feel free to try it there or wait until the next release (should be this week). |
|
Doctests seems not be supported on workspaces. Is this a bug or a limitation by rustc? |
|
It might be a rustc limitation. Can you link me an example I can test with? |
|
Sure: # New project
cargo new tarpaulin-test
cd tarpaulin-test/
# Create inner crate
cargo new inner --lib
echo -e "inner = { path = \"inner\" }\n\n[workspace]\nmembers = [\"inner\"]" >> Cargo.toml
# fill inner/src/lib.rs
echo -e '/// ```\n/// inner::test_func()\n/// ```\npub fn test_func() {\n println!("hello world!");\n}' > inner/src/lib.rs
### Test with `-p inner`
# [ERROR tarpaulin] Failed to report coverage! Error: No coverage results collected.
cargo tarpaulin --run-types Doctests -p inner
# Works
cargo tarpaulin --run-types Tests -p inner
# 0% coverage
cargo tarpaulin --run-types Doctests Tests -p inner
### Test with `--all`
# [ERROR tarpaulin] Failed to report coverage! Error: No coverage results collected.
cargo tarpaulin --run-types Doctests --all
# Works
cargo tarpaulin --run-types Tests --all
# 0% coverage
cargo tarpaulin --run-types Doctests Tests --all
### Test with default members:
# Add default member:
echo 'default-members = ["inner"]' >> Cargo.toml
# [ERROR tarpaulin] Failed to report coverage! Error: No coverage results collected.
cargo tarpaulin --run-types Doctests
# Works (expected)
cargo tarpaulin --run-types Tests
# 0% coverage
cargo tarpaulin --run-types Doctests Tests
### Test from inner directory works fine:
# 100% coverage
cargo tarpaulin --run-types Doctests
# 0% coverage (expected)
cargo tarpaulin --run-types Tests
# 100% coverage
cargo tarpaulin --run-types Doctests Tests |
|
I've got it, it's down to how I've implemented it. I need to crawl packages for the doctest folders as well! Should be a simple fix just seeing if there are any niceties in the cargo API to make it easier |
|
Fixed and pushed to develop! Good catch 👍 |
|
Cool, thanks!
…On Mar 11, 2019, 22:19, at 22:19, xd009642 ***@***.***> wrote:
Fixed and pushed to develop! Good catch 👍
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#13 (comment)
|
|
I have the same problem as TimDiekmann, but without a workspace. I have identical results with the nightly from 2019-02-21 and 2019-03-10. Version of tarpaulin: https://github.com/xd009642/tarpaulin?branch=develop#3bae9fcb I tested it on misc_utils and serde_with. For misc_utils I get the error Results for misc_utilsResults for serde_with |
|
Hmmm, with |
|
@xd009642 Thanks so much for looking into it. For serde_with I use tarpaulin during CI on the regular tests and it works splendidly there. I only have external tests (in the tests-folder) and there work well. So the 0% coverage seem to be related to the doctest nature. |
|
Interestingly enough it looks like for some of the normal tests in serde_with it can't find instrumentation points and does nothing and for the rest of the tests it finds them. I'd hazard a guess that as a result there may be some misses in coverage and for whatever reason the normal tests don't pick up any lines might be the same issue the doc tests experience... |
|
@jonasbb I'm going to open another issue for |
|
@xd009642 Hmm, I think I'm missing something... https://github.com/jonhoo/hashbag has only doctests, and I believe they should give pretty much full coverage, yet reported coverage is 0%? |
|
@jonhoo hmm I'll have to take a look at that. Hopefully I'll have some time this weekend. I'll tag you in another doctest issue on the off chance there could be some relevance (that one has coverage on weird lines) |
Ideally this would not be necessary, but gettin Tarpaulin to compute coverage for doctests (xd009642/tarpaulin#13) is supported, but trickier than it first appears. For me, the following command appears to hang during the compilation of a transitively-required crate: cargo tarpaulin --verbose --run-types Tests,Doctests so as a workaround to institute coverage, the doctests are duplicated as explicit tests.
I saw you had doc tests on the roadmap which is something I could really use in combine as a they should handle most of the coverage (https://coveralls.io/builds/12513027/source?filename=src%2Fbyte.rs#L47).
Opening this issue to inquire/track when this would be implemented.
The text was updated successfully, but these errors were encountered: