Coverage reporting missing for binaries under-test

[epage]

@mssun

Another issue about testing CLIs is to correctly calculate code coverage. If using std::process::Command to fork and execute a CLI, all existing code coverage tools (tarpaulin, gcov, kcov, etc) will failed to calculate the statistics. There is not a better way to properly calculate the line coverage including unit tests and integration tests.

From https://github.com/rust-lang-nursery/cli-wg/issues/9#issuecomment-385487756

[epage]

See also the discussion at assert-rs/assert_cli#104

[glehmann]

I'm exactly in this situation right now, and unfortunately, I have no idea how to fix it.
I've tried kcov, gcov and tarpaulin without success so far. I'm afraid I may have to go back to testing my cli tool directly by calling directly the main function from my crate.
I really like the idea to test the final executable, in order to make sure to not miss anything, and assert_cmd makes is quite pleasant to do, so this is a bit disappointing!

[glehmann]

I've been able to get kcov to report something by calling kcov instead of calling directly my program.

https://github.com/glehmann/hld/blob/f40f3b51f84969cc13714f81451ad17f33fbf2dc/tests/common/mod.rs#L79

I activate it with the kcov feature in my project. kcov doesn't seem to be able to run in parallel, so I force the test to run on a single thread:

cargo test --features kcov -- --test-threads 1

It would be nice to have something similar natively, for example with an environment variable that allow the developer to set a wrapper — in that case, kcov. Something like

RUST_ASSERT_CMD_WRAPPER="kcov --include-pattern=/src --exclude-pattern=/.cargo $PWD/cov" cargo test  -- --test-threads 1

[epage]

A wrapper sounds like a great way to resolve this since the coverage tools don't follow fork (granted, when running cargo would we want them to)

[epage]

I'm assuming I'll need to use shlex to separate out the command from the args when calling Command::new

[maxcountryman]

I’ll add that this would be a very nice feature. I’m currently using assert_cmd and tarpaulin and just like others have noted that this combination doesn’t work.

[brandonkal]

grcov (and kcov) appears to work with assert_cmd. Other tools failed to mark lines as tested as others have mentioned here.

[pkgw]

My project doesn't use assert_cmd, but based on my investigations of kcov I am pretty sure that special support will need to be added to properly track coverage in subcommand executions. You can check out tectonic-typesetting/tectonic#638 to see my approach:

• Gather the tests that do assert_cmd-type activities into one harness (called executable in my project)
• Run the test suite inside kcov, but not doing anything special when launching subcommands (I use cargo kcov which currently requires some hacking, but that's a separate issue)
• For the executable harness, rerun it outside of kcov, but with a special environment variable that causes it to launch the underlying tool inside kcov; it puts the coverage data into a mkdtemp-type directory name so that runs don't stop on each other's toes
• Merge all of the coverage results together at the end with kcov --merge

[frosklis]

Is there any plan on having this solved? Like @maxcountryman , I've seen that it doesn't work with tarpaulin coverage reports.

[epage]

Sorry for the delay (Texas storm and life).

I am not actively working on this but would welcome contributions from others.

[milesj]

@brandonkal I've been unable to get grcov to recognize code coverage from assert_cmd. Are you positive that it did work? Was there something special you did?

[jwodder]

I am currently able to get both cargo-llvm-cov and grcov to recognize coverage of crate binaries tested with assert_cmd (though it doesn't work for extra-crate binaries built via test-binary that use the crate under test). For cargo-llvm-cov, I simply ran cargo llvm-cov --all-features --lcov --output-path lcov.info, and for grcov, I ran:

export RUSTFLAGS="${RUSTFLAGS:+$RUSTFLAGS }-Cinstrument-coverage"
cargo build
export LLVM_PROFILE_FILE="target/coverage/prof/%p-%m.profraw"
cargo test --all-features

grcov \
    --source-dir . \
    --binary-path target/debug \
    --branch \
    --excl-start 'mod tests \{' \
    --ignore 'tests/*' \
    -t html \
    -o target/coverage/html \
    target/coverage/prof

An example repository where you can see coverage working: https://github.com/jwodder/ghrepo-rust

[LMH01]

For those that stumble across this issue because they can't find a solution on how to get cargo-llvm-cov to work:

I got it working by using the command from above (cargo llvm-cov --all-features --lcov --output-path lcov.info) and by placing my test file into a seperate tests/ folder instead of src/some_file.rs.