Possibility to support CLIs' integration tests?

[mssun]

Any chance to support CLI app's integration tests? For example, tarpaulin won't calculate coverage when using assert_cli https://github.com/assert-rs/assert_cli to fork and execute CLI and test. Thanks.

Related discussion: https://github.com/rust-lang-nursery/cli-wg/issues/9#issuecomment-385487756

[xd009642]

Do you have any examples of tests using assert_cli where tarpaulin doesn't work? Just because that's the first step in supporting this. Also, tarpaulin should follow forks down so is there something more going on in assert_cli?

[mssun]

Sure.

Using an hello_world app as an example:

First crate the app:
cargo new hello_world --bin

Add a integration test using assert_cli:

$ cat src/main.rs
fn main() {
    println!("Hello, world!");
}

$ cat tests/tests.rs
extern crate assert_cli;

#[test]
fn test_main() {
    assert_cli::Assert::main_binary().succeeds().stdout().is("Hello, world!\n").unwrap();
}

$ cat Cargo.toml
[package]
name = "hello_world"
version = "0.1.0"
authors = ["Bob Sun <bob@mssun.me>"]

[dependencies]

[dev-dependencies]
assert_cli = "0.5"

Here is the result of tarpaulin:

cargo tarpaulin
Building project
Launching test
running /ssd/mssun/hello_world/target/debug/deps/hello_world-d8edb80c0ec19daa

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Launching test
running /ssd/mssun/hello_world/target/debug/deps/tests-db878f2c6d8489d2

running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Coverage Results
src/main.rs: 0/2
tests/tests.rs: 2/2

50.00% coverage, 2/4 lines covered
Tarpaulin finished

The src/main.rs should be covered by the test case, but it is not showed in the result.

[xd009642]

This is definitely due to cargo run. Part of me hoped it would be as easy as removing the --release flag since that would cause an issue but the same thing happened. I'll have a look into it but it might take a while till I have anything, there are a lot of moving parts.

[io12]

It looks like tarpaulin doesn't support multiprocess tracing. When a test execs another process, tarpaulin detaches ptrace from it. Other parts of the code only focus on single-process tracing. For example, LinuxData has a current field that only tracks one PID.

[xd009642]

I did try and do some work on this a month ago or so which I summarised some things from here #107 and started here https://github.com/xd009642/tarpaulin/tree/follow_spawned_processes though I haven't pushed anything apart from a test.

Party of the problem I have to tackle is when a process is launched I need to stop it, get it's executable name and figure out if it's something I can instrument? (Is it in the target folder), instrument it and then potentially handle both two machines.

[xd009642]

If this involves fork it won't work yet, but if it just executes another process I should have a working prototype of following process exec's on #615 if you want to try it out. So far I've just tried it on a very simple toy project so more people testing it is appreciated 😄. Adding fork support should be trivial and a next step

[xd009642]

So I don't yet follow fork children. But I follow fork parents and execed processes now. I tried on assert_cmd and got some coverage but I did notice fork events so may not quite be as thorough as you'd want. I'm just going to close this in favour of a more specific issue for forks as they're the current missing piece. #616

[ChrisGreenaway]

Anyone finding this issue via search. Use --follow-exec to enable it.