Main binary assertions fail with SIGSEGV when attempting to collect coverage with kcov

[rask]

• assert_cli version: 0.5.4
• Rust version: Stable, 1.25.0
• OS and version: Linux Ubuntu 17.10

I have a CLI application in the works and wanted to see if assert_cli was of any use for me. The crate works fine when running cargo test. Neat stuff. :)

After running cargo test I use kcov to generate code coverage reports for my code. When assert_cli is in use kcov makes the tests fail with a signal: 11, SIGSEGV: invalid memory reference.

I'm not sure if this is related to my newbie Rust code or if kcov and assert_cli just don't like each other for some reason.

My code for the test:

// tests/lib_test.rs
extern crate assert_cli;

use std::env;

fn get_cwd() -> String {
    env::current_dir().unwrap().to_str().unwrap().to_string()
}

#[test]
fn test_app_config_works() {
    let cfg_file: &str = &format!("{}/tests/data/libtestwppr.toml", get_cwd());

    assert_cli::Assert::main_binary()
        .with_args(&["help"])
        .succeeds()
        .stdout()
        .contains("list")
        .unwrap();

    assert_cli::Assert::main_binary()
        .with_args(&["--configuration", cfg_file, "list"])
        .succeeds()
        .stdout()
        .contains("plugin.php")
        .stdout()
        .contains("0.1.2")
        .unwrap();
}

And the error I get with RUST_BACKTRACE=full:

failures:                                                                                                                                                                                                                                     
                                                                                                                                                                                                                                              
---- test_app_config_works stdout ----                                                                                                                                                                                                        
        thread 'test_app_config_works' panicked at 'CLI assertion failed: (command `cargo run -- --configuration /home/ojrask/Projects/ojrask/wp-plugin-repofier/tests/data/libtestwppr.toml help` expected to succeed)                       
status=failed                                                                                                                                                                                                                                 
stdout=``````                                                                                                                                                                                                                                 
stderr=```error: process didn't exit successfully: `rustc -vV` (signal: 11, SIGSEGV: invalid memory reference)                                                                                                                                
```', /home/ojrask/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/assert_cli-0.5.4/src/assert.rs:368:13                                                                                                                                      
stack backtrace:                                                                                                                                                                                                                              
   0:     0x5555556cc6f3 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h09c1ee131a74b1c4                                                                                                                                       
                               at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49                                                                                                                                                               
   1:     0x5555556c3924 - std::sys_common::backtrace::_print::h47a337b62b6d5e9e                                                                                                                                                              
                               at libstd/sys_common/backtrace.rs:71                                                                                                                                                                           
   2:     0x5555556c92fd - std::panicking::default_hook::{{closure}}::h945a649c9017832e                                                                                                                                                       
                               at libstd/sys_common/backtrace.rs:59                                                                                                                                                                           
                               at libstd/panicking.rs:380                                                                                                                                                                                     
   3:     0x5555556c8f9b - std::panicking::default_hook::hcc534c2d30fbcda3                                                                                                                                                                    
                               at libstd/panicking.rs:390                                                                                                                                                                                     
   4:     0x5555556c9770 - std::panicking::rust_panic_with_hook::h09a7a3a353dc2f38                                                                                                                                                            
                               at libstd/panicking.rs:576                                                                                                                                                                                     
   5:     0x5555556c962e - std::panicking::begin_panic::h8327f16bde15df70                                                                                                                                                                     
                               at libstd/panicking.rs:537                                                                                                                                                                                     
   6:     0x5555556c9529 - std::panicking::begin_panic_fmt::h42ff1d37404632d6                                                                                                                                                                 
                               at libstd/panicking.rs:521                                                                                                                                                                                     
   7:     0x5555555b667d - assert_cli::assert::Assert::unwrap::h0b7b03819134f9eb                                                                                                                                                              
                               at /home/ojrask/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/assert_cli-0.5.4/src/assert.rs:368                                                                                                             
   8:     0x55555557256c - lib_test::test_app_config_works::h1f9913950e84dc3c                                                                                                                                                                 
                               at tests/lib_test.rs:13                                                                                                                                                                                        
   9:     0x55555557f7f1 - <F as alloc::boxed::FnBox<A>>::call_box::h8106479744bcbe3a                                                                                                                                                         
                               at libtest/lib.rs:1462                                                                                                                                                                                         
                               at /checkout/src/libcore/ops/function.rs:223                                                                                                                                                                   
                               at /checkout/src/liballoc/boxed.rs:788                                                                                                                                                                         
  10:     0x5555556e1efe - __rust_maybe_catch_panic                                                                                                                                                                                           
                               at libpanic_unwind/lib.rs:102                                                                                                                                                                                  
  11:     0x555555577063 - std::sys_common::backtrace::__rust_begin_short_backtrace::hbe6e8f3a8f0dc6f8                                                                                                                                        
                               at /checkout/src/libstd/panicking.rs:458                                                                                                                                                                       
                               at /checkout/src/libstd/panic.rs:358                                                                                                                                                                           
                               at libtest/lib.rs:1414                                                                                                                                                                                         
                               at /checkout/src/libstd/sys_common/backtrace.rs:136                                                                                                                                                            
  12:     0x555555578162 - std::panicking::try::do_call::he62ddbc80b0afa86                                                                                                                                                                    
                               at /checkout/src/libstd/thread/mod.rs:406                                                                                                                                                                      
                               at /checkout/src/libstd/panic.rs:293                                                                                                                                                                           
                               at /checkout/src/libstd/panicking.rs:479                                                
  13:     0x5555556e1efe - __rust_maybe_catch_panic        
                               at libpanic_unwind/lib.rs:102                                                           
  14:     0x55555557f582 - <F as alloc::boxed::FnBox<A>>::call_box::h4290b0f48d980187
                               at /checkout/src/libstd/panicking.rs:458
                               at /checkout/src/libstd/panic.rs:358
                               at /checkout/src/libstd/thread/mod.rs:405
                               at /checkout/src/liballoc/boxed.rs:788
  15:     0x5555556d962b - std::sys::unix::thread::Thread::new::thread_start::h711c51a13a158afa
                               at /checkout/src/liballoc/boxed.rs:798
                               at libstd/sys_common/thread.rs:24
                               at libstd/sys/unix/thread.rs:90
  16:     0x7ffff71ae7fb - start_thread
  17:     0x7ffff6cc4b5e - clone
  18:                0x0 - <unknown>


failures:
    test_app_config_works

As I stated: the tests pass just fine when I run a plain cargo test instead of using kcov. kcov operates by taking the built test binaries and running them through again, while looking at what is being run. Would this pose a problem for kcov or assert_cli which is used to run "external" bins that are not directly built from the test case?

When I remove the lib_test.rs file from kcov runs the tests pass for kcov again, but nothing relating to assert_cli is reported as covered.

The application I'm working on is over at https://github.com/rask/wppr which contains the logic right now, the above tests are the only thing I've added which are not in the repo. You can see the coverage.sh for a script which I use to run coverage tests on my code. (The in-repo coverage.sh is missing the kcov definition for running the above lib_test.rs file, but it is the same as the wordpress_test- pattern inclusion in there.)

Please let me know if this issue should be moved somewhere else. :)

[rask]

Okay, I fixed the SIGSEGV by using the following:

extern crate assert_cli;

use std::env;

fn get_cwd() -> String {
    env::current_dir().unwrap().to_str().unwrap().to_string()
}

#[test]
fn test_app_config_works() {
    let cfg_file: &str = &format!("{}/tests/data/libtestwppr.toml", get_cwd());
    let bin: &str = &format!("{}/target/debug/wppr", get_cwd());

    assert_cli::Assert::command(&[bin, "help"])
        .succeeds()
        .stdout()
        .contains("list")
        .unwrap();

    assert_cli::Assert::command(&[bin, "--configuration", cfg_file, "list"])
        .succeeds()
        .stdout()
        .contains("plugin.php")
        .stdout()
        .contains("0.1.2")
        .unwrap();
}

So instead of using the main_binary() provided by assert_cli I just call the built binary instead, similarly to what @epage does in cobalt from what I quickly viewed. I would presume that this means kcov and main_binary() are not compatible on some level.

But now kcov says the lib code is not being covered at all so I need to tackle that next.

[tismith]

Thanks for the update and workaround here - I appreciate it. I'm now seeing the same problem as you @rask where kcov isn't seeing coverage for the binaries that assert_cli is running. Have you managed to tackle this at all?

[epage]

RE coverage reporting, see #104

(forgot to transfer it over from https://github.com/rust-lang-nursery/cli-wg/issues/9)

[epage]

The responsibility for this is moving to assert_cmd and tis will be addressed in assert-rs/assert_cmd#5

assert_cli is going to morph into a wider-focused tool, including things like assert_fs. See #41