libtest-mimic test failure not recognised

[niluxv]

I'm using a custom test harness using libtest-mimic (version 0.6.0), but when one of those tests fails, cargo nextest run still reports it as passed.

[sunshowers]

Hi there, thanks for the report! I don't seem to be able to reproduce this issue with datatest-stable, with:

• keeping libtest-mimic at 0.5.2 or bumping it to 0.6.0
• putting in a panic!
• returning Err(something)

Could you provide a minimal reproducible example?

[niluxv]

Hm, it seems to require using Trial::with_kind.

Example:

use libtest_mimic::{Arguments, Failed, Trial};

fn main() -> anyhow::Result<()> {
    let args = Arguments::from_args();
    let tests = vec![
        Trial::test("example.txt", erroring_test).with_kind("parse")
    ];
    libtest_mimic::run(&args, tests).exit();
}

fn erroring_test() -> Result<(), Failed> {
    // either return error or panic!, cargo nextest will still show as `PASS`
    //panic!();
    Err("Error, this doesn't work".into())
}

[sunshowers]

Ah thanks for the extended report. This is a bug in libtest-mimic.

[sunshowers]

I've proposed LukasKalbertodt/libtest-mimic#30 in libtest-mimic which will fix this. For now, please avoid using with_kind.

[niluxv]

Thank you for looking into this!

[sunshowers]

This should be fixed with a libtest-mimic update. Thanks for the report!