Mismatch in check counts in the end-of-test summary

[na--]

As reported in this forum topic, there's a difference between the two check counts in the end of test metrics.

It can be somewhat easily reproduced by running the following k6 script with k6 0.24.0:

import { check, sleep } from "k6";

export let options = {
    duration: "10s",
    vus: 5,
};

export default function () {
    sleep(Math.random() * 0.1);
    check(Math.random(), {
        "is below 99%": (r) => r < 0.99,
    });
}

That will result in something like the following output:

    duration: 10s, iterations: -
         vus: 5,   max: 5

    done [==========================================================] 10s / 10s

    ✗ is below 99%
     ↳  99% — ✓ 987 / ✗ 9

    checks...............: 99.09% ✓ 984 ✗ 9  
    data_received........: 0 B    0 B/s
    data_sent............: 0 B    0 B/s
    iteration_duration...: avg=50.15ms min=124.84µs med=50.51ms max=100.23ms p(90)=91.71ms p(95)=96.17ms
    iterations...........: 993    99.297203/s
    vus..................: 5      min=5 max=5
    vus_max..............: 5      min=5 max=5

I suspect that the issue stems from the current implementation of metric cutoff times. #652 has more information about that k6 detail and I suspect is a very similar issue. I'm almost positive that this would be fixed by the changes in #1007, since it improves the way we pipe metrics through the application. And even when testing the k6 build from that pull request with gracefulStop: "0s", to emulate the current k6 iteration interruption behavior, I couldn't reproduce the check count mismatch even once. That's the code I used:

import { check, sleep } from "k6";

export let options = {
    execution: {
        checktest: {
            type: "constant-looping-vus",
            vus: 5,
            duration: "10s",
            gracefulStop: "0s",
        },
    }
};

export default function () {
    sleep(Math.random() * 0.1);
    check(Math.random(), {
        "is below 99%": (r) => r < 0.99,
    });
}