Split stats for stages

[neilstuartcraig]

Hi

I've noticed that when i configure a test to have stages (e.g. the classic ramp up, steady state, ramp down), i only get a single set of stats when the test completes. This isn't so much of an issue if the steady state is a large % of the overall test but it can be when the steady state is short. It'd be great to have split stats to show the test results for each configured stage.

Does that make sense? Apologies if i missed something i can already configure to do this.

Cheers

[liclac]

The best way to do this would be to output your results to something like InfluxDB and using eg. Grafana or Chronograf to graph the output - a CLI is poorly suited for displaying more complex data, so we try to keep it more as a summary than anything.

[ragnarlonn]

Hi @neilstuartcraig ! Do you need to know what stage you are in, or do you just need to know the current VU level?

You should get frequent "vu" samples (a gauge metric. Emitted once per second, unless I'm mistaken) that tell you what load level you're at, at different points in time. Although not what specific stage in your stages schedule.

EDIT: @liclac responded just before I did (and also got the issue better than I did, I think).

[neilstuartcraig]

Sorry for the slow reply @liclac & @ragnarlonn
What i'd really like is for there to be a breakdown of the existing stats format for each stage, e.g.:

Current (combined output stats for all stages)

checks................: 100.00%
    http_req_blocked......: avg=613.34µs, max=638.61ms, med=3.25µs, min=1.74µs, p90=7.76µs, p95=197.5µs
    http_req_connecting...: avg=1.7µs, max=105.99ms, med=0s, min=0s, p90=0s, p95=0s
    http_req_duration.....: avg=56.04ms, max=1.01s, med=42.3ms, min=16.02ms, p90=110.64ms, p95=175.45ms
    http_req_looking_up...: avg=523ns, max=5.65ms, med=0s, min=0s, p90=0s, p95=0s
    http_req_receiving....: avg=2.55ms, max=631.9ms, med=42.17µs, min=18.84µs, p90=217.64µs, p95=5.63ms
    http_req_sending......: avg=96.44µs, max=194.87ms, med=22.64µs, min=11.19µs, p90=48.08µs, p95=63.96µs
    http_req_waiting......: avg=52.77ms, max=1.01s, med=41ms, min=15.96ms, p90=94.46ms, p95=163.6ms
    http_reqs.............: 103197
    iterations............: 103197
    vus...................: 1
    vus_max...............: 300

I'd like something like this:

Stage 1 ("<stage name>"):

checks................: 100.00%
    http_req_blocked......: avg=613.34µs, max=638.61ms, med=3.25µs, min=1.74µs, p90=7.76µs, p95=197.5µs
    http_req_connecting...: avg=1.7µs, max=105.99ms, med=0s, min=0s, p90=0s, p95=0s
    http_req_duration.....: avg=56.04ms, max=1.01s, med=42.3ms, min=16.02ms, p90=110.64ms, p95=175.45ms
    http_req_looking_up...: avg=523ns, max=5.65ms, med=0s, min=0s, p90=0s, p95=0s
    http_req_receiving....: avg=2.55ms, max=631.9ms, med=42.17µs, min=18.84µs, p90=217.64µs, p95=5.63ms
    http_req_sending......: avg=96.44µs, max=194.87ms, med=22.64µs, min=11.19µs, p90=48.08µs, p95=63.96µs
    http_req_waiting......: avg=52.77ms, max=1.01s, med=41ms, min=15.96ms, p90=94.46ms, p95=163.6ms
    http_reqs.............: 1000
    iterations............: 100
    vus...................: 1
    vus_max...............: 300

Stage 2 ("<Stage name>")

checks................: 99.15%
    http_req_blocked......: avg=613.34µs, max=638.61ms, med=3.25µs, min=1.74µs, p90=7.76µs, p95=197.5µs
    http_req_connecting...: avg=1.7µs, max=105.99ms, med=0s, min=0s, p90=0s, p95=0s
    http_req_duration.....: avg=56.04ms, max=1.01s, med=42.3ms, min=16.02ms, p90=110.64ms, p95=175.45ms
    http_req_looking_up...: avg=523ns, max=5.65ms, med=0s, min=0s, p90=0s, p95=0s
    http_req_receiving....: avg=2.55ms, max=631.9ms, med=42.17µs, min=18.84µs, p90=217.64µs, p95=5.63ms
    http_req_sending......: avg=96.44µs, max=194.87ms, med=22.64µs, min=11.19µs, p90=48.08µs, p95=63.96µs
    http_req_waiting......: avg=52.77ms, max=1.01s, med=41ms, min=15.96ms, p90=94.46ms, p95=163.6ms
    http_reqs.............: 103197
    iterations............: 103197
    vus...................: 300
    vus_max...............: 300

does that make sense?

The way the results are currently output is a combined form which means that i can't just see e.g. my steady-state - which i want to look at separately for obvious reasons. The same would go for stepped-style tests which we quite often run.

Cheers

[liclac]

Hm. My spontaneous reaction is that something like this should be done through something like InfluxDB + Grafana or Load Impact Insights rather than like this. Splitting stages like this would complicate the way we store metrics a whole bunch + make the output really tall.

One possible way to solve it would be to tag all metrics with the current stage, which would let you split off submetrics per-stage, eg. http_req_duration{stage: 1}, http_req_duration{stage: 2}, etc.

[na--]

I've created a new issue about tagging metrics according to the stage they were produced in. Once we have that and access to the metrics after the test is done, I think it would be better to let users decide whether they want to output some of them to the screen or not, so closing this issue for now.