Tag metrics from different stages

[na--]

Reading through this issue, I liked the idea of automatically tagging the metrics from different stages, I think it could be useful when analyzing the results later.

Also, instead (or in addition) of those system tags, we might want to add the option of users to add their own key=value tags to the stages, like they currently can do whole test runs or with individual metrics. Though this option would probably have to be part of the new configuration mechanism that would be required for the arrival-rate based executor

[ppcano]

@na-- I had today the same idea when observing different ramp-up configurations and thinking how to analyze their test results.

The API could look like:

stages: [
    { duration: "30s", target: 200, tag: 'ramp-up' },
    { duration: "5m", target: 200, tag: 'baseline' },
    { duration: "30s", target: 0,  tag: 'ramp-down' }
  ],

Metrics/Data would be tagged as: stages:ramp-up, stages:baseline, stages:ramp-down.
Today, the same problem can be solved by filtering by time.

cc @robingustafsson

[na--]

Something like this would have to wait for #1007, but it shouldn't be too difficult...

[na--]

Cross-posting to note another use case for this feature, related to setting thresholds only for specific stages: #1136 (comment)

[joseray]

@ppcano this is already implemented?

[na--]

@joseray, no, it's not yet implemented. You can watch this issue, since we'll close it when we implemet the feature.

[na--]

This issue would be useful, though I think it's best if we focus our energies on other similar ones that are more flexible. If we had #1320, i.e. be able to get the number of the current stage for ramping-vus and ramping-arrival-rate, we'd be able to manually tag metrics we care about or use a group() with the stage name.

And, depending on how we choose to implement #884, #1724 and #1321, we'd get much more flexible tagging as well... I'm not going to close this issue, but I'll tag it as evaluation needed, since at this point it seems unlikely we'll implement it as is. And even if we implement it (a name for each stage seems a good idea to have in the config from a UX perspective), the implementation is likely to just be a shortcut based on the code for the issues I listed above.

[robingustafsson]

Just leaving a note here that I spoke with a customer yd that wanted to be able to set thresholds on a specific stage (the steady-state after a ramp-up stage) and asked if this was possible. If this functionality existed it would be possible to also set up thresholds limited to a certain stage, such as a steady-state stage.

Edit: I'll add also that a possible workaround to this could be separating stages via scenarios and setting up a threshold filtered by scenario name. Thanks @sniku for this suggestion.

[codebien]

Few updates for this issue:

• js/k6/exec: Set VU Tags #2172 is going to be merged in the next few days that will introduce the ability to set a custom tag from the JS code using the js/execution API. Still not something automatic as requested from this issue but it helps for the same final goal.

exec.vu.tags['stage'] = 'ramp-up'

• Still some work to do for js/execution: Stage identifier #2215 that will provide an easier way for detecting the current running stage.
• A third iteration could finally provide an automatic tagging.

[codebien]

We have merged #2172, so from the next upcoming release, a code like the following example could be used for tagging the running stage.

import exec from 'k6/execution';
import {sleep} from 'k6';

export const options = {
	stages: [
		{ duration: '10s', target: 10 },
		{ duration: '20s', target: 20 },
		{ duration: '10s', target: 0 },
	]
};

//export const options = {
	//scenarios: {
		//example: {
			//executor: 'ramping-vus',
			//stages: [
				//{ duration: '10s', target: 10 },
				//{ duration: '20s', target: 20 },
				//{ duration: '10s', target: 0 },
			//]
		//}
	//}
//};

export default function() {
	let stage = getStage()
	exec.vu.tags['stage'] = `stage#${String(stage)}`

	// write your code doing some work here
	// ...
	//

	// some work simulation
	sleep(2)
}

function getStage() {
	let stages = []; 

	if (options.hasOwnProperty('stages')) {
		stages = options.stages;
	}

	if (options.hasOwnProperty('scenarios')) {
		stages = options.scenarios[exec.scenario.name].stages;
	} 

	let sum = 0;
	let elapsed = (new Date() - exec.scenario.startTime) / 1000;
	for (let i = 0; i < stages.length; i++) {
		sum += parseInt(stages[i].duration)
		if (elapsed < sum) {
			return i
		}
	}

	return stages.length-1
}

[codebien]

In the future, we might still consider to implement the #796 original proposal as a final step, where it would add the tags object as a field of the stages option, making it consistently with other similar implementations that the API already have, in the following form:

stages: [
    { duration: "30s", target: 200, tags: { stage: 'ramp-up' } },
    { duration: "5m", target: 200, tags: { stage: 'baseline' } },
    { duration: "30s", target: 0,  tags: { stage: 'ramp-down' } }
  ],

However, analysing detail turns out that it has some caveats we should consider:

• Limited: it seems a good idea for one of the most common cases, but it isn’t very flexible for addressing completely all of them, so it would require having more specific implementations for some other cases.
• Complex: the current ramping-arrival-rate and ramping-vu executors doesn’t have a tight relationship between stages and iteration that makes a requirement, with a good probability, to have a big refactor going through several layers.

For these reasons, we decided to implement stage identification and tagging as an intermediate step by providing some helpers as part of the jslib library.
It provides a solution for the previous points and adds the advantage of getting more feedback from the community without the risk to impact the stable core API which would require a more accurate guarantee about future maintainance, instead we could prefer some quick reactions by adding more features or improvements.

Proposal

1. New helper functions in the jslib library for detecting and tagging stages.
2. Documentation: create a section (or link to a dedicated page) in the docs/using-k6/options/#stages and in ramping-arrival-rate / ramping-vu executors explaining the helper functions and how to use them. Note: The current workaround posted in the issue has some bugs. It could be nice to update with a better version in the meantime.
3. Extend the k6/execution API for exporting consolidated options. Check out the detail in k6/execution: Export consolidated options #2450.

jslib helper functions details

Create new helper functions in the jslib library. It requires a new dedicated and versioned repository (e.g. k6-jslib-stages or k6-execution/1.0.0/stages.js).

getCurrentStageIndex

It returns an integer detecting the index from the options.stages array computing and identifing the current running stage. It can return undefined if the value can't be detected.
The implementation should be a more advanced and complete solution from the basic code we previously posted: #796 (comment)

function getCurrentStageIndex() Number { }

tagWithCurrentStageIndex

It adds a stage tag based on the current stage identified using the previous function.

function tagWithCurrentStageIndex() {
  exec.vu.tags['stage'] = getCurrentStageIndex()
}

tagWithCurrentStageProfile

It auto-detects and tags when a stage is a ramp-up, a ramp-down or a steady stage. The detection checks the target's value in the previous stage and returns:

• ramp-up when previous.target < current.target
• ramp-down when previous.target > current.target
• steady when prevuious.target = current.target

[codebien]

k6 v0.38.0 is released! 🎉 It contains the previously announced functions.

We added the tagWithCurrentStageIndex and tagWithCurrentStageProfile functions to the latest v1.3.0 release of k6-jslib-utils. The documentation and examples are available here and the JavaScript API docs here.

The functions are based on the new k6/execution.test.options API so k6 v0.38.0 or greater is required.

[olegbespalov]

After internal discussion, we decided to close this issue since the desired need could be achieved with #796 (comment)

If we find out something should be improved, we could consider re-opening this issue.