Improve execution information in scripts

[na--]

Just realized that, with execution segments (#997), we can easily make the __VU constants much more useful when we're executing scripts in the cloud or in the future native k6 distributed execution.

Currently, the __VU variable will start from 1 in each instance, so for multiple instances there would be duplicate values. With execution segments, we can very easily make each instance starts its VU numbers from the exact VU id that it should, so that each __VU value is globally unique, regardless of how many machines we run the load test on. And while I think this should be the default, I can sort of see a use case where we'd like to use the local machine sequential number, not the global one, so we should probably expose both...

This ties neatly into another topic we should look into, probably after merging #1007 - exposing more execution information to user scripts. Though, instead of magic __WHATEVER constants, I think we should do it by exposing a nice JS API that queries these things from the k6 engine/execution scheduler/executors/etc. This would change the model from a push-based one (i.e. k6 having to update all __WHATEVER variables every time they're changed), to a pull based one (a function would query the already existing data in a thread-safe way), which is much easier to support and way, way more efficient (i.e. basically no performance overhead if users don't care for the information).

So, something like this:

import { getLocalVUNumber, getGlobalVUNumber, getStartTime } from "k6/execution";

//

vuNum := getGlobalVUNumber()

⬆️ is definitely NOT the final API, since it probably makes sense to expose the logical execution-related entities (i.e. "Instance", "VU", "Iteration", "Executor") in some way instead of just having global functions, I'm just using it as an illustration...

We probably should expose things like:

• local and global VU numbers
• iteration numbers:
  • have a method to duplicate __ITER, i.e. "which iteration IN this VU is currently running"
  • but since VUs can be reused, some other interesting numbers would be "iteration number for the VU in the current executor"
  • for the arrival-rate executors, in particular, it might also be useful to expose (and we can easily do it) the global number of that iteration across all instances
• when did the script execution begin and/or what's the total execution time up until now
• when did the current executor start and how long has it been running
• which executor is currently responsible for running the code
• for executors with stages, which stage are we currently at, maybe even the "progress" percent?
• execution segment related things (though that probably should be a separate API that deals with data partitioning/streming/etc. ?)
• for the current instance, the number of active/initialized VUs and the number of complete/interrupted iterations up until now (can't think of a use case for this, besides maybe something in teardown(), but we can easily expose it, since we keep track of it for the CLI interface)

Probably other things as well, once we have the initial framework and API, we can add the actual objects/functions one by one, in order of usefulness, it doesn't have to be all at once...

[na--]

As mentioned in #1656, the deprecation of MaxVUs seems to have broken some users' logic, so when we implement this, we need to offer an alternative way to answer the question "how many VUs are initialized"...

[VoloBro]

+1 looking forward for the fix.

With migration to version 0.28.0 and using new options schema - there is no way to find out MaxVUs. I have also noticed that from within the script, in setup function - you can access very limited options if you provide options as an input parameter, if as a workaround you expose entire options file that will help to mitigate the issue.

[Aemilivs]

Also looking forward for a fix.

[anvarbro]

We need this fix. Hope it will be resolved soon thanks.

[na--]

#1539 (comment) describes an important use case for having something like a getCurrentIterationNumber() method in a shared-iterations executor, something that should be fairly easy to do.

[mstoykov]

As I mentioned in this comment where the thing the users seems to need is a global (per test run) integer that can go up (possibly with more functionality but still).
The linked extension, does make that as long as the test run can be run on a single instance of k6, but for a multiple instance/distributed testing this will not work.

Then the counter needs to go outside of k6 as an endpoint of some kind that you query to get the next id that should be processed. This though will probably be way too expensive, as will be any kind of in k6 ... multi instance synchronization on even such a thing as a single integer ... for example the global (for the test run) __ITER variable. After all sometimes instances run on different continents.

But I would argue we can ... make the question a little bit different and define a group(name is bad) of VUs . Which can have shared mutable state and will need to live inside the same k6 instance. This will mean that:

1. as long as your group size can be run on a single instance it will work both locally, in the cloud and in the distributed mode
2. it will enable to group multiple VUs (maybe in different scenarios?) so that they can have shared mutable state for w/e reasons as long as they are segmented on instances in such a way so that the whole group is one instance.

I would expect ... that this will be a bit more complex, and the configuration for this and the actual API for using something like that will take soem thinking, but I think it is a sane compromise, thatgives the users who need global mutable state in a way that it is obvious that this may not work in distributed execution.

[na--]

Global generic shared mutable state is probably not in the cards any time soon. It won't be just "a bit more complex", it will be tremendously more complicated than a simple iterator 😅 First we'd need to add locking in JS, figure out the APIs for that and everything... Then we'd need figure out the partitioning of these VU "groups" among instances, which is not actually going to be an easy problem with just ExecutionSegmentSequcence.

I think if we stick to iterators (#1539 (comment)), it will cover 80% of use cases for just 5% of the cost, since we already have distributed iterators to implement the various executors, we just have to expose them in JS.

[na--]

Another use case that can benefit from knowing the current global iteration number in shared-iterations: https://community.k6.io/t/run-all-tests-scripts-in-parallel/1149 (or a generic segmented iterator that can work across multiple instances)

[na--]

As I mentioned in #977 (comment), having functions that return the current working directory and the executed script name is probably a good idea as well.

[na--]

Another user wanting to get the current script name: https://community.k6.io/t/execution-how-do-i-get-the-name-of-the-script/1889

[imiric]

As discussed over Slack and somewhat in #1863, we decided to split the implementation of this feature in two parts: the core changes needed to gather the information which is done #1863, and a separate k6 JS extension to expose it, currently WIP in grafana/xk6-execution#1.

The reasoning is that this will allow us to quickly iterate on the user-facing JS API since we expect that to change in the next few weeks based on community feedback, and once the API is stable-ish make the extension part of core again. This might happen in v0.34.0 or later.

We're also probably going to close this issue and open a new one with a smaller and more focused scope.

[na--]

A big part of this was released in k6 v0.34.0: https://github.com/grafana/k6/releases/tag/v0.34.0
The remaining tasks will be split apart in separate issues, so we can close this catch-all one after that.

[na--]

I'm closing this issue because, as mentioned above, large parts of it were already implemented in the k6/execution API in k6 v0.34.0.

I've created new issues for the parts that weren't implemented (#2259 and #2260), which together with the previously created #796 and #2215 should account for everything missing. Feel free to comment in these issues or open a new one if you don't see some piece of execution information you'd like to have in your k6 test runs.

Finally, to circle back to one of the things that was mentioned at the start here, options.vusMax has had a replacement since k6 v0.34.0, but we forgot to explicitly mention that you can now use the much clearer execution.instance.vusInitialized property instead.

[xoscar]

Hello everyone, I'm wondering if there is something that can be used from the perspective of the extension to get these values, either from the vu instance or the goja runtime. I'm looking for the script name, k6 version, and any metadata about the vu. Thanks!

[oleiade]

Hey @xoscar 👋🏻

• script name: I think this is indeed not possible at the moment, and although it was mentioned "en passant" in issues such as k6/execution: add more information about the current test run #2259, it was deemed a rather low priority. Can you confirm you're interested in the name of the test script (not its path). If so, could you open a dedicated issue for this ?
• k6 version: I think this is also not possible at the moment, although the information is already present in the k6 binary, as we have a constant holding its value. I think giving access to such info might be a good candidate for the k6/execution module. Could you please open a dedicated issue for this too?
• any metadata about the vu: as you probably already know, we already expose some metadata about VUs as part of our k6/execution module, could you please be more specific about the kind of information you'd like access to that's not already exposed by it?

🙇🏻