Test suites / execute multiple scripts with k6

[na--]

I'm creating this issue as a place to discuss possible ways for k6 to execute multiple distinct scripts in a single run. This is prompted by https://community.k6.io/t/running-10-test-cases-out-of-100/448, and other similar previous user inquiries.

Something like this can currently be achieved by a simple shell script that sequentially calls k6 run "$script_path" for all files in a folder, but that workaround is not very convenient without an external output like InfluxDB. It also doesn't work well with k6 cloud, and isn't efficient in general (each separate k6 run has to re-initialize its VUs). GitLab’s performance testing tool is probably also worth investigating...

I also have a vague idea how we can implement something like "test suites" natively in k6 after #1007 is done, and do it relatively easily (:tm: :sweat_smile: ). To get 80% of the way there, we just need to implement one minor (sort-of-already-planned) improvement on the currently planned #1007 feature set 🎉. Instead of specifying startTime: "30s" for an executor, users should be able to specify startAfter: "someOtherExecutorName". This will be very easy to implement (minor changes in the config validation, here and here).

This will add a lot of convenience in general, so it's probably worth doing just for that. But it will also, when used with the exec option for each executor (to run some non-default function), probably satisfy the "test suite" requirements of the majority of our users, albeit with some manual configuration work (which we can ameliorate with some JS helper functions).

At a later point, to satisfy the remaining minority of more complex use cases, and to achieve something similar to k6 run test1.js test2.js test3.js (and, crucially, k6 cloud test1.js test2.js test3.js), we can make another (somewhat more tricky) change to k6. We need to transform the current local.ExecutionScheduler into just another executor. That is, we will add a new composite executor type that will be able to contain other executors. I may be wrong, but because of the new executor architecture in k6, and because we already have 95% of the required code in local.ExecutionScheduler, I think this will actually be very straightforward to do 🎉 😄 At least on the execution side, there will be some complexity elsewhere.

To illustrate, for example, if test1.js has vus: 20, duration: "30s" in its exported options, test2.js has iterations: 1000, vus: 20, and test3.js has something like this:

export let options = {
  execution: {
    constant_arr_rate: {
      type: "constant-arrival-rate",
      rate: 20,
      duration: "1m",
      preAllocatedVUs: 10,
    },
    per_vu_iters: {
      type: "per-vu-iterations",
      vus: 10,
      iterations: 100,
      maxDuration: "1m"
    },
  },
  // ...
}

then, when you run k6 run test1.js test2.js test3.js, it should be relatively easy to produce something like this as the final derived options.execution value:

{
  "test1.js": {
    type: "composite",
    exec: "test1.js/default", // this is actually the tricky bit :D 
    options: {
      vus: 20,
      duration: "30s",
    },
  },
  "test2.js": {
    type: "composite",
    exec: "test2.js/default", 
    startAfter: "test1.js",
    options: {
      iterations: 1000,
      vus: 20,
    },
  },
  "test3.js": {
    type: "composite",
    exec: "test3.js/default", 
    startAfter: "test2.js",
    options: {
      execution: {
        constant_arr_rate: {
          type: "constant-arrival-rate",
          rate: 20,
          duration: "1m",
          preAllocatedVUs: 10,
        },
        per_vu_iters: {
          type: "per-vu-iterations",
          vus: 10,
          iterations: 100,
          maxDuration: "1m"
        },
      },
    },
  },
}

Notice how, with a few minor changes, we have a whole new layer of flexibility - test suites, for lack of a better term. All the wile, we're preserving the nice new properties of being able to tell precisely how many VUs our test suite will use at most (20 in this case) and what the max test suite duration will be, giving us predictability and easy billing.

And, as I mention in the code comment above, the tricky bit isn't actually the execution. It would actually be things like figuring out how to reference the different default functions in the config. I don't think it's a huge issue, but we should be careful how we structure the archive bundles and the related things.

Also, because of the super-global k6 configuration (the curse of #883 strikes again 🤦‍♂️ ), I'm not sure how we should handle non-execution options that are different between scripts. We can probably refactor that at a later point though. And things like the planned new HTTP API (issue TBD 😊 ) would greatly reduce the issue... Still, some refactoring is likely required, even for a MVP version of the "full test suites 100% version"...

So, yeah, this seems reasonable, after we release #1007:

1. Add a small PR that implements startAfter, cover (hopefully) 80% of use cases
2. Add a medium-sized PR that implements the composite executor. No support for k6 run script1.js script2.js at this point, users have to manually configure the hierarchical config). Hopefully that covers 90% of use cases.
3. Figure out the tricky bits of implementing k6 run script1.js script2.js. This will give us great UX and hopefully 100% coverage of use cases. We can do this step much, much later than steps 1 and 2. Probably smart to do it after we have the new HTTP API.

[na--]

I'll give an example why I think points 1 and 2 above cover 80-90% of the use cases. Here's how, if we add the startAfter property and the composite executor, you can make an entry file test-suite.js that would allow you to run multiple independent tests:

import test01 from "./test01.js";
import test02 from "./test02.js";
import test03 from "./test03.js";

export let options = {
  execution: {
    "test1": {
      type: "composite",
      exec: "test01.default",
      options: test01.options,
    },
    "test2": {
      type: "composite",
      exec: "test01.default",
      options: test02.options,
      startAfter: "test1",
    },
    "test3": {
      type: "composite",
      exec: "test02.default",
      options: test02.options,
      startAfter: "test2",
    },
  }
}

You can execute any of the individual scripts like k6 run test01.js, and you can also execute all of them via test-suite.js. Of course, test-suite.js can be much more dynamic, with some environment variables and helper functions you can build much more flexible and complex execution pipelines, if that's required.

And you'd be able to do something similar even without the composite executor, though it will be a bit fiddly to construct the new options and you'd be restricted to only having a single executor in each sub-test.

[imiric]

I like this proposal, especially passing multiple scripts to k6 run. Being able to do k6 run tests/* would be great.

The idea for the composite executor makes sense, though I'm also worried about the complexity.

One improvement might be to pass actual functions to "exec" instead of string, so that one could do:

import * as test1 from "./test1.js";
import * as test2 from "./test2.js";
import * as test3 from "./test3.js";

export let options = {
  execution: {
    "test1": {
      type: "composite",
      exec: test1.default,
      options: test1.options,
    },
    ...

This seems intuitive and would avoid the parsing/matching unpleasantries if it were a string.

[na--]

The idea for the composite executor makes sense, though I'm also worried about the complexity.

I may be wrong, but I think this refactoring will actually reduce the complexity somewhat, at least when it comes to the script execution. We'll see when we come to it.

One improvement might be to pass actual functions to "exec" instead of string

Unfortunately, this probably wouldn't be possible. In the init context, k6 has to get the exported script options in a format that Go (and things like the cloud backend) understand. We can't just get a reference to a function in a specific goja VM (init context), because that reference won't be valid across the other goja VMs (VUs).

[mstoykov]

One improvement might be to pass actual functions to "exec" instead of a string

Unfortunately, this probably wouldn't be possible. In the init context, k6 has to get the exported script options in a format that Go (and things like the cloud backend) understand. We can't just get a reference to a function in a specific goja VM (init context), because that reference won't be valid across the other goja VMs (VUs).

While that is true I wonder if it is worth the trouble and (somewhat) inconsistency of having the go code figure out that test1.default is actually "test1.default" so that people can actually use the functions as are ... I also have a slight feeling that will be ... impossible but this depends on goja a lot, and I am not about to try to figure it out :). (also won't work with lambdas ... so maybe not such a great idea on my part).

In the same vein, I would like to point out that I am not certain we can just call test1.default if it isn't exported (as that exact name) by the actual main script ...
So there are some tricky stuff even with the "easy" part IMO, but let us leave it for when we actually can work on it.

[na--]

Thinking about an unrelated issue, I realized that my initial plan above has a major problem - startAfter wouldn't be as simple to implement as I thought 😞

Most executors have a specified duration that they can't deviate from. For constant-looping-vus, variable-looping-vus, externally-controlled, constant-arrival-rate, and variable-arrival-rate, startAfter can be a simple shortcut option that will save users from the effort of manually adding up times to calculate startTime values.

But iteration-based executors, currently only shared-iterations and per-vu-iterations, don't have a constant execution duration, they only have a maximum one. That is, they can finish early. So, any VU requirements of follow-up executors have to account for that. Which is problematic, since executors like the variable-looping-vus have very time-dependent VU requirements that aren't easy to calculate...

There are solutions to this, like reserving the max potentially required VUs for the overlapping periods, but they add significant complexity that probably isn't worth it at this point. Instead, I propose that we either initially implement startAfter as a simple shortcut option to automatically calculate the startTime value, or we don't implement it at all (i.e. rely only on startTime for the composite executor).

[na--]

Thinking some more, VU requirement calculations for a variable-looping-vus executor with a startAfter: "someIterationBasedExecutor" aren't a problem. They are complicated, but we already have the code to do them... 😅

Essentially, the VU requirements for a variable-looping-vus executor with an uncertain start time can be calculated by just "pretending" its gracefulStop value is larger 🎉 If, only for the purposes of GetExecutionRequirements(), we adjust its gracefulStop value exactly by how much the potential overlap with any previous executors is, we'll make sure that we always have sufficient VUs to run the executor, whenever it starts.

So, while I now have a good idea how to solve the startAfter issues, it adds sufficient complexity that I'd still advocate that we either start this issue without it, or to initially treat it as a startTime alias and do the optimizations later.

[Nicole1991]

I'll give an example why I think points 1 and 2 above cover 80-90% of the use cases. Here's how, if we add the startAfter property and the composite executor, you can make an entry file test-suite.js that would allow you to run multiple independent tests:

import test01 from "./test01.js";
import test02 from "./test02.js";
import test03 from "./test03.js";

export let options = {
  execution: {
    "test1": {
      type: "composite",
      exec: "test01.default",
      options: test01.options,
    },
    "test2": {
      type: "composite",
      exec: "test01.default",
      options: test02.options,
      startAfter: "test1",
    },
    "test3": {
      type: "composite",
      exec: "test02.default",
      options: test02.options,
      startAfter: "test2",
    },
  }
}

You can execute any of the individual scripts like k6 run test01.js, and you can also execute all of them via test-suite.js. Of course, test-suite.js can be much more dynamic, with some environment variables and helper functions you can build much more flexible and complex execution pipelines, if that's required.

And you'd be able to do something similar even without the composite executor, though it will be a bit fiddly to construct the new options and you'd be restricted to only having a single executor in each sub-test.

Hi, I'm wondering how can I run test-suite.js with k6 run test-suite.js?
I tried to define test-suite.js as below:

import test01 from "./test01.js";
import test02 from "./test02.js";
import test03 from "./test03.js";

export let options = {
  execution: {
    "test1": {
      type: "composite",
      exec: "test01.default",
      options: test01.options,
    },
   ...
  }
}

export default function() {
  test1.default();
  test2.default();
...
}

but it's not works as expected, could you please help me? Or can you give an example about how to run test1 & test2 & test3 together in test-suite.js?
Thanks.

[na--]

@Nicole1991, this is an open issue that discusses ways to implement test suites. Neither type: "composite", startAfter: "X" nor anything like them has been implemented yet.

The only way you can currently implement something like a test suite in k6 is if you manually calculate and specify the startTime property of each scenario. Depending on your use case, it might be easier to simply have a shell script that calls k6 run scriptN.js sequentially for your scripts.

[Nicole1991]

@Nicole1991, this is an open issue that discusses ways to implement test suites. Neither type: "composite", startAfter: "X" nor anything like them has been implemented yet.

The only way you can currently implement something like a test suite in k6 is if you manually calculate and specify the startTime property of each scenario. Depending on your use case, it might be easier to simply have a shell script that calls k6 run scriptN.js sequentially for your scripts.

Got it.
Thanks for your help.

[daniel-kun]

I'm also facing this issue, I want to run multiple tests sequentially after each other so that testing one API does not influence the results of the other API. scenarios with startAfter: "someOtherExecutorName" would totally solve my issue!
Or, alternatively, scenarios could optionally be an array instead of an object, which automatically runs the scenarios sequentially in the provided order.

[kevzettler]

are there currently any working methods for this?
@na-- after your last post here. it looks like you may have proposed a solution in the community forum at: https://community.k6.io/t/run-all-tests-scripts-in-parallel/1149 Is that solution viable?

Additionally there is a proposed solution on this blog post using groups https://trannguyenhung011086.medium.com/run-multi-load-test-scripts-with-k6-3dc57e8e26e2

Is that a viable solution? The merge of the options seems sketchy to me

[na--]

As far as I can see, the blog post merges only the thresholds and just sequentially executes the different function as a part of the default function, which seems fine if that's what you want to do.

In general, k6 can run many different scenarios in parallel without any problems. It can also run them sequentially, the only problem with that is that you have to manually calculate the startTime of each scenario to do that. That is what this issue is for.

I can't answer if something is a "viable solution" to your use case without knowing the use case... 😉 And the community forum is a better place for that discussion.

[loganknecht]

Hello!

I am chiming in here to provide a quick +1 on the requirement to be able to point to a directory and execute multiple scripts at a time.

That would be a great help for my organization! At the moment I can only serially execute the scripts and that's kind of a bummer.

If I had my pick between the suite directory execution vs composite configuration I would much much much more prefer the suite directory approach where you call k6 run "$script_path"

Thank you for the fantastic library!

[kkriegkxs]

+1
As a side note, it would be nice to have TestPlan type file that would outline scripts and their options object

[na--]

@kkriegkxs, can you explain what you think should be in the TestPlan type and how it should be used?

[kkriegkxs]

@na-- gladly. basically, If I was building it, I would add an external sidecar / playlist containing a combination of scripts and option (global and overrides for an individual test) and a mechanism to kick them off accordingly in parallel.
My current implementation relies on the external scheduling/dispatching (for automation' sake) that uses a combination of CMD params and re-writing of the options object within the script and starts a new parallel instance of K6 (w. diff control ports). (results are aggregated in influx / elastic (statsD out --> telegraf --> elastic ) and post-processing parsers.
the main gap is the need to rewrite options in the scripts to switch executor types for the same base script. (it may have changed by now, but as of the last time I looked into it, a whole bunch of things are not done at the runtime but are parsed out at script load, like options, so you cant code it in the script itself - it's the go engine that handles those).
my solution has a goal of 100% automation though, so I may be shooting for more... proxy --> autogenerated script (currently k6 js) --> source control --> pipeline --> dispatcher --> HTTP --> results storage --> aggregation/post-processing. so my angle here is automation convenience.

PM me if youd like we could chat on the phone ;)

[na--]

Thanks for the explanation! I was asking because we have had some recent internal discussions on this topic and it turns out that we might prioritize this issue and start working on it soon. I have some ideas on how to implement it without any dependency on scenarios, so basically avoiding all of the startAfter complexity that I've described above, with some additional side-benefits (e.g. also compatible with the upcoming native distributed execution I recently did a PoC for #2438). If I understand your explanation correctly, it will be closer to it than to what I've discussed previously in the issue.

So yeah, no promises or anything, but we are actively thinking about this issue and we might have something in the works soon(-ish) that will address it 🤞 I'll leave it at this for now, and will probably wait until I have a working PoC before I write up my alternative proposal for how this problem might also be solved.

[na--]

I've refactored my very basic PoC implementation of test suites and somewhat explained my ideas how it works and how a more robust version of it can be implemented into this design document. That was all a part of my specific proposal (#3218) on how to deliver native distributed execution to k6, parts of which will unlock test suites for both local and distributed tests 🤞

Though, to be clear, test suites do not explicitly depend on distributed execution, they just depend on some of the underlying APIs in the proposed execution.Controller. It should be possible to have purely local test suites just on top of slightly beefed up version of #3204 (with a ton more work around that, of course, since the current PoC is super basic).

[rilham97]

Any update for this feature ? @na--

[UjjwalSk]

Any update for this feature guys?

[kkriegkxs]

Roll your own, I guess. I do.

[Benzito164]

any update on this feature or something similar
thanks

[oleiade]

Hey folks 👋🏻

We have a rough design for the feature in mind, but no concrete implementation plans yet. Stay tuned for updates.