setup() per scenario

[sniku]

Ability to have a per-scenario setup() function.

Feature Description

The current setup() function is global for the test script. It gets executed only once before the test starts.
The usefulness of the global setup() function is limited when many scenarios are used.
The setup() function isn't scenario-aware, so it's not possible to return different data depending on the scenario.
While it's possible to return an object {} of data for all scenarios from the setup() function, it's not very good user experience.

Sample script showing the

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  scenarios: {
    my_web_test: {
      executor: 'constant-vus',
      exec: 'webtest',
      // ...
    },
    my_api_test_1: {
      executor: 'constant-arrival-rate',
      exec: 'apitest',
      // ...
    },
  },
};

export function setup(){
  return {
    websiteUsers: [
     {username: "webuser1", password: "supersecret1"},
     {username: "webuser2", password: "supersecret2"},
    ],
    apiUsers: [
     {username: "apiuser1", password: "supersecret1"},
     {username: "apiuser2", password: "supersecret2"},
    ]
  }

}

export function webtest(setupData) {
  let websiteUsers = setupData.websiteUsers;
  http.post("http://test.k6.io/login.php", {
    login: websiteUsers[0].username,
    password: websiteUsers[0].password, redir: '1'
  });
}

export function apitest(setupData) {
  let apiUsers = setupData.apiUsers;
  http.post(`https://test-api.k6.io/auth/token/login/`, {
    username: apiUsers[0].username,
    password: apiUsers[0].password
  });
}

Suggested Solution

This is open for discussion, but the obvious suggestion is to follow the same pattern as with the exec function.

  scenarios: {
    my_web_test: {
      executor: 'constant-vus',
      exec: 'webtest', 
      setup: `webtestSetup`
    },
    my_api_test_1: {
      executor: 'constant-arrival-rate',
      exec: 'apitest',
      setup: `apiTestSetup`
    },
  },

The scenario-specific setup() function should be executed right before the scenario starts rather than at the beginning of the test.

[mstoykov]

The scenario-specific setup() function should be executed right before the scenario starts rather than at the beginning of the test.

Currently, you can say that a scenario should start after X time of the test has passed. Where the idea is that it will actually start at that time ... not that a setup function will start and possibly a few seconds later(or minutes) it will actually start the actual test.

Also, this will mean that if the setup errors out ... there is this case where possibly parts of the test will execute while others won't

[na--]

Yeah, given that VUs can be reused between scenarios, something like this will be a corner case factory. Probably not impossible, but much, much trickier than #785

[na--]

Given these two things:

• we're mulling over a per-scenario VU init function, i.e. being able to specify that each VU should run some piece of code before it starts running a scenario: Per-VU init lifecycle function #785 (comment)
• we've recently added (shareable array implementation #1739) a SharedArray magical object that is essentially a sync.Once + read-only array, and we plan to eventually add a SharedMap / SharedObject type as well

I think we can use both of these to implement 90+% of this proposed feature with no additional changes to k6, almost completely avoiding the corner case factory I mentioned above. Essentially, if the per-scenario VU init function is something like this:

function vuInitForScenarioX(){
  return new SharedObject('vuInitForScenarioXData', function () {
    // This function will be called only once, and the request below will be executed only once, 
    / /but other VUs will get a memory-safe read-only reference to the data.
    let resp = http.get(/* ... */)
    return {foo: 123, data: JSON.parse(resp.body), /* ... */};
  });
}

The only potential drawback to this that I can think of is that for distributed/cloud tests that run on multiple instances, the lambda will be executed once on every instance. This may actually be a feature for some use cases, but it's definitely going to be a drawback for others. Still, for those, the global setup() is going to be executed only once for the whole test run, so it's still going to be a viable workaround.

So, yeah, given that per-scenario per-VU init functions are more flexible and can be used to implement 90+% of the per-scenario setup() requirements, I vote for not doing this at all and focusing on #785