The SetupData of the ActiveUVs has not changed when calling the Update/Run Setup by RestAPI

[chzhuo]

Brief summary

To achieve this, I think we should adjust this code:

k6/js/runner.go

Lines 733 to 745 in 4f5c142

	// Unmarshall the setupData only the first time for each VU so that VUs are isolated but we
	// still don't use too much CPU in the middle test
	if u.setupData == nil {
	if u.Runner.setupData != nil {
	var data interface{}
	if err := json.Unmarshal(u.Runner.setupData, &data); err != nil {
	return fmt.Errorf("error unmarshaling setup data for the iteration from JSON: %w", err)
	}
	u.setupData = u.Runtime.ToValue(data)
	} else {
	u.setupData = goja.Undefined()
	}
	}

k6 version

k6 v0.43.0 ((devel), go1.19.6, darwin/arm64)

OS

darwin/arm64

Docker version and image (if applicable)

No response

Steps to reproduce the problem

1. My testing script 1.js:

import http from 'k6/http';

export const options = {
    userAgent: "k6",
    scenarios: {
        constant_request_rate: {
            executor: 'constant-arrival-rate',
            rate: 1,
            timeUnit: '1s',
            duration: '10m',
            preAllocatedVUs: 1,
        },
    }
}
export function setup() {
    return { data: Math.random() };
}

export default function (data) {
    console.log(JSON.stringify(data))
    http.request('http://test.k6.io/')
}

2. k6 run 1.js
3. wait a few seconds unit the UV become be active
4. curl -X PUT http://localhost:6565/v1/setup -H 'Content-Type: application/json' -d '{"aa":"newdata"}'

Expected behaviour

console log new data newdata after changing the SetupData by RestAPI

Actual behaviour

still printing the old SetupData after changing the SetupData by RestAPI

[na--]

Right now setup() and the REST API's PUT /v1/setup endpoint only update the setupData in the js.Runner, i.e. the u.Runner.setupData in your snippet above. Which is somewhat confusingly named - the js.Runner is not the same thing as a JavaScript VU, it's the "factory" that makes new JS VUs.

So, this code is not a bug, it is working precisely as intended. The setup data can only be set before the test starts - by by running setup() automatically (the default behavior), or running it through the REST API (e.g. if you started k6 with --no-setup), or directly setting the raw setup data through the REST API. The way to run setup() or set the data from the REST API before the test starts is by using the paused option, e.g. k6 run --no-setup --paused script.js - k6 will initialize VUs, but won't actually start the test until you resume the test by calling this REST API or just executing k6 resume.

So, while all of this is working as intended, I am not going to close the issue. I just changed the label from a bug to an enhancement, since it's about expanding the currently allowed behavior. It will have performance and maintainability implications, so it's not a simple decision to make. Please share your use case as to why you'd want this, since that would help us with decided whether we want to change this behavior.

[chzhuo]

My testing URLs are stored in ETCD. I launch a new program to watch the ETCD and then call the K6 RestAPI(Update Setup Data) to set new URLs into K6

[na--]

Hmm for use cases like these, instead of relying on something external to k6 to update something like the setupData, I'd probably prefer a solution that's entirely internal to k6 🤔

For example, it seems like etcd has a gRPC API and k6 can make gRPC calls. So, if we had a data structure like SharedArray that is shared between VUs, but which was mutable, you could have one constant-arrival-rate scenario that queries etcd once every X seconds and updates these shared URL records, and another scenario that reads the URLs to load test them.

You wouldn't need to script anything outside of k6, and such a solution would be also useful for other things as well, e.g. renewing session tokens asynchronously on some schedule different from the main load test, dynamic auto-scaling, etc. It could also be misused and there are certain synchronization issues that need to be carefully handled with distributed execution (#140), but they wouldn't be much more complicated than regular setup() and SharedArray concerns which my limited PoC in #2816 already addresses 🤔

This issue definitely deserves a lot of further consideration (which is why I tagged it with evaluation needed), but an internal approach like that seems much better to me than allowing an external process to change the setup data mid-test 🤔