-
Notifications
You must be signed in to change notification settings - Fork 218
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
808/about executors #974
808/about executors #974
Changes from 21 commits
bb72e32
f970d1a
702149a
d46d5bb
6bddcc5
94aa2d9
99aacd9
c92b6dd
0c12d52
8596a32
ed10eb2
cb02cef
2becf58
b547cbc
020260c
b068f6f
98a361d
872b78f
c658116
4b7b088
977c45d
f580420
5fa0a38
6c0e924
11e7c43
19784d4
5c6f9f8
9aaf8b8
714d037
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
--- | ||
title: "About scenarios" | ||
excerpt: High-level explanations about how your executor configuration can change the test execution and test results | ||
--- | ||
|
||
These topics explain the essential concepts of how scenarios and their executors work. | ||
|
||
Different scenario configurations can affect many different aspects of your system, | ||
including the generated load, utilized resources, and emitted metrics. | ||
If you know a bit about how scenarios work, you'll design better tests and interpret test results with more understanding. | ||
|
||
| On this page | Read about | | ||
|-------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------| | ||
| [Open and closed models](/using-k6/scenarios/about-scenarios/open-vs-closed/) | Different ways k6 can schedule VUs, their affects on test results, and how k6 implements the open model in its arrival-rate executors | | ||
| [VU allocation](/using-k6/scenarios/about-scenarios/vu-allocation/) | How k6 allocates VUs in arrival-rate executors | | ||
| [Dropped iterations](/using-k6/scenarios/about-scenarios/dropped-iterations/) | Possible reasons k6 might drop a scheduled iteration | | ||
| [Graceful Stop](/using-k6/scenarios/about-scenarios/graceful-stop) | A configurable period to let iterations finish or ramp down after the test has reached it's scheduled duration | | ||
|
Original file line number | Diff line number | Diff line change | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
@@ -0,0 +1,102 @@ | ||||||||||
--- | ||||||||||
title: VU allocation | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "VU allocation" seems a bit misleading, since we are only talking about arrival-rate scenarios
Suggested change
or maybe even
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Need to think about this. Not sure. It's a good point though. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Now that I've thought about it, I don't want to go with "Arrival-rate configuration" because that should include Graceful stop and maybe more, which opens a whole new round of content structure. It's nice to keep info atomic. Is VU allocation in non-arrival-rate ever important? If so, we could just add it to the doc later. Either way, I choose these new titles, ranked by preference. You can pick and that's what will go with:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
graceful stop is not specific to arrival-rate executors and already has its own dedicated page that explains it: https://k6.io/docs/using-k6/scenarios/graceful-stop/
Well, considering the configuration of non-arrival-rate executors is specified in terms of VUs, there isn't really anything complicated to explain there 😅 Again, the complexity with arrival-rate is not just how VUs are allocated, but how to balance the VUs and the desired rate and how to find the right values for the former based on the latter and on iteration duration. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Is this not encompassed in There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "VU Pre-allocation" only makes sense to you because you already know it applies for arrival-rate executors. A new user won't know that fact and won't click on that menu entry at all, even if this is exactly the information they are looking for. |
||||||||||
excerpt: How k6 allocates VUs in the open-model, arrival-rate executors | ||||||||||
--- | ||||||||||
|
||||||||||
In arrival-rate executors, as long as k6 has VUs available, it starts iterations according to your target rate. | ||||||||||
The ability to set iteration rate comes with a bit more configuration complexity: you must pre-allocate a sufficient number of VUs. | ||||||||||
In other words, before the tests runs, you must both: | ||||||||||
- Configure load (as iteration per unit of time) | ||||||||||
MattDodsonEnglish marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
- Ensure that you've scheduled enough VUs. | ||||||||||
MattDodsonEnglish marked this conversation as resolved.
Show resolved
Hide resolved
MattDodsonEnglish marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
|
||||||||||
Read on to learn about how k6 allocates VUs in the arrival-rate executors. | ||||||||||
|
||||||||||
## Pre-allocation in arrival-rate executors | ||||||||||
|
||||||||||
As [open-model](/using-k6/scenarios/about-scenarios/open-vs-closed/#open-model) scenarios, arrival-rate executors start iterations according to a configured rate. | ||||||||||
For example, you can configure arrival-rate executors to start 10 iterations each second, or minute, or hour. | ||||||||||
This behavior is opposed to the closed-model scenarios, in which VUs wait for one iteration to finish before starting another | ||||||||||
|
||||||||||
Each iteration need needs a VU to run it. | ||||||||||
Because k6 VUs are single threaded, like other JS runtimes, a VU can only run the event loop of a single iteration at a time. | ||||||||||
MattDodsonEnglish marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
To ensure you have enough, you must pre-allocate a sufficient number. | ||||||||||
|
||||||||||
In your arrival-rate configuration, three properties determine the iteration rate: | ||||||||||
- `rate` determines how many iterations k6 starts. | ||||||||||
- `timeUnit` determines how frequently it starts the number of iterations. | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Having these 2 in different lines, with a separate explanation for each, is more confusing than helpful in my opinion. The current disjointed explanation is both longer, more confusing and less correct than "k6 will try to start |
||||||||||
- `preAllocatedVUs` sets the number of VUs to use to reach the target iterations per second. | ||||||||||
In practice, determining the right number of iterations might take some trial and error, | ||||||||||
as the necessary VUs entirely depends on how quickly the SUT can process iterations. | ||||||||||
MattDodsonEnglish marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
|
||||||||||
```javascript | ||||||||||
export const options = { | ||||||||||
scenarios: { | ||||||||||
constant_load: { | ||||||||||
executor: "constant-arrival-rate", | ||||||||||
preAllocatedVUs: 4, | ||||||||||
rate: 8, | ||||||||||
timeUnit: "1m", | ||||||||||
}, | ||||||||||
}, | ||||||||||
}; | ||||||||||
``` | ||||||||||
|
||||||||||
|
||||||||||
<Blockquote mod="attention" title=""> | ||||||||||
|
||||||||||
In cloud tests, **both `preAllocatedVUs` and `maxVUs` count against your subscription.** | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe this can be moved in the
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I just realized this might be confusing in another way, "both And, given that There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Another problem - you mention There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm still kind of of the opinion that There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Again, if I could rewrite history, maxVUs probably would not exist 😅 But now that it exists, we should try to make it as clear as possible how it function and why it might not be a good idea to use. An admonition only for And if we want to tuck There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nobody will read the text you have highlighted in the middle of that paragraph 😅 People will just see the two admonitions and get the very wrong impression that in the cloud we will charge them for There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Point taken. With 19784d4, it looks like this: |
||||||||||
|
||||||||||
When planning a test, consider doing a trial initialization on a local machine to ensure you're allocating VUs efficiently. | ||||||||||
|
||||||||||
</Blockquote> | ||||||||||
|
||||||||||
## How k6 uses allocated VUs | ||||||||||
|
||||||||||
Before an arrival-rate scenario starts, k6 first initializes the number of `preAllocatedVUs`. | ||||||||||
When the test runs, | ||||||||||
the number of available `preAllocatedVUs` determines how many iterations k6 can start. | ||||||||||
k6 tries to reach the target iterations per second, | ||||||||||
and one of two things can happen: | ||||||||||
|
||||||||||
| If the executor | Then.. | | ||||||||||
|-----------------------|----------------------------------------------------------------------------------------------------------------------------------------| | ||||||||||
| Has enough VUs | the extra VUs are "idle," ready to be used when needed. | | ||||||||||
| Has insufficient VUs. | k6 emits a [`dropped_iterations` metric](/using-k6/scenarios/about-scenarios/dropped-iterations) for each iteration that it can't run. | | ||||||||||
|
||||||||||
## Iteration duration affects the necessary allocation | ||||||||||
|
||||||||||
The necessary allocation depends on the iteration duration: | ||||||||||
Longer durations need more VUs. | ||||||||||
|
||||||||||
In a perfect world, you could estimate the number of pre-allocated VUs with this formula: | ||||||||||
|
||||||||||
``` | ||||||||||
preAllocatedVUs = [median_iteration_duration * rate] + constant_for_variance | ||||||||||
``` | ||||||||||
|
||||||||||
In the real world, if you know _exactly_ how long an iteration takes, you likely don't need to run a test. | ||||||||||
What's more, as the test goes on, iteration duration likely increases. | ||||||||||
If response times slow so much that k6 lacks the VUs to start iterations at the desired rate, | ||||||||||
the allocation might be insufficient and k6 will drop iterations. | ||||||||||
|
||||||||||
To determine your strategy, you can run tests locally and gradually add more pre-allocated VUs. | ||||||||||
As dropped iterations can also indicate that the system performance is degrading, this early experimentation can provide useful data on its own. | ||||||||||
|
||||||||||
## You probably don't need `maxVUs` | ||||||||||
|
||||||||||
The arrival-rate executors also have a `maxVUs` property. | ||||||||||
If you set it, k6 runs in this sequence: | ||||||||||
1. Pre-allocate the `preAllocatedVUs`. | ||||||||||
1. Run the test, trying to reach the target iteration rate. | ||||||||||
1. If the target exceeds the available VUs, allocate another VU. | ||||||||||
1. If the target still exceeds available VUs, continue allocating VUs until reaching the number set by `maxVUs`. | ||||||||||
|
||||||||||
Though it seems convenient, you should avoid using `maxVUs` in most cases. | ||||||||||
Allocating VUs has CPU and memory costs, and allocating VUs as the test runs **can overload the load generator and skew results**. | ||||||||||
If you're running in k6 Cloud, the `maxVUs` will count against your subscription. | ||||||||||
MattDodsonEnglish marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
In almost all cases, the best thing to do is to pre-allocate the number of VUs you need beforehand. | ||||||||||
|
||||||||||
Some of the times it might make sense to use `maxVUs` include: | ||||||||||
- To determine necessary allocation in first-time tests | ||||||||||
- To add a little "cushion" to the pre-allocated VUs that you expect the test needs | ||||||||||
- In huge, highly distributed tests, in which you need to carefully scale load generators as you increment VUs. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
--- | ||
title: Dropped iterations | ||
excerpt: Explanations about how your scenario configuration or SUT performance can lead to dropped iterations | ||
--- | ||
|
||
Sometimes, a scenario can't run the expected number of iterations. | ||
k6 tracks the number of unsent iterations in a counter metric, `dropped iterations`. | ||
The number of dropped iterations can be valuable data when you debug executors or analyze results. | ||
|
||
Dropped iterations usually happen for one of two reasons: | ||
- The executor configuration is insufficient. | ||
- The SUT can't handle the configured VU arrival rate. | ||
|
||
### Configuration-related iteration drops | ||
|
||
Dropped iterations happen for different reasons in different types of executors. | ||
|
||
With `shared-iterations` and `per-vu-iterations`, iterations drop if the scenario reaches its `maxDuration` before all iterations finish. | ||
To mitigate this, you likely need to increase the value of the duration. | ||
|
||
With `constant-arrival-rate` and `ramping-arrival-rate`, iterations drop if there are no free VUs. | ||
**If it happens at the beginning of the test, you likely just need to allocate more VUs.** | ||
If this happens later in the test, the dropped iterations might happen because SUT performance is degrading and iterations are taking longer to finish. | ||
|
||
### SUT-related iteration drops | ||
|
||
At a certain point of high latency or longer iteration durations, k6 will no longer have free VUs to start iterations with at the configured rate. | ||
As a result, the executor will drop iterations. | ||
|
||
The reasons for these dropped iterations vary: | ||
- The SUT response has become so long that k6 starts dropping scheduled iterations from the queue. | ||
- The SUT iteration duration has become so long that k6 needs to schedule more VUs to reach the target arrival rate, exceeding the number of scheduled iterations. | ||
|
||
As the causes vary, dropped iterations might mean different things. | ||
A few dropped iterations might indicate a quick network error. | ||
Many dropped iterations might indicate that your SUT has completely stopped responding. | ||
|
||
When you design your test, consider what an acceptable rate of dropped iterations is (the _error budget_). | ||
To assert that the SUT responds within this error budget, you can use the `dropped_iterations` metric in a [Threshold](/using-k6/thresholds). | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@MattDodsonEnglish I think we should list here all the executors. If not, readers might move to Concepts without getting the "general" idea of the different executor options.
I suggest adding something similar than the Executors table. For example:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ppcano I deleted that list in #936. I didn't like duplicating content in this way.
I'll make a new PR to put it back in, maybe just as a summary:
You can configure executors in to distribute workload according to:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#985