-
Notifications
You must be signed in to change notification settings - Fork 218
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
808/about executors #974
Merged
Merged
808/about executors #974
Changes from 28 commits
Commits
Show all changes
29 commits
Select commit
Hold shift + click to select a range
bb72e32
Scenarios, make new About section to explain
MattDodsonEnglish f970d1a
Scenarios, move other explanatory topics to about
MattDodsonEnglish 702149a
Scenarios, rename arrival rate as open vs closed
MattDodsonEnglish d46d5bb
Scenarios, clean up the list pages
MattDodsonEnglish 6bddcc5
punctuation fixes
MattDodsonEnglish 94aa2d9
Scenarios, clean up new explanation texts
MattDodsonEnglish 99aacd9
Formatting fixes to ul
MattDodsonEnglish c92b6dd
List fixes
MattDodsonEnglish 0c12d52
Fix language in iterations dropped doc
MattDodsonEnglish 8596a32
Better explain how duration affects allocation
MattDodsonEnglish ed10eb2
Add note arrival rate jitter
MattDodsonEnglish cb02cef
Merge branch 'main' into 808/about-executors
MattDodsonEnglish 2becf58
Proofreading
MattDodsonEnglish b547cbc
typo fix.
MattDodsonEnglish 020260c
Fix admonitions and simplify sentence.
MattDodsonEnglish b068f6f
Update src/data/markdown/translated-guides/en/02 Using k6/14 Scenario…
MattDodsonEnglish 98a361d
Update src/data/markdown/translated-guides/en/02 Using k6/14 Scenario…
MattDodsonEnglish 872b78f
Update src/data/markdown/translated-guides/en/02 Using k6/14 Scenario…
MattDodsonEnglish c658116
Update src/data/markdown/translated-guides/en/02 Using k6/14 Scenario…
MattDodsonEnglish 4b7b088
Fix jitter admonition
MattDodsonEnglish 977c45d
redo introductory sections
MattDodsonEnglish f580420
Explain why maxVUs counts againts rate.
MattDodsonEnglish 5fa0a38
Apply suggestions from code review
MattDodsonEnglish 6c0e924
Update src/data/markdown/translated-guides/en/02 Using k6/14 Scenario…
MattDodsonEnglish 11e7c43
Last suggestions from Neds review:
MattDodsonEnglish 19784d4
Move parenthetical to admonition
MattDodsonEnglish 5c6f9f8
Change files names and add redirects
MattDodsonEnglish 9aaf8b8
Link arrival-rate reference docs to explanation
MattDodsonEnglish 714d037
Merge branch 'main' into 808/about-executors
MattDodsonEnglish File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
18 changes: 18 additions & 0 deletions
18
src/data/markdown/translated-guides/en/02 Using k6/14 Scenarios/00 Concepts.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
--- | ||
title: "Concepts" | ||
excerpt: High-level explanations about how your executor configuration can change the test execution and test results | ||
--- | ||
|
||
These topics explain the essential concepts of how scenarios and their executors work. | ||
|
||
Different scenario configurations can affect many different aspects of your system, | ||
including the generated load, utilized resources, and emitted metrics. | ||
If you know a bit about how scenarios work, you'll design better tests and interpret test results with more understanding. | ||
|
||
| On this page | Read about | | ||
|-------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------| | ||
| [Open and closed models](/using-k6/scenarios/concepts/open-vs-closed/) | Different ways k6 can schedule VUs, their affects on test results, and how k6 implements the open model in its arrival-rate executors | | ||
| [Arrival-rate VU allocation](/using-k6/scenarios/concepts/arrival-rate-vu-allocation/) | How k6 allocates VUs in arrival-rate executors | | ||
| [Dropped iterations](/using-k6/scenarios/concepts/dropped-iterations/) | Possible reasons k6 might drop a scheduled iteration | | ||
| [Graceful Stop](/using-k6/scenarios/concepts/concepts/graceful-stop) | A configurable period to let iterations finish or ramp down after the test has reached it's scheduled duration | | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
114 changes: 114 additions & 0 deletions
114
...guides/en/02 Using k6/14 Scenarios/00 Concepts/02 Arrival-rate VU allocation.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,114 @@ | ||
--- | ||
title: Arrival-rate VU allocation | ||
excerpt: How k6 allocates VUs in the open-model, arrival-rate executors | ||
--- | ||
|
||
In arrival-rate executors, as long as k6 has VUs available, it starts iterations according to your target rate. | ||
The ability to set iteration rate comes with a bit more configuration complexity: you must pre-allocate a sufficient number of VUs. | ||
In other words, before the tests runs, you must both: | ||
- Configure load (as new iterations per unit of time) | ||
- Ensure that you've allocated enough VUs. | ||
|
||
Read on to learn about how k6 allocates VUs in the arrival-rate executors. | ||
|
||
<Blockquote mod="attention" title=""> | ||
|
||
In cloud tests, **`preAllocatedVUs` count against your subscription.** | ||
|
||
When planning a test, consider doing a trial initialization on a local machine to ensure you're allocating VUs efficiently. | ||
|
||
</Blockquote> | ||
|
||
## Pre-allocation in arrival-rate executors | ||
|
||
As [open-model](/using-k6/scenarios/concepts/open-vs-closed/#open-model) scenarios, arrival-rate executors start iterations according to a configured rate. | ||
For example, you can configure arrival-rate executors to start 10 iterations each second, or minute, or hour. | ||
This behavior is opposed to the closed-model scenarios, in which VUs wait for one iteration to finish before starting another | ||
|
||
Each iteration need needs a VU to run it. | ||
Because k6 VUs are single threaded, like other JavaScript runtimes, a VU can only run a single iteration (and its event loop) at a time. | ||
To ensure you have enough, you must pre-allocate a sufficient number. | ||
|
||
In your arrival-rate configuration, three properties determine the iteration rate: | ||
- k6 starts `rate` number of iterations evenly across the `timeUnit` (default 1s) | ||
- `preAllocatedVUs` sets the number of VUs to initialize to meet the target iteration rate. | ||
|
||
For example, with a `constant-arrival-rate` executor and `rate: 10`, k6 tries to start a new iteration every 100 milliseconds. If the scenario has `rate: 10, timeUnit: '1m'`, k6 tries to start a new iteration every 6 seconds. | ||
Whether it _can_ start the target iteration rate depends on whether the number of allocated VUs is sufficient. | ||
|
||
```javascript | ||
export const options = { | ||
scenarios: { | ||
constant_load: { | ||
executor: "constant-arrival-rate", | ||
preAllocatedVUs: 10, | ||
rate: 10, | ||
timeUnit: "1m", | ||
}, | ||
}, | ||
}; | ||
``` | ||
|
||
In practice, determining the right number of VUs to be able to reach the target iteration rate might take some trial and error, | ||
as the necessary VUs entirely depends on what your iteration code looks like and how quickly the system under test can process requests. | ||
|
||
|
||
## How k6 uses allocated VUs | ||
|
||
Before an arrival-rate scenario starts, k6 first initializes the number of `preAllocatedVUs`. | ||
When the test runs, | ||
the number of available `preAllocatedVUs` determines how many iterations k6 can start. | ||
k6 tries to reach the target iterations per second, | ||
and one of two things can happen: | ||
|
||
| If the executor | Then.. | | ||
|-----------------------|----------------------------------------------------------------------------------------------------------------------------------------| | ||
| Has enough VUs | the extra VUs are "idle," ready to be used when needed. | | ||
| Has insufficient VUs. | k6 emits a [`dropped_iterations` metric](/using-k6/scenarios/about-scenarios/dropped-iterations) for each iteration that it can't run. | | ||
|
||
## Iteration duration affects the necessary allocation | ||
|
||
The necessary allocation depends on the iteration duration: | ||
longer durations need more VUs. | ||
|
||
In a perfect world, you could estimate the number of pre-allocated VUs with this formula: | ||
|
||
``` | ||
preAllocatedVUs = [median_iteration_duration * rate] + constant_for_variance | ||
``` | ||
|
||
In the real world, if you know _exactly_ how long an iteration takes, you likely don't need to run a test. | ||
What's more, as the test goes on, iteration duration likely increases. | ||
If response times slow so much that k6 lacks the VUs to start iterations at the desired rate, | ||
the allocation might be insufficient and k6 will drop iterations. | ||
|
||
To determine your strategy, you can run tests locally and gradually add more pre-allocated VUs. | ||
As dropped iterations can also indicate that the system performance is degrading, this early experimentation can provide useful data on its own. | ||
|
||
## You probably don't need `maxVUs` | ||
|
||
<Blockquote mod="attention" title=""> | ||
|
||
In cloud tests, the number of `maxVUs` counts against your subscription, | ||
**overriding the number set by `preAllocatedVUs`**. | ||
|
||
</Blockquote> | ||
|
||
|
||
The arrival-rate executors also have a `maxVUs` property. | ||
If you set it, k6 runs in this sequence: | ||
1. Pre-allocate the `preAllocatedVUs`. | ||
1. Run the test, trying to reach the target iteration rate. | ||
1. If the target exceeds the available VUs, allocate another VU. | ||
1. If the target still exceeds available VUs, continue allocating VUs until reaching the number set by `maxVUs`. | ||
|
||
Though it seems convenient, you should avoid using `maxVUs` in most cases. | ||
Allocating VUs has CPU and memory costs, and allocating VUs as the test runs **can overload the load generator and skew results**. | ||
In Cloud tests, the number of `maxVUs` count against your subscription. | ||
This is because k6 must allocate sufficient resources for `maxVUs` to be initialized, even if they never are. | ||
In almost all cases, the best thing to do is to pre-allocate the number of VUs you need beforehand. | ||
|
||
Some of the times it might make sense to use `maxVUs` include: | ||
- To determine necessary allocation in first-time tests | ||
- To add a little "cushion" to the pre-allocated VUs that you expect the test needs | ||
- In huge, highly distributed tests, in which you need to carefully scale load generators as you increment VUs. |
40 changes: 40 additions & 0 deletions
40
...nslated-guides/en/02 Using k6/14 Scenarios/00 Concepts/03 Dropped iterations.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
--- | ||
title: Dropped iterations | ||
excerpt: Explanations about how your scenario configuration or SUT performance can lead to dropped iterations | ||
--- | ||
|
||
Sometimes, a scenario can't run the expected number of iterations. | ||
k6 tracks the number of unsent iterations in a counter metric, `dropped iterations`. | ||
The number of dropped iterations can be valuable data when you debug executors or analyze results. | ||
|
||
Dropped iterations usually happen for one of two reasons: | ||
- The executor configuration is insufficient. | ||
- The SUT can't handle the configured VU arrival rate. | ||
|
||
### Configuration-related iteration drops | ||
|
||
Dropped iterations happen for different reasons in different types of executors. | ||
|
||
With `shared-iterations` and `per-vu-iterations`, iterations drop if the scenario reaches its `maxDuration` before all iterations finish. | ||
To mitigate this, you likely need to increase the value of the duration. | ||
|
||
With `constant-arrival-rate` and `ramping-arrival-rate`, iterations drop if there are no free VUs. | ||
**If it happens at the beginning of the test, you likely just need to allocate more VUs.** | ||
If this happens later in the test, the dropped iterations might happen because SUT performance is degrading and iterations are taking longer to finish. | ||
|
||
### SUT-related iteration drops | ||
|
||
At a certain point of high latency or longer iteration durations, k6 will no longer have free VUs to start iterations with at the configured rate. | ||
As a result, the executor will drop iterations. | ||
|
||
The reasons for these dropped iterations vary: | ||
- The SUT response has become so long that k6 starts dropping scheduled iterations from the queue. | ||
- The SUT iteration duration has become so long that k6 needs to schedule more VUs to reach the target arrival rate, exceeding the number of scheduled iterations. | ||
|
||
As the causes vary, dropped iterations might mean different things. | ||
A few dropped iterations might indicate a quick network error. | ||
Many dropped iterations might indicate that your SUT has completely stopped responding. | ||
|
||
When you design your test, consider what an acceptable rate of dropped iterations is (the _error budget_). | ||
To assert that the SUT responds within this error budget, you can use the `dropped_iterations` metric in a [Threshold](/using-k6/thresholds). | ||
|
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@MattDodsonEnglish I think we should list here all the executors. If not, readers might move to Concepts without getting the "general" idea of the different executor options.
I suggest adding something similar than the Executors table. For example:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ppcano I deleted that list in #936. I didn't like duplicating content in this way.
I'll make a new PR to put it back in, maybe just as a summary:
You can configure executors in to distribute workload according to:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#985