k6 v0.42.0 updates - results (4/5) #24
Conversation
Command ran: ./k6bench.sh --vus=6000 --duration=10m -e SLEEP=5 scripts/website.js \ | tee results/v0.42.0/m5.large-website.js-10s.csv To generate the graph (from the results/v0.42.0/ directory, using gnuplot 5.4 patchlevel 5): gnuplot -e 'k6version="v0.42.0"; ec2instance="m5.large"; script="website.js-10s"' ../../k6bench.gnuplot
Command ran: ./k6bench.sh --vus=20000 --duration=10m -e SLEEP=3 scripts/website.js \ | tee results/v0.42.0/m5.4xlarge-website.js-6s.csv To generate the graph (from the results/v0.42.0/ directory, using gnuplot 5.4 patchlevel 5): gnuplot -e 'k6version="v0.42.0"; ec2instance="m5.4xlarge"; script="website.js-6s"' ../../k6bench.gnuplot
Command ran: ./k6bench.sh --vus=30000 --duration=5m -e SLEEP=0.5 scripts/website.js \ | tee results/v0.42.0/m5.24xlarge-website.js-1s.csv To generate the graph (from the results/v0.42.0/ directory, using gnuplot 5.4 patchlevel 5): gnuplot -e 'k6version="v0.42.0"; ec2instance="m5.24xlarge"; script="website.js-1s"' ../../k6bench.gnuplot
Command ran: ./k6bench.sh --vus=60000 --duration=1m scripts/RPS-optimized.js \ | tee results/v0.42.0/m5.24xlarge-RPS-optimized.js.csv To generate the graph (from the results/v0.42.0/ directory, using gnuplot 5.4 patchlevel 5): gnuplot -e 'k6version="v0.42.0"; ec2instance="m5.24xlarge"; script="RPS-optimized.js"' ../../k6bench.gnuplot
Command ran: ./k6bench.sh --vus=1000 --duration=1m scripts/file-upload.js \ | tee results/v0.42.0/m5.24xlarge-file-upload.js.csv To generate the graph (from the results/v0.42.0/ directory, using gnuplot 5.4 patchlevel 5): gnuplot -e 'k6version="v0.42.0"; ec2instance="m5.24xlarge"; script="file-upload.js"' ../../k6bench.gnuplot To generate the 26MB.zip file (from the scripts/ directory): dd if=/dev/urandom of=26MB bs=1M count=25; zip 26MB 26MB; rm -f 26MB
This was assembled manually using the following commands:
To calculate the average values (column 2 is for CPU, column 3 for RAM):
awk -F',' 'NR>1 { sum += $3 } END { printf "%.2f", sum / (NR-1) }' m5.large-website.js-10s.csv
To calculate the max values (column 4 is for VUs, column 5 for RPS):
awk -F',' 'BEGIN { max = 0 } NR>1 { max = ($5>max ? $5 : max) } END { printf "%.2f", max }' m5.large-website.js-10s.csv
The network throughput and data transferred were retrieved from the k6
end-of-test summary.
| @@ -0,0 +1,1903 @@ | |||
|
|
|||
There was a problem hiding this comment.
shouldn't the name be 10m instead of 10s
There was a problem hiding this comment.
That suffix indicates the sleep() period per iteration, which is a more distinguishing factor for some of these tests. Notice that we also refer to it in the graph titles, and the README sections. The actual test duration is not that important.
| - VUs reached (max): 60,000 | ||
| - Memory used (avg): 21.65 GB | ||
| - CPU load (avg): 72.03 (out of 96.0) | ||
| - RPS (max): 161,651.73 |
There was a problem hiding this comment.
🤔 accoarding the current article it looks like degradation (there it stays Peak RPS: ~188.500.). Is it because we adjusted the test script? #21 (comment)
There was a problem hiding this comment.
Yes. As I explained there, the test needed to be changed because of SUT differences, so we can't determine whether it's a performance degradation or not compared to v0.26.2. Considering the other results, I don't think there is a degradation, but it's important that we keep the same test variables moving forward, otherwise we can't reliably compare results between test runs.
There was a problem hiding this comment.
The non-blocking thing, maybe we could measure the results for the current scripts (and current SUT) in k6 v0.26.2 or any other from the past.
This is PR 4/5, part of updating the "Running large tests" article and running benchmarks on k6 v0.42.0.
These changes add the results themselves, and a short summary. See the commits for the commands used in each case.
The idea is that all of this, including the summary, would eventually be fully automated.
PR order: #21 -> #22 -> #23 -> #24 -> #25 (review and merge in reverse)