Reliable benchmarking in noisy environments #264
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mizdra
force-pushed
the
stabilize-benchmark
branch
2 times, most recently
from
03862d9
to
d0d9de3
Compare
mizdra
force-pushed
the
stabilize-benchmark
branch
2 times, most recently
from
32e0e01
to
1c70d08
Compare
mizdra
force-pushed
the
stabilize-benchmark
branch
from
1c70d08
to
111736a
Compare
mizdra
force-pushed
the
stabilize-benchmark
branch
from
2ce12fa
to
61ead05
Compare
mizdra
force-pushed
the
stabilize-benchmark
branch
from
61ead05
to
b751cfd
Compare
mizdra
force-pushed
the
stabilize-benchmark
branch
5 times, most recently
from
62a5373
to
11da893
Compare
mizdra
force-pushed
the
stabilize-benchmark
branch
from
11da893
to
8c09f8e
Compare
mizdra
changed the title
mizdra
commented
Feb 11, 2023
Comment on lines
+102
to
+104
| # There is 3.6% noise (maximum score is 3.68% greater than minimum score) for 30 runs. | ||
| # This noise is a little loud but negligible in most situations. | ||
| # ref: https://github.com/mizdra/eslint-interactive/commit/b751cfdef788ac6eb6b39d2d015494123cae51c1#comments |
There was a problem hiding this comment.
This is a much larger value than the noise presented at https://pythonspeed.com/articles/consistent-benchmarking-in-ci/.
The detailed cause is unknown, but it is probably related to the fact that the process performed by eslint-interactive is dominated by I/O to the file system. valgrind does not virtualize I/O to the file system, so it is strongly affected by the CI noise.
There was a problem hiding this comment.
But, previous benchmarks showed 300% noise. Given that, that's a huge improvement!
There was a problem hiding this comment.
I rewrote the code to cause performance degradation as a test.
- diff
- before score:
21065268522instructions - after score:
33588677349instructions
The score increased by 159.4 %. This satisfies the conditions for an alert to be triggered.
mizdra
force-pushed
the
stabilize-benchmark
branch
from
7f2884a
to
119db02
Compare
mizdra
force-pushed
the
stabilize-benchmark
branch
from
119db02
to
65ac695
Compare
This reverts commit 65ac695.
|
Perfect! 💯💯💯 |
This was referenced Feb 11, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
close: #258