fix: Removing flakiness from staking tests #3150
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The flakiness was primarily due to insufficient number of blocks between the epoch switch, and checking for the expected stakes. The epoch was switching at heights equal to 2 modulo 10, and the checks were executed at the same heights. Moreover, the height was retrieved from one node, but the checks were against the other. Fixing it by extending the epoch by two blocks, and moving the checks two blocks ahead. Also adding checks before sending the second set of stakes, it covers a situation when the latest stake during the current epoch is higher than the current stake (a situation the test was not previously testing). Reducing the timeout to 6 minutes from 10 minutes. staking2.py, staking_repro1.py and staking_repro2.py mostly test the same things, so it is still 18 minutes of testing per run. All the failures in the observable past were occurring much earlier than during the first 6 minutes of the test (e.g. see failures here: http://nayduck.eastus.cloudapp.azure.com:3000/#/run/67). Test plan: ---------- No failures in 150 runs: http://nayduck.eastus.cloudapp.azure.com:3000/#/run/69
|
|
Codecov Report
@@ Coverage Diff @@
## master #3150 +/- ##
=======================================
Coverage 87.35% 87.35%
=======================================
Files 212 212
Lines 41525 41525
=======================================
Hits 36274 36274
Misses 5251 5251 Continue to review full report at Codecov.
|
bowenwang1996
approved these changes
Aug 12, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The flakiness was primarily due to insufficient number of blocks between the epoch switch, and checking for the expected stakes. The epoch was switching at heights equal to 2 modulo 10, and the checks were executed at the same heights. Moreover, the height was retrieved from one node, but the checks were against the other.
Fixing it by extending the epoch by two blocks, and moving the checks two blocks ahead.
Also adding checks before sending the second set of stakes, it covers a situation when the latest stake during the current epoch is higher than the current stake (a situation the test was not previously testing).
Reducing the timeout to 6 minutes from 10 minutes. staking2.py, staking_repro1.py and staking_repro2.py mostly test the same things, so it is still 18 minutes of testing per run. All the failures in the observable past were occurring much earlier than during the first 6 minutes of the test (e.g. see failures here: http://nayduck.eastus.cloudapp.azure.com:3000/#/run/67).
Fixes #3128
Test plan:
No failures in 150 runs:
http://nayduck.eastus.cloudapp.azure.com:3000/#/run/69