Support Stability Testing for Tests + CI Bug Fixes

[kj4ezj]

Change Description

While working on service item AUTO-1004 to provide a low-effort stability testing solution to Blockchain for their unit and integration tests, I found a way to satisfy AUTO-495 so that the pipeline upload script natively supports stability testing within the organizational Buildkite job limit.

This pull request introduces the following:

• Restores the ability to run the pipeline upload script locally to see what YAML it generates in your shell or editor of choice
• Fixes the code that prevents pipeline documentation from being uploaded when the pipeline upload step is retried
• Fixes the alignment of an fi statement
• Alphabetizes the serial and long-running tests by test name so it is easier to find tests in the list of jobs
• Fixes null SKIP variable bug
• Removes extraneous newlines
• Requires explicit action from the user to run all test suites more than once

Additionally, stability testing of EOSIO tests is supported natively by the Buildkite pipelines, without having to edit the pipeline upload script or cherry-pick code from pull request 9675.

---

Instructions

Stability testing of EOSIO unit and integration tests is done in the eosio-test-stability pipeline. It will take thousands of runs of any given test to identify it as "stable" or "unstable". Runs should be split evenly across "pinned" (fixed dependency version) and "unpinned" (default dependency version) builds because, sometimes, test instability is only expressed in one of these environments. Finally, stability testing should be performed on the Linux fleet first because this fleet is effectively infinite. Once stability is demonstrated on Linux, testing can be performed on the finite macOS Anka fleet.

The eosio-test-stability pipeline uses the same pipeline upload script as eosio, eosio-build-unpinned, and eosio-lrt, so all variables from the pipeline documentation apply. However, there are five primary environment variables relevant to stability testing:

PINNED='true|false'    # whether to perform the test with pinned dependencies, or default dependencies
ROUNDS='ℕ'             # natural number defining the number of gated rounds of tests to generate
ROUND_SIZE='ℕ'         # number of test steps to generate per operating system, per round
SKIP_MAC='true|false'  # conserve finite macOS Anka agents by excluding them from your testing
TEST='name'            # PCRE expression defining the tests to run, preceded by '^' and followed by '$'
TIMEOUT='ℕ'            # set timeout in minutes for all Buildkite steps

The TEST variable is parsed as pearl-compatible regular expression where the expression in TEST is preceded by ^ and followed by $. To specify one test, set TEST equal to the test name (e.g. TEST='read_only_query'). Specify two tests as TEST='(nodeos_short_fork_take_over_lr_test|read_only_query)'. Or, perhaps, you want all of the restart_scenarios tests. Then, you could define TEST='restart-scenario-test-.*' and Buildkite will generate ROUND_SIZE steps each round for each operating system for all three restart scenarios tests.

The number of total test runs will be:

RUNS = ROUNDS * ROUND_SIZE * OS_COUNT * TEST_COUNT # where:
OS_COUNT   = 'ℕ' # the number of supported operating systems
TEST_COUNT = 'ℕ' # the number of tests matching the PCRE filter in TEST

We recommend stability testing one test per build with two builds per test, on Linux at first. Kick off one pinned build on Linux...

PINNED='true'
ROUNDS='42'
ROUND_SIZE'5'
SKIP_MAC='true'
TEST='read_only_query'

...and one unpinned build on Linux:

PINNED='true'
ROUNDS='42'
ROUND_SIZE'5'
SKIP_MAC='true'
TEST='read_only_query'

Once the Linux runs have proven stable, and if instability was observed on macOS, kick off two equivalent builds on macOS instead of Linux. One pinned build on macOS...

PINNED='true'
ROUNDS='42'
ROUND_SIZE'5'
SKIP_LINUX='true'
SKIP_MAC='false'
TEST='read_only_query'

...and one unpinned build on macOS:

PINNED='true'
ROUNDS='42'
ROUND_SIZE'5'
SKIP_LINUX='true'
SKIP_MAC='false'
TEST='read_only_query'

If these runs are against eos:develop and develop has five supported operating systems, this pattern would consist of 2,100 runs per test across all four builds. If the runs are against eos:release/2.1.x which, at the time of this writing, supports eight operating systems, this pattern would consist of 3,360 runs per test across all four builds. This gives you strong confidence that any test instability occurs less than 1% of the time.

---

See Also

• eos
  • Pull Request 9675 -- nodeos_irreversible_mode_lr_test Stability Testing
  • Pull Request 10446 -- Stability Test nodeos_short_fork_take_over_lr_test
  • Pull Request 10460 -- Support Stability Testing for Tests + CI Bug Fixes (eos:develop)
  • Pull Request 10463 -- Support Stability Testing for Tests + CI Bug Fixes (eos:release/2.1.x)
  • Pull Request 10464 -- Support Stability Testing for Tests + CI Bug Fixes (eos:release/2.0.x)
  • Pull Request 10465 -- Bug Fixes for the eosio-test-stability Pipeline (eos:develop)
  • Pull Request 10467 -- Another Bug Fix for the eosio-test-stability Pipeline (eos:develop)
  • Pull Request 10468 -- Another Bug Fix for the eosio-test-stability Pipeline (eos:release/2.1.x)
  • Pull Request 10469 -- Another Bug Fix for the eosio-test-stability Pipeline (eos:release/2.0.x)
• eos-buildkite-pipelines
  • Commit 50cd1ca -- Modernize eosio-test-stability pipeline
  • Commit 326177f -- Reduce number of default rounds so we don't exceed the org-wide Buildkite job limit
  • Commit 8e33ef4 -- Skip macOS by default

Change Type

Select ONE:

• Documentation
• Stability bug fix
• Other
• Other - special case

Support stability testing of EOSIO tests.

Testing Changes

Select ANY that apply:

• New Tests
• Existing Tests
• Test Framework
• CI System
• Other

Support stability testing of EOSIO tests, plus other bug fixes to the pipeline upload script.

Consensus Changes

• Consensus Changes

None.

API Changes

• API Changes

None.

Documentation Additions

• Documentation Additions

None.