nodeos_irreversible_mode_lr_test Stability Testing

[kj4ezj]

Change Description

nodeos_irreversible_mode_lr_test failure is MacOnly issue.

From EPE-389, AUTO-319, and cherry-picking commits from pull request 9551. Blockchain has asked me to prove or disprove the stability of the I have modified the CI system to run the nodeos_irreversible_mode_lr_test N times, given as:

N = ROUNDS x OS_COUNT where
OS_COUNT = number of operating systems under test
ROUNDS = A natural number as a Buildkite environment variable defining how many gated groups of test runs there shall be

These jobs are run in groups of G jobs:

G = ROUND_SIZE x OS_COUNT where
ROUND_SIZE = A natural number as a Buildkite environment variable defining how many times a test should be run on each operating system per round

This pull request has been opened to block this branch from being merged, and to share the results.

See Also

• Pull Request 7905 -- eosio-test-stability Pipeline Fixes
• Pull Request 7906 -- Concurrency Groups + Test Stability Pipeline Fixes
• Pull Request 9675 -- nodeos_irreversible_mode_lr_test Stability Testing

Change Type

Select ONE:

• Documentation
• Stability bug fix
• Other
  CI
• Other - special case

Testing Changes

Select ANY that apply:

• New Tests
• Existing Tests
• Test Framework
• CI System
  Stability Testing
• Other

Consensus Changes

• Consensus Changes
  None.

API Changes

• API Changes
  None.

Documentation Additions

• Documentation Additions
  None.

[kj4ezj]

I kicked off pinned build 25402 and unpinned build 4327, each with...

ROUNDS='150'
ROUND_SIZE='5'
SKIP_LONG_RUNNING_TESTS='false'

...for a total of 2400 runs across both builds.

Update: The pinned build failed in the first round (1/40), therefore, this test is not stable. The unpinned build is still running.

Update 2: The unpinned build failed during the 11th round (1/440).

[kj4ezj]

@heifner suggested this might be a mac-only issue, so I kicked off pinned build 25428 and unpinned build 4261 to test this on Linux only:

ROUNDS='150'
ROUND_SIZE='5'
SKIP_LONG_RUNNING_TESTS='false'
SKIP_MAC='true'

This gives a total of 2100 runs across both builds.

Update: All linux runs passed.

[kj4ezj]

Closing in favor of pull request 10460 and pull request 10465.