bug: test failure ./tests/sanity/restaked.py

[pmnoxx]

Test pytest/tests/sanity/restacked.py fails,
Build got broken somewhere between b0a80d9e9a0ba0cbc008b05634e6a21de024c9d3 commit
and b0a80d9e9a0ba0cbc008b05634e6a21de024c9d3~200.

Need to write a script to automatically find the issue.

works:

> cd pytest
> git checkout b0a80d9e9a0ba0cbc008b05634e6a21de024c9d3~33; cargo build; python3 ./tests/sanity/restaked.py

ALL GOOD

doesn't

>cd pytest
>git checkout b0a80d9e9a0ba0cbc008b05634e6a21de024c9d3~32; cargo build; python3 ./tests/sanity/restaked.py

[2021-11-27 00:00:22] INFO: Starting restaked for /home/pmnoxx/.near/test0, rpc = 0.0.0.0:3041
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "RpcError { error_struct: Some(HandlerError(Object({\"name\": String(\"UNKNOWN_EPOCH\")}))), code: -32000, message: \"Server error\", data: Some(String(\"Unknown Epoch\")) }"', tools/restaked/src/main.rs:86:48
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "./tests/sanity/restaked.py", line 70, in <module>
    assert False, "Validator test0 is not in current validators after 5 epochs"
AssertionError: Validator test0 is not in current validators after 5 epochs

[pmnoxx]

git log b0a80d9e9a0ba0cbc008b05634e6a21de024c9d3~32

commit 1a2d8a4c339ea9a302bdce6de9e6f68e2b098619 (HEAD)
Author: pmnoxx <piotr.mikulski@gmail.com>
Date:   Mon Oct 4 15:29:29 2021 -0700

    Move routing table exchange to another thread (#4886)
    
    In this PR we will creating copy of routing table to another thread.

[pmnoxx]

[2021-11-27 00:17:42] INFO: Use default config {'local': True, 'near_root': '../target/debug/', 'binary_name': 'neard', 'release': False, 'bridge': {'bridge_repo': 'https://github.com/near/rainbow-bridge.git', 'bridge_dir': '~/.rainbow-bridge', 'config_dir': '~/.rainbow', 'ganache_dir': 'testing/vendor/ganache', 'ganache_bin': 'testing/vendor/ganache/node_modules/.bin/ganache-cli', 'ganache_block_prod_time': 10}}
[2021-11-27 00:17:42] INFO: Starting restaked for /home/pmnoxx/.near/test0, rpc = 0.0.0.0:3041
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "RpcError { error_struct: Some(HandlerError(Object({\"name\": String(\"UNKNOWN_EPOCH\")}))), code: -32000, message: \"Server error\", data: Some(String(\"Unknown Epoch\")) }"', tools/restaked/src/main.rs:86:48
stack backtrace:
   0: rust_begin_unwind
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:515:5
   1: core::panicking::panic_fmt
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/panicking.rs:92:14
   2: core::result::unwrap_failed
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/result.rs:1599:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/result.rs:1281:23
   4: restaked::main
             at /home/pmnoxx/repos/nearcore.master/tools/restaked/src/main.rs:86:26
   5: core::ops::function::FnOnce::call_once
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[pmnoxx]

        let validators = user.validators(None).unwrap();

[pmnoxx]

diff --git i/tools/restaked/src/main.rs w/tools/restaked/src/main.rs
index 52ed48827..228701570 100644
--- i/tools/restaked/src/main.rs
+++ w/tools/restaked/src/main.rs
@@ -1,5 +1,6 @@
 use std::path::Path;
 use std::sync::Arc;
+use std::thread::sleep;
 use std::time::Duration;
 
 use clap::{App, Arg};
@@ -81,6 +82,7 @@ fn main() {
         "Only can stake for the same account as given signer key"
     );
 
+    sleep(Duration::from_secs(5));
     let user = RpcUser::new(rpc_url, account_id.clone(), Arc::new(signer));
     loop {
         let validators = user.validators(None).unwrap();

Fixes the issue.

[pmnoxx]

Routing table calculation used to happen every time a new edge gets added. Now it happens every second. That broke the test.

[pmnoxx]

I make a fix for that issue. It seems that the test is no longer failing. See branch piotr-fix-updating-routing-table-after-adding-edge.

L .~/r/r/t/nayduck master (1/0/0) RUST_BACKTRACE=1 cargo run nayduck --regex "restaked.py" --num-tests-limit 1 --locally --build --run --repeat=10 --continue-on-failure

2021-12-06T03:36:15.633915Z  INFO nayduck::nayduck: branch="piotr-fix-updating-routing-table-after-adding-edge" 
2021-12-06T03:36:15.633983Z  INFO nayduck::nayduck: success=10 failure=0

Here are stats from master:

RUST_BACKTRACE=1 cargo run nayduck --regex "restaked.py" --num-tests-limit 1 --locally --build --run --repeat=10 --continue-on-failure

2021-12-06T03:08:16.306735Z INFO nayduck::nayduck: branch="master" 
2021-12-06T03:08:16.306775Z  INFO nayduck::nayduck: cmd=["tests/sanity/restaked.py"]
2021-12-06T03:08:16.306793Z  INFO nayduck::nayduck: success=3 failure=7

[pmnoxx]

I scheduled 100 runs on nayduck http://nayduck.near.org/#/run/2291 with command

RUST_BACKTRACE=1 cargo run nayduck --regex "restaked.py" --num-tests-limit 1 --remotely --build --run --repeat=100 --continue-on-failure

[pmnoxx]

Status | FAILED:7 / PASSED:93

[mina86]

Status | FAILED:7 / PASSED:93

I am surprised you got 93% success to be honest. I get much worse results.

I actually have a fix for the Unknown Epoch failure: #5775