[Bug] Peers are not sufficiently connecting to each other

[vicsn]

🐛 Bug Report

It seems that our nodes are not connected as optimally as we'd like. Perhaps existing PRs (link) will improve the situation

Steps to Reproduce

Set up a network with 10 validators and 100 clients. The 10 validators should connect to each other, and each validator should connect to 10 trusted peers using --peers. We would prefer snarkos_router_connected_total to approach 19, but it is only 8.

In contrast, a local default devnet.sh run does reveal N-1 validators are connected via snarkos_router_connected_total.

Your Environment

SnarkOS commit 491f3078a.

[kpandl]

Some further observations when testing with commit b1bd3bf7457a162641b835d97adb8b6405bef6ed (branch optional_node_ip):

• In the past, the validators logged Connected to ... peers [...] not only for clients but also for validators. It seems like currently it shows only clients, or when no clients are running, it shows No connected peers
• Client connections seem to be dropped, potentially for reasons other than a PeerRefresh. One log I saw after dropping was WARN Attempted to send to a non-connected peer ...

[vicsn]

We should check that this PR did not introduce a regression: https://github.com/AleoHQ/snarkOS/pull/3163