Syncer stuck fix

[goran-ethernal]

Description

Problem

If for some reason, the node does not get status change events from its peers, a situation may arise where the consensus will stuck, and no blocks will be created.

For example, let's say we have a situation where we have 5 validator nodes, and they are building block 100. 4 nodes (which is quorum) send the commit messages to each other, but for some reason, only 3 of the nodes receive enough commit messages and they insert the block to their state. Because of some network issues, the other two, even though they've sent their commit messages, did not receive enough to insert the block through consensus, so they remain reliable to syncer to insert the new block. But because network was in some weird state, where some messages were not received, the syncer also on the two problematic nodes did not receive status change events from connected peers, so they missed that block through syncer as well.

Solution

The syncer has a field called block timeout (basically block time * 3) used as a way to stop the syncing from some peer if it does not respond in appropriate time. The PR uses the same field to check if we did not receive anything from some peer, and if not, it will manually ping the best peer, without the need to wait for its status change.

We have all the peers in a peer map, and that map holds the last block on each peer, so the algorithm will always choose a peer that is responsive and has the highest block number.

Changes include

• Bugfix (non-breaking change that solves an issue)
• Hotfix (change that solves an urgent issue, and requires immediate attention)
• New feature (non-breaking change that adds functionality)
• Breaking change (change that is not backwards-compatible and/or changes current functionality)

Checklist

• I have assigned this PR to myself
• I have added at least 1 reviewer
• I have added the relevant labels
• I have updated the official documentation
• I have added sufficient documentation in code

Testing

• I have tested this code with the official test suite
• I have tested this code manually