fix: peer connection to stale nodes (tari-project#5579)

Description
---
This fixes nodes continuously selecting bad stale nodes as connections
and attempting to connect to them.

Motivation and Context
---
If nodes have bad nodes in their peer database they should not keep
trying to connect to them.
This will attempt to connect to a node once, and if the node was never
online and it tried to connect to the node once, it will ignore the node
till the node tries to connect to it first.

This stops the problem with if a node has too many stale connections in
its peer database, it will keep selecting stale nodes and try to connect
to them. If the amount of stale nodes are too much, it can cause a
constant read/write on the peer database of the node.

How Has This Been Tested?
---
Manually verified with database with mostly stale nodes. Node will try
them once, then ignore them.

comms/core/src/peer_manager/peer.rs

@@ -158,6 +158,23 @@ impl Peer {
         &self.supported_protocols
     }
 
+    pub fn last_connect_attempt(&self) -> Option<NaiveDateTime> {
+        let mut last_connected_attempt = None;
+        for address in self.addresses.addresses() {
+            if let Some(address_time) = address.last_attempted {
+                match last_connected_attempt {
+                    Some(last_time) => {
+                        if last_time < address_time {
+                            last_connected_attempt = address.last_attempted
+                        }
+                    },
+                    None => last_connected_attempt = address.last_attempted,
+                }
+            }
+        }
+        last_connected_attempt
+    }
+
     /// Returns true if the peer is marked as offline
     pub fn is_offline(&self) -> bool {
         self.addresses.offline_at().is_some()

comms/dht/src/connectivity/mod.rs

@@ -793,6 +793,10 @@ impl DhtConnectivity {
                 {
                     return false;
                 }
+                // we have tried to connect to this peer, and we have never made a successful attempt at connection
+                if peer.offline_since().is_none() && peer.last_connect_attempt().is_some() {
+                    return false;
+                }
 
                 let is_excluded = excluded.contains(&peer.node_id);
                 if is_excluded {