PeerManager failed to connect to subnet peers

[twoeths]

Describe the bug

In hetzner-c0, there is a missed attestation after I deploy master there:

Apr-20 02:54:24.422[]                ^[[31merror^[[39m: Error publishing attestations slot=3633270 Internal Server Error: PublishError.InsufficientPeers
Error: Internal Server Error: PublishError.InsufficientPeers
    at HttpClient.request (/usr/app/node_modules/@chainsafe/lodestar-api/src/client/utils/httpClient.ts:109:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at HttpClient.json (/usr/app/node_modules/@chainsafe/lodestar-api/src/client/utils/httpClient.ts:71:12)
    at Object.request [as submitPoolAttestations] (/usr/app/node_modules/@chainsafe/lodestar-api/src/client/utils/client.ts:74:19)
    at AttestationService.signAndPublishAttestations (/usr/app/node_modules/@chainsafe/lodestar-validator/src/services/attestation.ts:156:9)
    at AttestationService.runAttestationTasks (/usr/app/node_modules/@chainsafe/lodestar-validator/src/services/attestation.ts:69:5)

PeerManager is supposed to find subnet peers before validator submit subnet attestations but it failed to do that

Expected behavior

PeerManager should be able to find and connect to subnet peers before validators submit subnet attestations

[twoeths]

part of #3527

[twoeths]

it shows that some of our peers have 0 long lived subnets, we should prefer disconnecting those peers per heartbeat

also just double check that lighthouse does that

[twoeths]

grep -e "PublishError.InsufficientPeers" -rn validator*.log | wc -l returns 100 after 4 days

also there are a lot of unaggregated attestations that's sent to 1 peer
grep -e "sentPeers=1" -rn beacon*.log | wc -l returns 786

[twoeths]

the issue is not reproducable on prater

[twoeths]

The missed attestation in hetzner-c0 is very correlated to the low long-lived subnets metric

[twoeths]

#3955 and #3956 resolved the issue