WIP - Reduce the Load Induced by the Polling Service

[nepet]

We had some trouble on large nodes sending out all poll messages (see #185), maybe because we tried to send them out concurrently. This PR updates the polling service to use a sequential, time-ordered model to send out the poll messages.
Therefore we add a queue which elements are time ordered and by time we mean the timestamp at which we should send the next message to the peer.

Our main loop now is tick based and checks the following every other second:

• Does the first peer in queue need a new poll msg sent out (is the timestamp past now)
• If so -> send msg and re-queue with new timestamp
• If not -> wait for next tick and check again

This way we reduce the stress that we put on the node and the network connections.
If this was the cause for the disappearing peers, this resolves #185

[wtogami]

#211 is running on Blockstream Store now. Please report if you continue to see it disappear from peerswap-listpeers. @grubles

For now don't run this yourself yet. Perhaps test this after 12 hours of success to see if it works both ways.

[wtogami]

Testing: I want to see what happens to the timing of polls if networking is delayed by 10 seconds. In that case does 10 polls happen simultaneously or does it happen one at a time separated by at least 1 second between each poll?

[grubles]

It finally disappeared about 5 minutes ago for me running 725ca2c. The patches at least seem to have extended the amount of time it takes for the node to disappear in some cases? In the past it would only take 3-4 hours to drop from peerswap-listpeers.

[wtogami]

Summary of discussion

• This reduces the chance of polls conflicting so it happens less often, but it still can disappear.
• The issue is custommessage is fire-and-forget with no delivery guarantees. Random network delays (like Tor) can mean multiple custommessage happen at the same time and somehow none are delivered due to a conflict.
• Warren thinks we didn't have the disappearing peer problem prior to Silence noisy poll log #189 because polling happened a lot more often. In that case it mattered less that some of them wouldn't be delivered.
• Short-term Fix: Poll more often (with redundant logging still silenced). Keep the WIP - Reduce the Load Induced by the Polling Service #211 approach but perhaps use a longer delay like 3 seconds?

[wtogami]

@grubles reported when he runs #211 on his node, Blockstream Store is not visible in peerswap-listpeers. It sounds like #211 as of d9d7d39 doesn't fix the original problem and somehow makes it worse when both sides are running it.

[wtogami]

Blockstream Store experienced another curious problem where an incoming swap-in attempt got stuck in state State_SwapInReceiver_AwaitTxConfirmation. The requesting node successfully ended in State_ClaimedCsv so no funds were lost. Unclear if this weird state is caused by this issue but we haven't seen this problem prior to now.

For the moment I recommend not testing #211. It needs work.

[wtogami]

• CLN devs say custommsg delivery is guaranteed unless there is a disconnect so the original purpose of WIP - Reduce the Load Induced by the Polling Service #211 turned out to be untrue. We don't think there were disconnects so we are uncertain as to why delivery was not working.
• In any case Fix Possible Polling Deadlock #213 fixed a regression introduced in Silence noisy poll log #189 solving this bug, so closing this PR.