Make client and server to resync active connections

[aruiz14]

Issue: 44576

Relates to #68
Depends on #78

Problem

remotedialer allows multiplexing connections between two peers using a single websocket connections by including a connection ID in the messages and using separate buffers. The protocol specifies different message types for different actions (Connect, Data, Error, Pause, Resume, etc.). In particular, the Error type is used to communicate the other end that a certain connection must be closed. However, depending on the cause of the original error, this message may never be successfully transmitted, as the sender will give up on sending it (#67 adds additional logging for this situation).

When this happens, one of the peers will never receive a termination message for that connection, making the underlying buffers to get stuck on Read() forever, hence causing goroutine and memory leaks.

Solution

This PR adds a new message type to the protocol (Resync), whose payload contains a list of connection IDs. Similarly to how clients sends Ping control messages, Resync messages will periodically tell the receiving peer that any connection not contained in the provided list is no longer needed and can be pruned.

Small caveat: we cannot use Control messages for this purpose, since websocket set a limit of 125 bytes for their payload, which would impose a tight restriction on the number of connections.

CheckList

• Test

[moio]

LGTM now, thanks

[tomleb]

This seems to follow what we discussed during our meeting.

Given the potential deadlock mentioned in the comments below, I think this requires more testing. It should be tested in a custom Rancher build and it would be beneficial imo to have integration tests for this, though I understand that right now remotedialer doesn't have this in place.

[maxsokolovsky]

My comments are mostly nits at this point, since I am not yet familiar with this project. I may have to come back and think about the solution itself once I learn a bit more about remotedialer.

[moio]

@maxsokolovsky do we have your final blessing to merge this, so that it has a chance to get into future 2.9.0 alphas?