A gossip spy node will OOM if zeros are shoveled at it.

[mvines]

STR:

1. $ solana-gossip spy --gossip-port 1234
2. $ dd if=/dev/zero bs=1232 > /dev/udp/127.0.0.1/1234

Within a minute or two, the spy node will be killed by the kernel.

cc: #5414

[sagar-solana]

I spent some time debugging this.

At first it looked like the calls to "verify" the incoming gossip messages were causing the recycler to blow up. After further inspection, it became clear that if the channel's consumer take even a few nanoseconds the socket receiver is able to collect items at a much higher rate.
This leads to the Recycler making new allocations instead of recycling old ones.

It is not a memory leak, just a resource exhaustion where the channel's consumer isn't fast enough.

Suggested fix:
In Gossip, perform a greedy receive on the receiver. If the number of items exceeds some computed limit (number of nodes * expected messages per node * expected messages per second) use stakes to drop lower staked messages. If no stakes are known, drop items at random.

While that approach seems simple enough, it needs to be efficient otherwise there will be no improvement. Figuring out the stakes and which ones are "lower" will require 2 passes over the incoming packets or will need cache some of the data possibly increasing memory consumption (by a negligible amount).