Notify web-socket event listeners independently

[rymsha]

Currently websokets use RemoteEndpoint.Basic (blocking ) sendText and iterate over all clients.

First improvement we can make is to send events in parallel independently to all web-socket clients - so we don't sum up waiting time.

Also need to look at all possible timeout settings for websokets.

[rymsha]

some things to read up
jetty/jetty.project#847
jetty/jetty.project#2061

[rymsha]

jetty/jetty.project#1410

[rymsha]

Websockets definitely limit throughput of events pipeline.
They are the "greedy": all kinds/types of messages are sent. (Other listeners skip unwanted events quickly)
"Slow": ~20ms per message due to RTT. (50 events per second max. Period)
Unreliable: clients may disappear ( WiFi network change)
Substantial: one server can have hundreds of simultaneous clients connected. Each treated as own event listener.

Solutions:
"Greedy" can be mitigated by subscriptions to certain wanted types. (Breaking change)

"Slow" RTT usually mitigated by batching. (Breaking change)

I mostly focused on the "substantial" part. Tried to send events to different clients in parallel.

The fact that they are "unreliable" is probably the worst, can only be dealt with timeout fine-tunings

[rymsha]

• check that one thread can saturate a socket. YES.
• check how many sockets browser opens per tab: each CS tab creates new websocket.
• check when exactly async "transmitted" callback gets called (before or after physical delivery?)
• socket close if max exceeded.
• configurable max events in queue per websocket. Default 10_000? (will be not configurable 100_000)
• task for frontend to reconnect No error message when there's no connection to Websockets app-contentstudio#1723
• configurable idle timeout

[rymsha]

I tested network outage (cable unplug): in blocking mode jetty keeps reporting messages sent, so from jetty point of view "transmitted" actually means "network layer accepted packets".
Wireshark was showing packets sent, nothing received back, a few retransmit attempts and finally (thanks to router) FIN, PSH, ACK in reply - that forcibly closes connection.

[rymsha]

a new discussion about websocket async behavior in jetty just started jetty/jetty.project#4824

[rymsha]

• https://bugs.eclipse.org/bugs/show_bug.cgi?id=474488 states that sync sendText is for multi-threading (still correct these days. It throws java.lang.IllegalStateException: Blocking message pending 10000 for BLOCKING).
• in sync mode (it must be single threaded) one slow client can block the thread (when TCP buffers are full) and so, bock entire event flow.
• in async mode it may still happen but max events queue limit quickly swaps away slow clients. (unfortunately not so slow clients may be swapped away as well, but it is better than to keep entire event flow stuck)
• in addition to that async methods can be called from different threads, which reduces the chances of slow clients affecting others. (but if most clients are slow event flow gets stuck again)
• another unfortunate is that javax.websocket api does not provide disconnect() method for the socket (close() is a way too graceful - it is just yet another message that will stuck in buffers). Jetty eventually times out those finally reclaiming memory (see next note)
• there is another knob which helps: WebSocketPolicy.idleTimeout. But its default value is quite high (5 minutes)