feat(event-streaming): new streams (orders, swaps, eth_fee_estimation)

[mariocynicys]

This PR aims to support event streaming in some of MM2's API endpoints.

The work in this PR till now is only a refactor of the event streaming code after multiple iterations of discussing how this should end up looking like.

The flow goes as follows:

• A client initially opens an SSE connection with MM2 to receive events on. For wasm, since it can't support SSEs, we use a worker to deliver the events to the client (let's consider it as SSE too).
• If a client wants to receive some events (balance change, network topology change, new gas estimations, etc...), they need to subscribe to these events (enable these streamers) from the API.
• A streamer is a background task that is responsible for firing some events. Initially MM2 doesn't have any streamers running. When a client enables a streamer (= subscribes to a particular event), they start receiving events from that streamer. Until they unsubscribe to them.
• If no client is listening to some active streamer, it will shutdown. A new streamer will start again if a client re-subscribes.
• (little irrelevant now) Multiple clients subscribing to the same event on MM2 doesn't mean we will have a new streamer for every client. Only one streamer is booted up for the first subscribed client and new subscribing clients will be sharing the same streamer.

The role of the StreamingManager:

• Manage different clients (each client has an id cid when it first connects through SSE).
• Add a new (cid, sid) pair (translates to: client with id cid wants to listen to the streamer with id sid). If such a streamer with sid is already running, it is just instructed that a new client is gonna be listening along, otherwise, it will be spawned in the background.
• Stop listening (cid, sid) pair (= client with cid no longer wants to receive events from streamer with id sid, if no more clients are listening, the streamer will die).
• Manage different streamers.
• Manage the connections between the streamers (the background thread) and MM2. Streamers either work in a periodic manner (do some job and possibly fire some events then sleep for a while and repeat) or they need to receive a notification from MM2 (e.g. UTXO balance event streaming rely on electrum subscriptions: the electrum server sends a notification to MM2 when an address balance has changed and MM2 in turn sends this notification to the streamer responsible for firing balance events). To send some data to a streamer, StreamingManager::send(streamer_id, arbitrary_data) is used.

Breaking Changes:

• No more event_stream_configuration in MM2 json config. Requested streams are initialized and configured dynamically through the API.
• Thus event_stream_configuration.access_control_allow_origin config was moved to access_control_allow_origin (one scope out) b9d1218.
• And event_stream_configuration.worker_path was changed to event_stream_worker_path (one scope out) ee32fd1.
• No more filters are required in the initial SSE setup with MM2. (e.g. mm2.com/event-stream?filter=NETWORK,BALANCE,DATA_NEEDED:datatype -> mm2.com/event-stream?id=1), but a client id is now required (a random u64 identifier for [& generated by] the client) (e.g. mm2.com/event-stream?id=799384531)
• How to filter the wanted events then (client gets no events by default)? The client needs to activate specific event streamers through RPC (e.g. stream::enable::balance).
• Contrary to the note in the previous point (client gets no events by default), some special events are now delivered to the client even if they never requested/filtered them (this happens using broadcast_all). example of these events are DATA_NEEDED:datatype, and HEARTBEAT should be similar later as well.

(so for keplr integration, only ?id=<RANDOM_U64> must be added to the SSE init request and everything should work. ?filter= shall be removed)
(the <RANDOM_U64> should be stored by the client since it's required in any event streamer activation/deactivation. If for keplr only though, this isn't necessary for now since DATA_NEEDED:datatype event streamer is a special event that's broadcasted to all clients and doesn't require any sort of activation/subscription before hand)

[onur-ozkan]

I'm not sure if I mentioned this before (since my last review was a month ago), but why are we implementing this at the trait level? Coins don't create MmArc instances—they're already available at higher levels and some coins just receive them during initialization. We could likely pass MmArc where needed, as we do for certain coins. Adding this function seems unnecessary and inconsistent across different coins.

[mariocynicys]

needed in MmCoin so we can access it in generic impls over MmCoins, e.g.:

komodo-defi-framework/mm2src/coins/utxo/utxo_tx_history_v2.rs

Line 593 in 547fc2b

let streaming_manager = ctx.coin.get_ctx().unwrap().event_stream_manager.clone();

i think i have answered that somewhere in the threads above though, could you please go over it and resolve addressed comments.

p.s. in the e.g. above, we could add it in the UtxoCoinOps instead, but this completely makes no sense IMO. We could also opt for making it get_streaming_manager but i opted for get_ctx since we already have global states other than just the streaming manager and some coins already accesses this global state.

[onur-ozkan]

Why it's not in

komodo-defi-framework/mm2src/coins/utxo/utxo_tx_history_v2.rs

Lines 145 to 152 in 547fc2b

	struct UtxoTxHistoryStateMachine<Coin: UtxoTxHistoryOps, Storage: TxHistoryStorage> {
	coin: Coin,
	storage: Storage,
	metrics: MetricsArc,
	/// Last requested balances of the activated coin's addresses.
	/// TODO add a `CoinBalanceState` structure and replace [`HashMap<String, BigDecimal>`] everywhere.
	balances: HashMap<String, BigDecimal>,
	}

?

[mariocynicys]

that's a good place as well. im down to storing it (streaming manager) there if u are keen on the coins to expose no ctx :(

[onur-ozkan]

I'd prefer storing it where it's needed rather than forcing it into an unrelated trait.

[mariocynicys]

into an unrelated trait

gonna have to disagree with that, but guess that's a topic for another day :)

addressed in 65bd662 & e65ace0