Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(event streaming): implement push-only streaming channels and SSE #1945

Merged
merged 21 commits into from
Sep 28, 2023

Conversation

onur-ozkan
Copy link
Member

@onur-ozkan onur-ozkan commented Aug 23, 2023

This implements streaming channels using mpsc(underlying part of SSE) and SSE for sending data to clients continiously. Currently, only NETWORK event is implemented, further events will be added after the collaboration with the GUI team.

Supported platforms: all other than WASM(Simply because we can't serve a server on browser. I have a solution idea to this, but it's too early to talk about).

Enabling Event Streaming:

  1. To enable event streaming, add the event_stream_configuration property to the mm2 configuration. Use the following sample values:
{
	"gui": "core_readme",
	"netid": 7777,
	// ... other parts
	"event_stream_configuration": {
		"access_control_allow_origin": "*",
		"active_events": {
			"NETWORK": { "stream_interval_seconds": 0.5 }
		}
	}
}

2- Start mm2 and wait for the initialization log message similar to: mm2_net::network_event:54] INFO NETWORK event is activated with 0.5 seconds interval.

3- Open examples/sse/index.html in your browser; events should start appearing on the screen.

Client-Side Filtering:

By default, clients listen to all events activated on the mm2 side. If you're interested in specific events, you can filter incoming events (using the filter query string parameter) to reduce network overhead.

For instance, open examples/sse/index.html with a text editor and append ?filter=TEST,DEMO,BALANCES to the EventSource url. After that, you will no longer receive NETWORK events. (Note that currently we only stream NETWORK events on mm2 side. Filtering out the NETWORK event will result in no events being received.)

Helps to #1901

Signed-off-by: ozkanonur <work@onurozkan.dev>
Signed-off-by: ozkanonur <work@onurozkan.dev>
@onur-ozkan onur-ozkan added the in progress Changes will be made from the author label Aug 23, 2023
Signed-off-by: ozkanonur <work@onurozkan.dev>
Signed-off-by: ozkanonur <work@onurozkan.dev>
Signed-off-by: ozkanonur <work@onurozkan.dev>
Signed-off-by: ozkanonur <work@onurozkan.dev>
Signed-off-by: ozkanonur <work@onurozkan.dev>
Signed-off-by: ozkanonur <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Copy link
Member Author

@onur-ozkan onur-ozkan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These notes are not related with this work. They don't seem right in general. Having TODOs will help us tracking them later.

@@ -1,3 +1,5 @@
// TODO: a lof of these implementations should be handled in `mm2_net`
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

☝️

Comment on lines +307 to +308
// TODO: This should exclude TCP internals, as including them results in having to
// handle various protocols within this function.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

☝️

@onur-ozkan onur-ozkan added under review and removed in progress Changes will be made from the author labels Aug 29, 2023
@onur-ozkan onur-ozkan marked this pull request as ready for review August 29, 2023 10:51
@onur-ozkan onur-ozkan added the enhancement New feature or request label Aug 29, 2023
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Copy link
Collaborator

@shamardy shamardy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great Work! First review iteration!

mm2src/mm2_net/src/sse_handler.rs Outdated Show resolved Hide resolved
mm2src/mm2_event_stream/src/controller.rs Show resolved Hide resolved
mm2src/mm2_event_stream/src/lib.rs Outdated Show resolved Hide resolved
Signed-off-by: onur-ozkan <work@onurozkan.dev>
@shamardy shamardy requested a review from laruh September 11, 2023 10:05
shamardy
shamardy previously approved these changes Sep 20, 2023
Copy link
Collaborator

@shamardy shamardy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approve

@shamardy
Copy link
Collaborator

@KomodoPlatform/qa: Refer to this comment #1945 (comment) to see how you can test this new event streaming feature. The above comment is also a starting point for documenting this feature, if you wish to begin documentation at this early stage. If you have any questions, please direct them to @onur-ozkan

@smk762
Copy link

smk762 commented Sep 22, 2023

Open examples/sse/index.html

I assume this means http://127.0.0.1:7783/examples/sse/index.html?
Response is {"error":"EOF while parsing a value at line 1 column 0"}

and in mm2.log 22 07:03:25, mm2_main::mm2::rpc:286] ERROR RPC error response: EOF while parsing a value at line 1 column 0

@onur-ozkan
Copy link
Member Author

onur-ozkan commented Sep 22, 2023

I assume this means http://127.0.0.1:7783/examples/sse/index.html?

No, it's independent HTML page(for testing SSE connection), you just need to open it in browser. You can serve using any web server like using python SimpleHTTPServer(see https://github.com/KomodoPlatform/komodo-defi-framework/blob/push-only-streaming-channels/examples/sse/README.md#listening-event-stream-from-komodo-defi-framework)
), nodejs http-server, nginx. Or you can open it using the filesystem directly like:

image

Copy link

@smk762 smk762 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirm that event steam appears in browser from examples/sse/index.html, and modifying index.html to add a filter works.

Is there a plan to allow the filter to be applied via browser url in future? E.g. examples/sse/index.html?filter=BALANCES instead of editing index.html

@onur-ozkan
Copy link
Member Author

Confirm that event steam appears in browser from examples/sse/index.html, and modifying index.html to add a filter works.

Is there a plan to allow the filter to be applied via browser url in future? E.g. examples/sse/index.html?filter=BALANCES instead of editing index.html

GUI can already achieve that by using URL parameters(e.g., https://developer.mozilla.org/en-US/docs/Web/API/Window/location) in the SSE connection string. mm2 can't know what page it's being used for.

Copy link

@kivqa kivqa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved.
Event steam displayed for following places :
http://localhost:7783/event-stream
komodo-defi-framework/examples/sse/index.html
http://127.0.0.1:8000/
image
Filter works as expected: for ?filter=NETWORK events are displayed, for other values of filter event stream is empty.

@shamardy shamardy merged commit e9559c2 into dev Sep 28, 2023
26 of 32 checks passed
@shamardy shamardy deleted the push-only-streaming-channels branch September 28, 2023 16:46
@smk762 smk762 removed the docs label Nov 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants