How to consume server-streams with JavaScript inside a Browser

[maja42]

📚 Documentation / Question

Our UI-team struggles implementing grpc server-streams provided via the gateway.

Our use case:
We want to use backend-streams as an alternative to polling:
The REST/stream call is started as soon as the website is opened and stays active until the tab/browser is closed.
The backend sends changes regarding certain topics on this stream as soon as they appear.

As an example, consider this protobuf definition:

message RandomStreamResponse {
    string randomResponse = 1;
}
service TestService {
    rpc RandomStreamResponses (google.protobuf.Empty) returns (stream RandomStreamResponse) {
        option (google.api.http) = {
            get: "/test/randomStream"
        };
    }
}

Each message is encoded as json and sent to the browser in a separate line of the response body.
Within JavaScript, it's then possible to read partial responses from the backend.

However, there is one issue:
Everytime JavaScript reads the (partial) response body after a new backend-message, it receives the whole response, not only the new line/message.
It seems that there is no way of telling the browser "Hey, I already read that. Please delete it and don't bother me anymore".

While this might look like a minor inconvenience at first (after all, JavaScript could remember the bytes-already-read and ignore them), it makes our use-case pretty tricky.

If the stream is active for a long time and/or a lot of messages are sent by the backend and/or if the individual response messages contain big json-objects, a lot of unneccesary data will accumulate and fill up the user's RAM.

I couldn't find any examples or documentation that shows how those streams should be consumed from the browser/JavaScript side. Is there a solution to this problem?
I'd like to avoid aborting and repeating the REST-call every few minutes after some arbitrary amount of data was received.

If this is not possible - is there a way to provide these grpc-streams via web sockets? Or, even better, via SSE (server-side-events)? What would be the correct solution for our use-case?

[johanbrandhorst]

Hi again Maja and thank you for your detailed issue. We don't have any documentation around this already but as it happens I have had some personal experience implementing this. My recommendations are as follows:

1. Use the Fetch API to initiate the request to the backend. If you use XHR, it will buffer the entire response in memory, whereas IIRC the Fetch API lets you do true streaming.
2. Chrome has a limit of 256MiB in a single streaming XHR call or something like that. Be wary of long running connections that may get close to this limit if you stay with XHR for compatibility.
3. I think we ended up implementing a catchup mechanic in the backend when we were doing this, so that the user could reload the page and have all the results they missed fed back before the streaming starts again. If your use case doesn't require this, avoid it. It adds a lot of complexity to the backend.

If you end up getting this working, I'd love to add a new section about this to the docs. Let us know!

[maja42]

Thanks for your quick response.
The Fetch-API looks promising, I didn't know it existed. Compatibility should not be an issue - for us at least.
Let's see what our UI-guys come up with, and I'll report back if it worked out.

Regarding 3:
We solved (or plan to solve) it in a different way. Our protobuf-grpc-APIs are structured in such a way, that the first response of a server-stream is always the (full) current state - the client does not maintain cache. And all subsequent messages are partial-updates to that initial state, or(if partial-updates are not possible) the new state.

A downside of this approach (in contrast to websockets) is that each request requires an active connection to our backend, consuming a server port. So with multiple clients having multiple open streams, we might run into a port shortage at some time. But I guess with HTTP/2 that might not be an issue.

[johanbrandhorst]

I forgot to mention, you can also bridge the grpc-gateway with websockets using https://github.com/tmc/grpc-websocket-proxy. I've used it in the past and it works well. It might be preferred to a Fetch API stream.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[maja42]

Okay, I finally found the time to get back to this problem.
Unfortunately we have the issue that we need many streams running at once (they will mostly be idle and only receive server-messages sporadically).
With the way streams are implemented in the grpc-gateway right now, every stream consumes its own connection.

This is a problem. The max. number of simultaneous connections a browser allows is quite limited: https://stackoverflow.com/a/985704/2224996

In a POC, we only managed to open 5 streams at once (when using Chrome). Using the Fetch API doesn't help with this issue, since fetch() also count towards this limit: https://developers.cloudflare.com/workers/platform/limits

Did you also face this issue and, if so, how did you solve it?
I'm not really sure if I like the idea of putting grpc-websocket-proxy on top of the already-big grpc-gateway. It sound's like an overkill. But in it's current state, the gateway's streaming capabilities are completely insufficient :(

[johanbrandhorst]

I don't think there's anything the grpc-gateway can really do to help you here. We're not going to implement some sort of multiplexing of connections (nevermind how we'd actually do that). If you want some sort of notifications, you may need to do that in a separate RPC that is the only thing you use to stream data from the backend. Or use the websocket proxy, apparently Chrome supports 200 concurrent websocket connections.

[paulsmithkc]

@maja42 This sounds like a use case were just using the Web Sockets API directly might just be better suited to your needs.

See Writing WebSocket client applications

[maja42]

Hi, because if those reasons we switched to grpc-web which solves all these issues.
We are now able to communicate directly via grpc inside the browser (no rest) and have all types/clients directly generated from the original protobufs, which reduced our amount of work compared to grpc-gateway. It also allows unlimited streaming like in the backend, only client-side streams (and bidirectional streams) are not supported yet.

So this issue is not a problem for us anymore, but it would have been great if these limitations were documented somewhere more prominent, because it could have saved us a lot of time, and we would have figured out earlier that this technology is not suitable for our use-case.

[maja42]

@paulsmithkc As a side note: We are exclusively using grpc on the backend and explicitly didn't want to use web-sockets. It would require us to write additional code for every single RPC that we expose to the UI. And splitting an API up into two different technologies (Rest + web-sockets) based on an arbitrary property (stream vs. no-stream) also doesn't look very desirable. In addition to two new technologies, we would need to test both our grpc API, and the rest/web-sockets API, vastly increasing the test-overhead. The API-documentation would also need to exist twice (once for grpc for internal backend-services, and once for rest/web-sockets.
I know that this is the common approach nowadays, and many things can be auto-generated (with a varying quality) - but I think the additional complexity and overhead is a big reason why many web-applications are still based on polling.
While grpc-web is very new, it worked flawless so far, and the amount of work we need to do for each RPC is as low as you can get.