p2p: stream response payload #56
Comments
bajtos
added a commit
that referenced
this issue
Feb 23, 2023
This PR adds a built-in libp2p node based on rust-libp2p. The node is shared by all Station Modules running on Zinnia. This way, we can lower the number of open connections and avoid duplicate entries in DHTs. **Networking stack** - Transport: `tcp` using system DNS resolver - Multistream-select V1 - Authentication: `noise` with `XX` handshake pattern using X25519 DH keys - Stream multiplexing: both `yamux` and `mplex` **`Zinnia.peerId`** Type: `string` Return the peer id of Zinnia's built-in libp2p peer. The peer id is ephemeral, Zinnia generates a new peer id every time it starts. **`Zinnia.requestProtocol(remoteAddress, protocolName, requestPayload)`** ```ts requestProtocol( remoteAddress: string, protocolName: string, requestPayload: Uint8Array, ): Promise<PeerResponse>; ``` Dial a remote peer identified by the `remoteAddress` and open a new substream for the protocol identified by `protocolName`. Send `requestPayload` and read the response payload. The function returns a promise that resolves with a readable-stream-like object. At the moment, this object implements "async iterable" protocol only, it's not a full readable stream. This is enough to allow you to receive response in chunks, where each chunk is an `Uint8Array` instance. Notes: - The peer address must include both the network address and peer id. - The response size is limited to 10MB. Larger responses will be rejected with an error. - We will implement stream-based API supporting unlimited request & response sizes in the near future, see #56 and #57. --------- Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently, our libp2p node reads the entire response into memory with a hard-coded limit of 10MB.
On the JavaScript side, we provide API based on async iterable that's prepared for streaming.
Now we need to rework the Rust side to read responses in chunks too.
Nice to have: Change the JS response object to be a proper ReadableStream. This could potentially simplify the Rust side if we can find an easy way how to leverage Deno infrastructure for readable streams and wrap the rust-libp2p substream into a readable Deno resource.
The text was updated successfully, but these errors were encountered: