feat: built-in libp2p node

[bajtos]

See #9

---

Quoting from the documentation:

Zinnia comes with a built-in libp2p node based on rust-libp2p. The node is shared by all Station Modules running on Zinnia. This way we can keep the number of open connections lower and avoid duplicate entries in routing tables.

The initial version comes with a limited subset of features. We will be adding more features based on feedback from our users (Station Module builders).

Networking stack

• Transport: tcp using system DNS resolver
• Multistream-select V1
• Authentication: noise with XX handshake pattern using X25519 DH keys
• Stream multiplexing: both yamux and mplex

Zinnia.peerId

Type: string

Return the peer id of Zinnia's built-in libp2p peer. The peer id is ephemeral, Zinnia generates a new peer id every time it starts.

Zinnia.requestProtocol(remoteAddress, protocolName, requestPayload)

requestProtocol(
  remoteAddress: string,
  protocolName: string,
  requestPayload: Uint8Array,
): Promise<PeerResponse>;

This PR adds a built-in libp2p node based on rust-libp2p. The node is shared by
all Station Modules running on Zinnia. This way, we can lower the number of
open connections and avoid duplicate entries in DHTs.

Networking stack

• Transport: tcp using system DNS resolver
• Multistream-select V1
• Authentication: noise with XX handshake pattern using X25519 DH keys
• Stream multiplexing: both yamux and mplex

Zinnia.peerId

Type: string

Return the peer id of Zinnia's built-in libp2p peer. The peer id is ephemeral,
Zinnia generates a new peer id every time it starts.

Zinnia.requestProtocol(remoteAddress, protocolName, requestPayload)

requestProtocol(
  remoteAddress: string,
  protocolName: string,
  requestPayload: Uint8Array,
): Promise<PeerResponse>;

Dial a remote peer identified by the remoteAddress and open a new substream for the protocol identified by protocolName. Send requestPayload and read the response payload.

The function returns a promise that resolves with a readable-stream-like object. At the moment, this object implements "async iterable" protocol only, it's not a full readable stream. This is enough to allow you to receive response in chunks, where each chunk is an Uint8Array instance.

Notes:

• The peer address must include both the network address and peer id.
• The response size is limited to 10MB. Larger responses will be rejected with an error.
• We will implement stream-based API supporting unlimited request & response sizes in the near future, see p2p: stream response payload #56 and p2p: stream request payload #57.

Example

const response = await Zinnia.requestProtocol(
  "/dns/example.com/tcp/3030/p2p/12D3okowHR71QRJe5vrPm6zZXoH4K7z5mDsWWtxXpRIG9Dk8hqxk",
  "/ipfs/ping/1.0.0",
  new Uint8Array(32),
);
for (const chunk of response) {
  console.log(chunk);
}

[bajtos]

The PR is ready for review 🙏🏻

@juliangruber Please focus on especially on the JavaScript bits (the public API)

@tchardin @mxinden if you have the bandwidth to review the libp2p side, then I am happy to hear your feedback! You can find the libp2p bits in feat-libp2p/ext/libp2p/peer.rs and feat-libp2p/ext/libp2p/peer/*.rs.

/cc @patrickwoodhead in case you were interested

[juliangruber]

Nice work! As far as I understand it, lgtm!

[mxinden]

Thanks for the ping. Great to see rust-libp2p in action here.

Let me know in case I can be of any help.

Also let me know in case you need help upgrading to the next release (v0.51.0).

[mxinden]

Where is the clone happening? You could consider adding a lifetime to RequestProtocol. Also you could consider using buffer pools. All that said, I wouldn't introduce such optimizations without previous benchmarks showing that this is an actual bottleneck.

[bajtos]

For request payloads:

1. The JS engine allocates Uint8Array on the GC-managed heap

2. Our op receives this value as a ZeroCopyBuf. Quoting from Deno API docs:

   A ZeroCopyBuf encapsulates a slice that's been borrowed from a JavaScript ArrayBuffer object. JavaScript objects can normally be garbage collected, but the existence of a ZeroCopyBuf inhibits this until it is dropped. It behaves much like an Arc<u8>.

   I think we may be able to use &[u8] instead of ZeroCopyBuf in our op function, but I am not sure.

   The concept is the same though - we get a reference to a slice pointing into the memory managed by the JS runtime.

   Now that I am writing this, I got an idea: maybe I can change RequestPayload to become an alias for ZeroCopyBuf and see how that works.

3. Our op converts that slice into Vec<u8> and that's where to clone/copy happens.

   zinnia/ext/libp2p/lib.rs

   Line 77 in bc2712a

   request_payload.to_vec(),

A somewhat-related discussion in Deno that may shed more light on this topic: denoland/deno#4788

For response payloads:

1. We allocate Vec<u8> so we have a place to read the response payload into.
   

   zinnia/ext/libp2p/peer/protocol.rs

   Lines 72 to 73 in bc2712a

   	let mut response: ResponsePayload = Default::default();
   	io.take(10 * 1024 * 1024).read_to_end(&mut response).await?;

2. The op returns this value to Deno runtime. The runtime allocates a new Uint8Array on the GC-managed heap and copies the data there.

I want to rework this part very soon (see #56) to allow the JavaScript side to repeatedly read a chunk of data from the sub stream into a buffer provided by the JavaScript side (a.k.a BYOB). I expect that work to require changes to the ResponsePayload; thus I am not too worried about this response part.

[bajtos]

Also let me know in case you need help upgrading to the next release (v0.51.0).

@mxinden Thank you for the offer! I'll reach out in case of issues I am not able to resolve myself.

BTW, I was expecting Dependabot to open a pull request to upgrade libp2p to the next release. Do you happen to know why it did not and/or how can we change our config to fix that?

I suspect this is related to Dependabot's config option versioning-strategy (docs):

• For apps, the version requirements are increased, for example: npm, pip and Composer.
• For libraries, the range of versions is widened, for example: Bundler and Cargo.

In our workspace, we have both an app (cli/Cargo.toml) and several libraries (e.g. ext/libp2p/Cargo.toml and runtime/Cargo.toml).

Ideally, I want to build the CLI with the latest versions of dependencies but keep our libraries supporting a range of older versions that are compatible with the latest.

Do you have any suggestions or recommendations on specifying our dependency version ranges and configuring Dependabot to achieve that?

[mxinden]

In our workspace, we have both an app (cli/Cargo.toml) and several libraries (e.g. ext/libp2p/Cargo.toml and runtime/Cargo.toml).

As far as I can tell, you are only pointing dependabot to / for cargo. Instead I think you need to point to specific locations, see e.g.

https://github.com/mxinden/libp2p-perf/blob/master/.github/dependabot.yml