Introduce `identify` and `rendezvous` network protocols #304

mycognosist · 2023-03-17T12:15:24Z

Addresses #303

📋 Checklist

Add tests that cover your changes
Add this PR to the Unreleased section in CHANGELOG.md
Link this PR to any issues it closes
New files contain a SPDX license header

codecov · 2023-03-17T12:25:19Z

Codecov Report

Patch coverage: 23.40% and project coverage change: -1.48 ⚠️

Comparison is base (3ebe1eb) 92.42% compared to head (63aa43f) 90.94%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #304      +/-   ##
==========================================
- Coverage   92.42%   90.94%   -1.48%     
==========================================
  Files          51       52       +1     
  Lines        3987     4065      +78     
==========================================
+ Hits         3685     3697      +12     
- Misses        302      368      +66

Impacted Files	Coverage Δ
aquadoggo/src/network/config.rs	`100.00% <ø> (ø)`
aquadoggo/src/network/service.rs	`23.07% <6.34%> (-20.30%)`	⬇️
aquadoggo/src/network/behaviour.rs	`58.06% <58.06%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

…tion establishment

mycognosist · 2023-03-20T12:59:27Z

aquadoggo/src/network/config.rs

@@ -13,6 +14,9 @@ use crate::network::identity::Identity;
 /// Key pair file name.
 const KEY_PAIR_FILE_NAME: &str = "private-key";

+/// The namespace used by the `identify` network behaviour.
+pub const NODE_NAMESPACE: &str = "aquadoggo";


We might want to make this user-definable in the future. The namespace value is used when registering a rendezvous client with a rendezvous server and querying for other registered peers.

Yes, ideally that parameter would be reused by other discovery mechanisms, like mDNS (probably not a "namespace" there but something similar?)

Agreed, sharing a common identifier would be best. mDNS uses a 'service name' which is currently not user-configurable in the Rust implementation (see libp2p/rust-libp2p#2898 for more detail).

mycognosist · 2023-03-20T13:00:06Z

aquadoggo/src/network/mod.rs

@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: AGPL-3.0-or-later

+mod behaviour;


Moving the network behaviour into a module makes the service less busy.

mycognosist · 2023-03-20T13:00:47Z

aquadoggo/src/network/service.rs

-        let remote: Multiaddr = addr.parse()?;
-        swarm.dial(remote)?;
+    // Dial each peer identified by the multi-address provided via `--remote-node-addresses` if given
+    for addr in network_config.remote_peers.clone() {


Ensure we dial all remote nodes and not just the first one in the list.

mycognosist · 2023-03-20T13:02:01Z

aquadoggo/src/network/service.rs

+                    swarm
+                        .behaviour_mut()
+                        .rendezvous_client
+                        .as_mut()


This looks messy, unfortunately. We have to call .as_mut() because the rendezvous_client behaviour is behind a Toggle type.

mycognosist · 2023-03-20T13:03:16Z

aquadoggo/src/network/service.rs

@@ -140,13 +110,34 @@ pub async fn network_service(
                } => {
                    debug!("ConnectionClosed: {peer_id} {endpoint:?} {num_established} {cause:?}")
                }
+                SwarmEvent::ConnectionEstablished { peer_id, .. }
+                    if network_config.rendezvous_client
+                        && network_config.rendezvous_peer_id.unwrap() == peer_id =>


We should always be safe to unwrap the rendezvous_peer_id here because the CLI will ensure it's provided if rendezvous_client was set to true.

Give it a comment in the code 👍🏻

mycognosist · 2023-03-20T13:04:02Z

aquadoggo/src/network/service.rs

+                                    address.clone()
+                                };
+
+                                swarm.dial(address_with_p2p).unwrap();


Dial every peer we learn about from the rendezvous server.

mycognosist · 2023-03-20T13:05:15Z

aquadoggo_cli/src/main.rs

@@ -25,7 +27,7 @@ struct Cli {

    /// URLs of remote nodes to replicate with.
    #[arg(short, long)]
-    remote_node_addresses: Vec<String>,
+    remote_node_addresses: Vec<Multiaddr>,


By using Multiaddr and PeerId (instead of String) we get validation of the input automatically :) No need to try and .parse() later on.

mycognosist · 2023-03-20T13:11:30Z

@adzialocha @sandreae

This is now ready for review. I've left some comments to explain bits and pieces.

I'm interested to hear what you think of the rendezvous-related API (especially as exposed by the CLI). I'm not sure the naming is the best but @adzialocha mentioned that this can always be refactored later on when more has been implemented.

In a future PR, I'm quite keen to separate the swarm event loop into handlers to make it a bit neater. So, for example, there would be a handle_mdns() method which would encapsulate that functionality and be called when an mDNS event occurs. The same could be done for rendezvous and identify etc. Right now it's a bit time-consuming to comb through all the swarm event handlers and get a sense for what's happening. Here's an example from ursa: https://github.com/fleek-network/ursa/blob/9311c6d0a4514a1f6734ce22863373902551ef0d/crates/ursa-network/src/service.rs#L708

aquadoggo_cli/Cargo.toml

adzialocha

Looking all very good, mostly just small questions ☺️

aquadoggo/src/network/behaviour.rs

adzialocha · 2023-03-23T08:08:48Z

aquadoggo/src/network/config.rs

@@ -13,6 +14,9 @@ use crate::network::identity::Identity;
 /// Key pair file name.
 const KEY_PAIR_FILE_NAME: &str = "private-key";

+/// The namespace used by the `identify` network behaviour.
+pub const NODE_NAMESPACE: &str = "aquadoggo";


Yes, ideally that parameter would be reused by other discovery mechanisms, like mDNS (probably not a "namespace" there but something similar?)

adzialocha · 2023-03-23T08:10:00Z

aquadoggo/src/network/service.rs

@@ -140,13 +110,34 @@ pub async fn network_service(
                } => {
                    debug!("ConnectionClosed: {peer_id} {endpoint:?} {num_established} {cause:?}")
                }
+                SwarmEvent::ConnectionEstablished { peer_id, .. }
+                    if network_config.rendezvous_client
+                        && network_config.rendezvous_peer_id.unwrap() == peer_id =>


Give it a comment in the code 👍🏻

aquadoggo/src/network/service.rs

sandreae

All looks really good to me, don't have any change requests!

👏 👏 👏

sandreae · 2023-03-23T15:11:03Z

aquadoggo/src/network/config.rs

@@ -71,7 +75,23 @@ pub struct NetworkConfiguration {
    pub quic_port: u16,

    /// The addresses of remote peers to replicate from.
-    pub remote_peers: Vec<String>,
+    pub remote_peers: Vec<Multiaddr>,


* main: Create and validate abstract queries (#302) Handle null value for viewId in nextArgs Introduce `identify` and `rendezvous` network protocols (#304) Restructure graphql modules (#307)

mycognosist added 4 commits March 17, 2023 14:12

Add rendezvous and identify feature dependencies for libp2p

e8c5f7a

Introduce configuration parameters for rendezvous client and server

9969eef

Move network behaviour composer into separate module

ca40ec4

Add behaviour handlers for identify and rendezvous

6312f00

mycognosist marked this pull request as draft March 17, 2023 12:15

mycognosist added 14 commits March 18, 2023 19:17

Add custom validator for rendezvous client mode

9caf18d

Add handler to dial peers discovered via rendezvous server

38c561c

Log failed registration attempt with rendezvous server

9cc5c3e

Add handlers for Sent, Pushed and Error identify events

f136010

Add serde feature to libp2p dependency

2a1d96e

Add libp2p dependency to CLI for parser

d789431

Update lockfile

fbd2201

Replace String with Multiaddr and PeerId in CLI and config

8481075

Add glyph to authors

ce2c7d0

Update CHANGELOG

edac3b7

Remove TODOs and errant whitespace

c5ef331

Manually add external address and discover peers on rendezvous connec…

45b7b81

…tion establishment

Make log formatting syntax more consistent

fef9056

Update CLI usage instructions

56224b7

mycognosist marked this pull request as ready for review March 20, 2023 12:57

mycognosist commented Mar 20, 2023

View reviewed changes

sandreae reviewed Mar 21, 2023

View reviewed changes

aquadoggo_cli/Cargo.toml Show resolved Hide resolved

adzialocha reviewed Mar 23, 2023

View reviewed changes

adzialocha self-requested a review March 23, 2023 08:13

Add additional rendezvous client behaviour check and unwrap comments

158beca

adzialocha approved these changes Mar 23, 2023

View reviewed changes

Use if let to avoid unwrap

63aa43f

sandreae approved these changes Mar 23, 2023

View reviewed changes

adzialocha merged commit 76c1d44 into main Mar 23, 2023

adzialocha deleted the add_rendezvous branch April 18, 2023 10:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce `identify` and `rendezvous` network protocols #304

Introduce `identify` and `rendezvous` network protocols #304

mycognosist commented Mar 17, 2023 •

edited

Loading

codecov bot commented Mar 17, 2023 •

edited

Loading

mycognosist Mar 20, 2023

adzialocha Mar 23, 2023

mycognosist Mar 23, 2023

mycognosist Mar 20, 2023

mycognosist Mar 20, 2023

mycognosist Mar 20, 2023

mycognosist Mar 20, 2023

adzialocha Mar 23, 2023

mycognosist Mar 20, 2023

mycognosist Mar 20, 2023

mycognosist commented Mar 20, 2023 •

edited

Loading

adzialocha left a comment

adzialocha Mar 23, 2023

adzialocha Mar 23, 2023

sandreae left a comment

sandreae Mar 23, 2023

		@@ -1,5 +1,6 @@
		// SPDX-License-Identifier: AGPL-3.0-or-later

		mod behaviour;

Introduce identify and rendezvous network protocols #304

Introduce identify and rendezvous network protocols #304

Conversation

mycognosist commented Mar 17, 2023 • edited Loading

📋 Checklist

codecov bot commented Mar 17, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mycognosist commented Mar 20, 2023 • edited Loading

adzialocha left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sandreae left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Introduce `identify` and `rendezvous` network protocols #304

Introduce `identify` and `rendezvous` network protocols #304

mycognosist commented Mar 17, 2023 •

edited

Loading

codecov bot commented Mar 17, 2023 •

edited

Loading

mycognosist commented Mar 20, 2023 •

edited

Loading