feat(swarm): allow behaviours to share addresses of peers #4371

StemCll · 2023-08-21T21:42:25Z

Description

Resolves: #4302.

Notes & open questions

Still WIP

Change checklist

I have performed a self-review of my own code
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
A changelog entry has been made in the appropriate crates

mxinden

This is going in the right direction. Thank you for the work!

protocols/kad/src/behaviour.rs

swarm/src/lib.rs

protocols/kad/src/behaviour/test.rs

thomaseizinger

Great progress! I've left a few comments :)

What I'd also like to see in this PR is:

mDNS emitting ToSwarm::NewExternalAddrOfPeer
identify emitting ToSwarm::NewExternalAddrOfPeer for the self-advertised listen addresses of a peer
the rendezvous client emitting ToSwarm::NewExternalAddrOfPeer whenever it receives a peer record

Something that we should discuss is whether we want to add a public function to Swarm that allows users to trigger this event without having to call particular behaviour. For example, at the moment, libp2p-request-response offers a function add_address to populate an internal map of peer addresses. I'd like that function to go away because I think it is none of libp2p-request-responses business to do address management.

If we were to add a function to Swarm that triggers this event, we could deprecate the function in libp2p-request-response and instead add an internal cache that stores peer addresses. To avoid duplicated code across multiple behaviours, we could add a PeerAddresses to libp2p-swarm similar to how we have ExternalAddresses.

I don't think all of this has to be done in one PR but what we do should be consistent. For example, I think it would be to merge this whilst also deprecating libp2p-request-response's add_address function to point to the new Swarm function instead. We can always introduce the PeerAddresses struct at a later point because it is a backwards-compatible change.

protocols/autonat/src/behaviour.rs

protocols/identify/src/behaviour.rs

protocols/request-response/src/lib.rs

swarm-derive/src/lib.rs

swarm/src/behaviour.rs

swarm/src/lib.rs

protocols/kad/src/behaviour.rs

protocols/kad/src/behaviour/test.rs

thomaseizinger · 2023-09-12T00:22:16Z

For example, at the moment, libp2p-request-response offers a function add_address to populate an internal map of peer addresses. I'd like that function to go away because I think it is none of libp2p-request-responses business to do address management.

The same actually applies to Kademlia::add_address. This function could be made crate-private in favour of the new function on Swarm.

thomaseizinger · 2023-09-17T11:08:21Z

Something that we should discuss is whether we want to add a public function to Swarm that allows users to trigger this event without having to call particular behaviour. For example, at the moment, libp2p-request-response offers a function add_address to populate an internal map of peer addresses. I'd like that function to go away because I think it is none of libp2p-request-responses business to do address management.

If we were to add a function to Swarm that triggers this event, we could deprecate the function in libp2p-request-response and instead add an internal cache that stores peer addresses. To avoid duplicated code across multiple behaviours, we could add a PeerAddresses to libp2p-swarm similar to how we have ExternalAddresses.

I don't think all of this has to be done in one PR but what we do should be consistent. For example, I think it would be to merge this whilst also deprecating libp2p-request-response's add_address function to point to the new Swarm function instead. We can always introduce the PeerAddresses struct at a later point because it is a backwards-compatible change.

@mxinden Let me know what you think of this.

swarm/src/behaviour.rs

swarm-derive/CHANGELOG.md

swarm/src/behaviour.rs

mxinden · 2023-09-17T18:55:38Z

Something that we should discuss is whether we want to add a public function to Swarm that allows users to trigger this event without having to call particular behaviour. For example, at the moment, libp2p-request-response offers a function add_address to populate an internal map of peer addresses. I'd like that function to go away because I think it is none of libp2p-request-responses business to do address management.
If we were to add a function to Swarm that triggers this event, we could deprecate the function in libp2p-request-response and instead add an internal cache that stores peer addresses. To avoid duplicated code across multiple behaviours, we could add a PeerAddresses to libp2p-swarm similar to how we have ExternalAddresses.
I don't think all of this has to be done in one PR but what we do should be consistent. For example, I think it would be to merge this whilst also deprecating libp2p-request-response's add_address function to point to the new Swarm function instead. We can always introduce the PeerAddresses struct at a later point because it is a backwards-compatible change.

@mxinden Let me know what you think of this.

I am in favor of the additional method on Swarm. I am as well fine for that to happen in a follow-up pull request.

mxinden

As always, appreciate the thorough work @StemCll!

swarm/src/behaviour.rs

thomaseizinger · 2023-09-17T21:47:26Z

Something that we should discuss is whether we want to add a public function to Swarm that allows users to trigger this event without having to call particular behaviour. For example, at the moment, libp2p-request-response offers a function add_address to populate an internal map of peer addresses. I'd like that function to go away because I think it is none of libp2p-request-responses business to do address management.
If we were to add a function to Swarm that triggers this event, we could deprecate the function in libp2p-request-response and instead add an internal cache that stores peer addresses. To avoid duplicated code across multiple behaviours, we could add a PeerAddresses to libp2p-swarm similar to how we have ExternalAddresses.
I don't think all of this has to be done in one PR but what we do should be consistent. For example, I think it would be to merge this whilst also deprecating libp2p-request-response's add_address function to point to the new Swarm function instead. We can always introduce the PeerAddresses struct at a later point because it is a backwards-compatible change.

@mxinden Let me know what you think of this.

I am in favor of the additional method on Swarm. I am as well fine for that to happen in a follow-up pull request.

I am wondering what a good scope is for this PR. It is not very big at this stage. Given that it is a breaking change, perhaps we should at least use it "end-to-end":

Have at least 1 behaviour emit this event (perhaps identify?)
Have at least 1 behaviour consume it, most likely kademlia
Add the function to Swarm
Deprecate at least 1 of the existing add_address functions and use the new Swarm method everywhere instead. This will have to be the same behaviour as in (2).

If that works, we can write follow-up issues to "transfer" the design in this PR to other behaviours.

Thoughts @StemCll @mxinden ?

thomaseizinger

Thanks for addressing the continued work!

I made some more comments :)
Plus, I'd still like to see the Swarm::add_address_of_peer function and deprecate Kademlia::add_address with a note pointing to the other function. In the future, we will then make that private.

protocols/identify/src/behaviour.rs

swarm-derive/CHANGELOG.md

swarm/CHANGELOG.md

protocols/identify/Cargo.toml

swarm/src/behaviour.rs

thomaseizinger · 2023-09-25T23:11:03Z

We will also need to update the docs here: https://github.com/libp2p/rust-libp2p/blob/master/protocols/kad/src/lib.rs#L21-L34

mergify · 2023-09-28T06:21:43Z

This pull request has merge conflicts. Could you please resolve them @StemCll? 🙏

protocols/identify/src/behaviour.rs

thomaseizinger · 2023-10-24T00:14:28Z

I am removing this from the next milestone because we can hopefully ship it as a non-breaking change once #4581 lands.

StemCll · 2023-11-29T21:17:58Z

hi @thomaseizinger @mxinden 👋
Had to rework the PR because of the multiple conflicts with the newest master

swarm/src/behaviour/peer_addresses.rs

thomaseizinger · 2024-01-02T15:10:40Z

I have limited availability the next weeks, will re-review mid-January :)

mergify · 2024-01-05T07:47:13Z

This pull request has merge conflicts. Could you please resolve them @StemCll? 🙏

thomaseizinger

Thanks! Sorry for the delay on this.

Let me know if you are able to work in a few more comments, otherwise I'll finalize this at some point in the next two weeks.

protocols/identify/src/behaviour.rs

protocols/request-response/src/lib.rs

swarm/src/behaviour/peer_addresses.rs

StemCll · 2024-01-16T07:47:12Z

Thanks! Sorry for the delay on this.

Let me know if you are able to work in a few more comments, otherwise I'll finalize this at some point in the next two weeks.

Hi @thomaseizinger. No worries at all and thank you for getting back to this 😄. I will add these changes in a day or two.

thomaseizinger

Awesome! Thank you so much for this contribution and working in all my feedback :)

I've made a note for myself to add another test, will merge that in the upcoming days!

protocols/identify/src/behaviour.rs

This allows us to remove the `put` function and we can now safely add all addresses individually to the cache. With the addition of the `ToSwarm::NewExternalAddrOfPeer` event, the safeguard of overwriting all addresses in the peer cache was no longer useful because any other behaviour would have to apply this as well. As such, we needed to build this safe-guard directly _into_ `PeerAddresses`. This also makes the added test pass. Previously, the events would never get emitted because after calling `.put`, the addresses were already in the cache and the `add`-function always returned `false`.

thomaseizinger · 2024-01-21T06:05:27Z

The test didn't initially pass so I had to make some more changes to the implementation.

Approvals have been dismissed because the PR was updated after the send-it label was applied.

Resolves: libp2p#4302. Pull-Request: libp2p#4371.

Resolves: #4302. Pull-Request: #4371.

StemCll changed the title ~~swarm: Add NewExternalAddrOfPeer~~ feat(swarm): Add NewExternalAddrOfPeer Aug 21, 2023

StemCll force-pushed the feat/swarm/report_remote_address branch from e435d95 to 711b03b Compare August 21, 2023 21:45

mxinden reviewed Aug 25, 2023

View reviewed changes

protocols/kad/src/behaviour.rs Outdated Show resolved Hide resolved

swarm/src/lib.rs Outdated Show resolved Hide resolved

mxinden mentioned this pull request Aug 27, 2023

feat(swarm): allow NetworkBehaviour to report remote address #4302

Closed

StemCll force-pushed the feat/swarm/report_remote_address branch from 711b03b to 3c4d34e Compare September 11, 2023 20:49

StemCll commented Sep 11, 2023

View reviewed changes

protocols/kad/src/behaviour/test.rs Outdated Show resolved Hide resolved

thomaseizinger changed the title ~~feat(swarm): Add NewExternalAddrOfPeer~~ feat(swarm): allow behaviours to share addresses of peers Sep 12, 2023

thomaseizinger reviewed Sep 12, 2023

View reviewed changes

StemCll force-pushed the feat/swarm/report_remote_address branch from 18ba25d to 0b292c0 Compare September 17, 2023 08:01

thomaseizinger reviewed Sep 17, 2023

View reviewed changes

swarm/src/behaviour.rs Outdated Show resolved Hide resolved

swarm-derive/CHANGELOG.md Outdated Show resolved Hide resolved

swarm-derive/CHANGELOG.md Outdated Show resolved Hide resolved

swarm/src/behaviour.rs Outdated Show resolved Hide resolved

mxinden reviewed Sep 17, 2023

View reviewed changes

swarm/src/behaviour.rs Outdated Show resolved Hide resolved

swarm/src/behaviour.rs Outdated Show resolved Hide resolved

thomaseizinger mentioned this pull request Sep 21, 2023

refactor(kad): make Kademlia::add_address return Result #4280

Draft

4 tasks

StemCll force-pushed the feat/swarm/report_remote_address branch from bec1aaf to 6534f13 Compare September 23, 2023 21:06

thomaseizinger reviewed Sep 24, 2023

View reviewed changes

thomaseizinger added this to the v0.53.0 milestone Sep 28, 2023

thomaseizinger reviewed Oct 9, 2023

View reviewed changes

protocols/identify/src/behaviour.rs Outdated Show resolved Hide resolved

protocols/identify/src/behaviour.rs Outdated Show resolved Hide resolved

thomaseizinger removed this from the v0.53.0 milestone Oct 24, 2023

dhuseby mentioned this pull request Nov 2, 2023

v0.53 release #4545

Closed

StemCll force-pushed the feat/swarm/report_remote_address branch 3 times, most recently from 3cdcfac to 1704ee9 Compare November 28, 2023 18:21

StemCll commented Nov 29, 2023

View reviewed changes

swarm/src/behaviour/peer_addresses.rs Outdated Show resolved Hide resolved

StemCll requested a review from thomaseizinger January 1, 2024 22:22

Merge branch 'master' into feat/swarm/report_remote_address

af0a91c

thomaseizinger reviewed Jan 13, 2024

View reviewed changes

protocols/identify/src/behaviour.rs Show resolved Hide resolved

protocols/request-response/src/lib.rs Show resolved Hide resolved

swarm/src/behaviour/peer_addresses.rs Outdated Show resolved Hide resolved

swarm/src/behaviour/peer_addresses.rs Outdated Show resolved Hide resolved

StemCll added 2 commits January 17, 2024 20:56

Merge branch 'master' into feat/swarm/report_remote_address

41b533e

Review suggestions resolved

31723f2

StemCll requested a review from thomaseizinger January 18, 2024 08:26

Merge branch 'master' into feat/swarm/report_remote_address

5059d51

thomaseizinger previously approved these changes Jan 20, 2024

View reviewed changes

protocols/identify/src/behaviour.rs Outdated Show resolved Hide resolved

Allow new event in dctur test

6fae501

thomaseizinger added the send-it label Jan 21, 2024

This was referenced Jan 21, 2024

kad: emit ToSwarm::NewExternalAddrOfPeer #5103

Closed

mDNS: emit ToSwarm::NewExternalAddrOfPeer #5104

Open

rendezvous: emit ToSwarm::NewExternalAddrOfPeer #5105

Open

Fix clippy warnings

1866abd

thomaseizinger mentioned this pull request Jan 21, 2024

misc: add a PeerStore behaviour #4103

Open

thomaseizinger approved these changes Jan 21, 2024

View reviewed changes

thomaseizinger removed the send-it label Jan 24, 2024

Merge branch 'master' into feat/swarm/report_remote_address

b1bedf7

thomaseizinger added the send-it label Jan 24, 2024

Merge branch 'master' into feat/swarm/report_remote_address

49d53d1

mergify bot merged commit c48a16a into libp2p:master Jan 24, 2024
72 of 73 checks passed

umgefahren pushed a commit to umgefahren/rust-libp2p that referenced this pull request Mar 8, 2024

feat(swarm): allow behaviours to share addresses of peers

32d05ec

Resolves: libp2p#4302. Pull-Request: libp2p#4371.

guillaumemichel pushed a commit that referenced this pull request Mar 28, 2024

feat(swarm): allow behaviours to share addresses of peers

a153df9

Resolves: #4302. Pull-Request: #4371.

vnermolaev mentioned this pull request May 10, 2024

request-response: Inconsistent documentation. #5372

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(swarm): allow behaviours to share addresses of peers #4371

feat(swarm): allow behaviours to share addresses of peers #4371

StemCll commented Aug 21, 2023 •

edited

Loading

mxinden left a comment

thomaseizinger left a comment

thomaseizinger commented Sep 12, 2023

thomaseizinger commented Sep 17, 2023

mxinden commented Sep 17, 2023

mxinden left a comment

thomaseizinger commented Sep 17, 2023 •

edited

Loading

thomaseizinger left a comment

thomaseizinger commented Sep 25, 2023

mergify bot commented Sep 28, 2023

thomaseizinger commented Oct 24, 2023

StemCll commented Nov 29, 2023

thomaseizinger commented Jan 2, 2024

mergify bot commented Jan 5, 2024

thomaseizinger left a comment

StemCll commented Jan 16, 2024

thomaseizinger left a comment

thomaseizinger commented Jan 21, 2024

feat(swarm): allow behaviours to share addresses of peers #4371

feat(swarm): allow behaviours to share addresses of peers #4371

Conversation

StemCll commented Aug 21, 2023 • edited Loading

Description

Notes & open questions

Change checklist

mxinden left a comment

Choose a reason for hiding this comment

thomaseizinger left a comment

Choose a reason for hiding this comment

thomaseizinger commented Sep 12, 2023

thomaseizinger commented Sep 17, 2023

mxinden commented Sep 17, 2023

mxinden left a comment

Choose a reason for hiding this comment

thomaseizinger commented Sep 17, 2023 • edited Loading

thomaseizinger left a comment

Choose a reason for hiding this comment

thomaseizinger commented Sep 25, 2023

mergify bot commented Sep 28, 2023

thomaseizinger commented Oct 24, 2023

StemCll commented Nov 29, 2023

thomaseizinger commented Jan 2, 2024

mergify bot commented Jan 5, 2024

thomaseizinger left a comment

Choose a reason for hiding this comment

StemCll commented Jan 16, 2024

thomaseizinger left a comment

Choose a reason for hiding this comment

thomaseizinger commented Jan 21, 2024

StemCll commented Aug 21, 2023 •

edited

Loading

thomaseizinger commented Sep 17, 2023 •

edited

Loading