Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

[client/network] Update authority discovery after authority-discovery w/ PR#3230 is released #13338

Closed
melekes opened this issue Feb 8, 2023 · 1 comment
Labels
U3-nice_to_have Issue is worth doing eventually.

Comments

@melekes
Copy link
Contributor

melekes commented Feb 8, 2023

libp2p/rust-libp2p#3230 invalidates the assumptions in Substrate's Authority Discovery.

impl Default for WorkerConfig {
fn default() -> Self {
Self {
// Kademlia's default time-to-live for Dht records is 36h, republishing records every
// 24h through libp2p-kad. Given that a node could restart at any point in time, one can
// not depend on the republishing process, thus publishing own external addresses should
// happen on an interval < 36h.
max_publish_interval: Duration::from_secs(1 * 60 * 60),
keystore_refresh_interval: Duration::from_secs(60),
// External addresses of remote authorities can change at any given point in time. The
// interval on which to trigger new queries for the current and next authorities is a
// trade off between efficiency and performance.
//
// Querying 700 [`AuthorityId`]s takes ~8m on the Kusama DHT (16th Nov 2020) when
// comparing `authority_discovery_authority_addresses_requested_total` and
// `authority_discovery_dht_event_received`.
max_query_interval: Duration::from_secs(10 * 60),
publish_non_global_ips: true,
strict_record_validation: false,
}
}
}

"Either you go with the new defaults and update Authority Discovery, or you override the defaults of libp2p."

libp2p/rust-libp2p#3230 (comment)

opening an issue so we don't forget

@melekes melekes added the U3-nice_to_have Issue is worth doing eventually. label Feb 8, 2023
@melekes melekes moved this to Backlog 🗒 in Networking Feb 9, 2023
melekes added a commit to melekes/substrate that referenced this issue Mar 27, 2023
@melekes melekes changed the title [client/network] Update authority discovery after libp2p 0.51 is released [client/network] Update authority discovery after authority-discovery w/ PR#3230 is released Mar 28, 2023
@github-project-automation github-project-automation bot moved this from Backlog 🗒 to Blocked ⛔️ in Networking May 12, 2023
@melekes
Copy link
Contributor Author

melekes commented May 12, 2023

Looks like libp2p/rust-libp2p#3230 was not merged. I will reopen this issue if libp2p team decide to reopen & merge it again.

pgherveou pushed a commit that referenced this issue May 12, 2023
* client/network: upgrade to libp2p 0.51.0

* make discovery.rs compile

* make peer_info.rs compile

* changes to notifications and request-response proto

* make service.rs compile

* towards making request_responses.rs compile

* make request_responses.rs compile

* make request_responses.rs compile

* fix notifications/behaviour.rs tests

* fix warnings

* remove old code

* allow deprecated code (temporary)

* upgrade to libp2p 0.51.1

* add TODO for behaviour tests

* return empty vec if peer_id is absent

#13587 (comment)

fyi: I don't really know what the old behaviour was.

* update comment to reflect new defaults

Closes #13338

* Revert "update comment to reflect new defaults"

This reverts commit 7a981ab.

* remove config.rs (from wrong merge)

* upgrade to libp2p 0.51.2

* fix formatting

* use handle_pending_outbound_connection in networt_state RPC

* update deps

* use re-exports when we use other libp2p packages

* Apply suggestions from code review

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* format code

* handle potential errors in network_state RPC

* only update libp2p crate

* update libp2p-core

* fix docs

* use libp2p-identity instead of libp2p

where it's possible. libp2p-identity is much smaller, hence makes sense
to use it instead of larger libp2p crate.

* Update client/network/src/discovery.rs

Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>

* update Cargo.lock

* add comment for per_connection_event_buffer_size

current value is somewhat arbitrary and needs to be tweaked depending on
memory usage and network worker sleep stats.

* fix link format

* update Cargo.lock

* upgrade to libp2p 0.51.3

* deprecate mplex

* Revert "deprecate mplex"

This reverts commit 9e25820.

* Revert "upgrade to libp2p 0.51.3"

This reverts commit 6544dd4.

* use new libp2p version in `statement` crate

* pin version temporarily

* libp2p 0.51.3

* deprecate mplex

* deprecate legacy noise handshake

* fix build error

* update libp2p-identity

* enable libp2p-identity:ed25519 feature in sc-consensus

* enable ed25519 for peerset as well

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Co-authored-by: parity-processbot <>
@altonen altonen moved this from Blocked ⛔️ to Done ✅ in Networking Jun 12, 2023
nathanwhit pushed a commit to nathanwhit/substrate that referenced this issue Jul 19, 2023
* client/network: upgrade to libp2p 0.51.0

* make discovery.rs compile

* make peer_info.rs compile

* changes to notifications and request-response proto

* make service.rs compile

* towards making request_responses.rs compile

* make request_responses.rs compile

* make request_responses.rs compile

* fix notifications/behaviour.rs tests

* fix warnings

* remove old code

* allow deprecated code (temporary)

* upgrade to libp2p 0.51.1

* add TODO for behaviour tests

* return empty vec if peer_id is absent

paritytech#13587 (comment)

fyi: I don't really know what the old behaviour was.

* update comment to reflect new defaults

Closes paritytech#13338

* Revert "update comment to reflect new defaults"

This reverts commit 7a981ab.

* remove config.rs (from wrong merge)

* upgrade to libp2p 0.51.2

* fix formatting

* use handle_pending_outbound_connection in networt_state RPC

* update deps

* use re-exports when we use other libp2p packages

* Apply suggestions from code review

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* format code

* handle potential errors in network_state RPC

* only update libp2p crate

* update libp2p-core

* fix docs

* use libp2p-identity instead of libp2p

where it's possible. libp2p-identity is much smaller, hence makes sense
to use it instead of larger libp2p crate.

* Update client/network/src/discovery.rs

Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>

* update Cargo.lock

* add comment for per_connection_event_buffer_size

current value is somewhat arbitrary and needs to be tweaked depending on
memory usage and network worker sleep stats.

* fix link format

* update Cargo.lock

* upgrade to libp2p 0.51.3

* deprecate mplex

* Revert "deprecate mplex"

This reverts commit 9e25820.

* Revert "upgrade to libp2p 0.51.3"

This reverts commit 6544dd4.

* use new libp2p version in `statement` crate

* pin version temporarily

* libp2p 0.51.3

* deprecate mplex

* deprecate legacy noise handshake

* fix build error

* update libp2p-identity

* enable libp2p-identity:ed25519 feature in sc-consensus

* enable ed25519 for peerset as well

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Co-authored-by: parity-processbot <>
Neopallium pushed a commit to PolymeshAssociation/substrate that referenced this issue Dec 7, 2023
* client/network: upgrade to libp2p 0.51.0

* make discovery.rs compile

* make peer_info.rs compile

* changes to notifications and request-response proto

* make service.rs compile

* towards making request_responses.rs compile

* make request_responses.rs compile

* make request_responses.rs compile

* fix notifications/behaviour.rs tests

* fix warnings

* remove old code

* allow deprecated code (temporary)

* upgrade to libp2p 0.51.1

* add TODO for behaviour tests

* return empty vec if peer_id is absent

paritytech#13587 (comment)

fyi: I don't really know what the old behaviour was.

* update comment to reflect new defaults

Closes paritytech#13338

* Revert "update comment to reflect new defaults"

This reverts commit 7a981ab.

* remove config.rs (from wrong merge)

* upgrade to libp2p 0.51.2

* fix formatting

* use handle_pending_outbound_connection in networt_state RPC

* update deps

* use re-exports when we use other libp2p packages

* Apply suggestions from code review

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* format code

* handle potential errors in network_state RPC

* only update libp2p crate

* update libp2p-core

* fix docs

* use libp2p-identity instead of libp2p

where it's possible. libp2p-identity is much smaller, hence makes sense
to use it instead of larger libp2p crate.

* Update client/network/src/discovery.rs

Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>

* update Cargo.lock

* add comment for per_connection_event_buffer_size

current value is somewhat arbitrary and needs to be tweaked depending on
memory usage and network worker sleep stats.

* fix link format

* update Cargo.lock

* upgrade to libp2p 0.51.3

* deprecate mplex

* Revert "deprecate mplex"

This reverts commit 9e25820.

* Revert "upgrade to libp2p 0.51.3"

This reverts commit 6544dd4.

* use new libp2p version in `statement` crate

* pin version temporarily

* libp2p 0.51.3

* deprecate mplex

* deprecate legacy noise handshake

* fix build error

* update libp2p-identity

* enable libp2p-identity:ed25519 feature in sc-consensus

* enable ed25519 for peerset as well

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Co-authored-by: parity-processbot <>
dmitrylavrenov pushed a commit to humanode-network/substrate that referenced this issue Jun 24, 2024
* client/network: upgrade to libp2p 0.51.0

* make discovery.rs compile

* make peer_info.rs compile

* changes to notifications and request-response proto

* make service.rs compile

* towards making request_responses.rs compile

* make request_responses.rs compile

* make request_responses.rs compile

* fix notifications/behaviour.rs tests

* fix warnings

* remove old code

* allow deprecated code (temporary)

* upgrade to libp2p 0.51.1

* add TODO for behaviour tests

* return empty vec if peer_id is absent

paritytech#13587 (comment)

fyi: I don't really know what the old behaviour was.

* update comment to reflect new defaults

Closes paritytech#13338

* Revert "update comment to reflect new defaults"

This reverts commit 7a981ab.

* remove config.rs (from wrong merge)

* upgrade to libp2p 0.51.2

* fix formatting

* use handle_pending_outbound_connection in networt_state RPC

* update deps

* use re-exports when we use other libp2p packages

* Apply suggestions from code review

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* format code

* handle potential errors in network_state RPC

* only update libp2p crate

* update libp2p-core

* fix docs

* use libp2p-identity instead of libp2p

where it's possible. libp2p-identity is much smaller, hence makes sense
to use it instead of larger libp2p crate.

* Update client/network/src/discovery.rs

Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>

* update Cargo.lock

* add comment for per_connection_event_buffer_size

current value is somewhat arbitrary and needs to be tweaked depending on
memory usage and network worker sleep stats.

* fix link format

* update Cargo.lock

* upgrade to libp2p 0.51.3

* deprecate mplex

* Revert "deprecate mplex"

This reverts commit 9e25820.

* Revert "upgrade to libp2p 0.51.3"

This reverts commit 6544dd4.

* use new libp2p version in `statement` crate

* pin version temporarily

* libp2p 0.51.3

* deprecate mplex

* deprecate legacy noise handshake

* fix build error

* update libp2p-identity

* enable libp2p-identity:ed25519 feature in sc-consensus

* enable ed25519 for peerset as well

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Co-authored-by: parity-processbot <>
dmitrylavrenov added a commit to humanode-network/substrate that referenced this issue Jun 25, 2024
…344`) (#104)

* Bump ed25519-zebra to 4.0.3

* Update Update schnorrkel to 0.11.4 and related to it deps

* Bump ed25519-dalek to 2.0

* Upgrade to libp2p 0.51.3 (paritytech#13587)

* client/network: upgrade to libp2p 0.51.0

* make discovery.rs compile

* make peer_info.rs compile

* changes to notifications and request-response proto

* make service.rs compile

* towards making request_responses.rs compile

* make request_responses.rs compile

* make request_responses.rs compile

* fix notifications/behaviour.rs tests

* fix warnings

* remove old code

* allow deprecated code (temporary)

* upgrade to libp2p 0.51.1

* add TODO for behaviour tests

* return empty vec if peer_id is absent

paritytech#13587 (comment)

fyi: I don't really know what the old behaviour was.

* update comment to reflect new defaults

Closes paritytech#13338

* Revert "update comment to reflect new defaults"

This reverts commit 7a981ab.

* remove config.rs (from wrong merge)

* upgrade to libp2p 0.51.2

* fix formatting

* use handle_pending_outbound_connection in networt_state RPC

* update deps

* use re-exports when we use other libp2p packages

* Apply suggestions from code review

Co-authored-by: Dmitry Markin <dmitry@markin.tech>

* format code

* handle potential errors in network_state RPC

* only update libp2p crate

* update libp2p-core

* fix docs

* use libp2p-identity instead of libp2p

where it's possible. libp2p-identity is much smaller, hence makes sense
to use it instead of larger libp2p crate.

* Update client/network/src/discovery.rs

Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>

* update Cargo.lock

* add comment for per_connection_event_buffer_size

current value is somewhat arbitrary and needs to be tweaked depending on
memory usage and network worker sleep stats.

* fix link format

* update Cargo.lock

* upgrade to libp2p 0.51.3

* deprecate mplex

* Revert "deprecate mplex"

This reverts commit 9e25820.

* Revert "upgrade to libp2p 0.51.3"

This reverts commit 6544dd4.

* use new libp2p version in `statement` crate

* pin version temporarily

* libp2p 0.51.3

* deprecate mplex

* deprecate legacy noise handshake

* fix build error

* update libp2p-identity

* enable libp2p-identity:ed25519 feature in sc-consensus

* enable ed25519 for peerset as well

---------

Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Co-authored-by: parity-processbot <>

* chore: update libp2p to 0.52.1 (paritytech#14429)

* update libp2p to 0.52.0

* proto name now must implement `AsRef<str>`

* update libp2p version everywhere

* ToSwarm, FromBehaviour, ToBehaviour

also LocalProtocolsChange and RemoteProtocolsChange

* new NetworkBehaviour invariants

* replace `Vec<u8>` with `StreamProtocol`

* rename ConnectionHandlerEvent::Custom to NotifyBehaviour

* remove DialError & ListenError invariants

also fix pending_events

* use connection_limits::Behaviour

See libp2p/rust-libp2p#3885

* impl `void::Void` for `BehaviourOut`

also use `Behaviour::with_codec`

* KademliaHandler no longer public

* fix StreamProtocol construction

* update libp2p-identify to 0.2.0

* remove non-existing methods from PollParameters

rename ConnectionHandlerUpgrErr to StreamUpgradeError

* `P2p` now contains `PeerId`, not `Multihash`

* use multihash-codetable crate

* update Cargo.lock

* reformat text

* comment out tests for now

* remove `.into()` from P2p

* confirm observed addr manually

See https://github.com/libp2p/rust-libp2p/blob/master/protocols/identify/CHANGELOG.md#0430

* remove SwarmEvent::Banned

since we're not using `ban_peer_id`, this can be safely removed.
we may want to introduce `libp2p::allow_block_list` module in the future.

* fix imports

* replace `libp2p` with smaller deps in network-gossip

* bring back tests

* finish rewriting tests

* uncomment handler tests

* Revert "uncomment handler tests"

This reverts commit 720a068.

* add a fixme

* update Cargo.lock

* remove extra From

* make void uninhabited

* fix discovery test

* use autonat protocols

confirming external addresses manually is unsafe in open networks

* fix SyncNotificationsClogged invariant

* only set server mode manually in tests

doubt that we need to set it on node since we're adding public addresses

* address @dmitry-markin comments

* remove autonat

* removed unused var

* fix EOL

* update smallvec and sha2

in attempt to compile polkadot

* bump k256

in attempt to build cumulus

---------

Co-authored-by: parity-processbot <>

* Bump libp2p to 0.52.3

* Bump libp2p to 0.52.4

---------

Co-authored-by: Anton <anton.kalyaev@gmail.com>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
U3-nice_to_have Issue is worth doing eventually.
Projects
Status: Done
Development

No branches or pull requests

1 participant