Fix the networking #364

tomaka · 2018-07-18T10:40:31Z

Fixes the most pressing issues with the networking:

Fixes a panic in IdentifyTransport.
Adds a 20 seconds timeout to connections. We try to dial nodes again later if an earlier connection fails.
Do not take into account connections that only have a Kademlia substream (and not a substrate substream) for min_peers/max_peers.
Print the error in the logs when a connection attempt fails.
Instead of connecting to the results of the Kademlia queries, we just pick random nodes from the peer store.
We immediately try to connect to random nodes from the peer store at startup. The first time a node is started, it will connect to the bootstrap nodes, but afterwards it will connect to nodes it learned about earlier.

svyatonik · 2018-07-18T11:09:24Z

substrate/network-libp2p/src/network_state.rs

+		if self.reserved_only.load(atomic::Ordering::Relaxed) {
+			0
+		} else {
+			self.min_peers - self.num_open_custom_connections()


This could fail with overflow, right?

gavofyork · 2018-07-18T11:53:17Z

substrate/network-libp2p/src/service.rs

+
+		trace!(target: "sub-libp2p", "Dialing bootnode {:?}", peer_id);
+		for proto in shared.protocols.read().0.clone().into_iter() {
+			open_peer_custom_proto(shared.clone(), transport.clone(),


style: long function calls should be one line per arg

gavofyork · 2018-07-18T11:54:19Z

substrate/network-libp2p/src/service.rs

-				process_kad_results(shared.clone(), transport.clone(),
-					swarm_controller.clone(), results, &local_peer_id);
+			move |_| {
+				connect_to_nodes(shared.clone(), transport.clone(),


style: this line is fine to avoid the join. otherwise it should be one line per arg, but that's overkill.

gavofyork · 2018-07-18T11:54:48Z

substrate/network-libp2p/src/service.rs

-				discovered_peer.clone(),
-				&swarm_controller
-			);
+			open_peer_custom_proto(shared.clone(), base_transport.clone(),


this run-on line would be fine to be all on that same line.

arkpar · 2018-07-18T11:54:49Z

substrate/network-libp2p/src/service.rs

+		// Choose a random peer. We are potentially already connected to
+		// this peer, but this is not a problem as this function is called
+		// regularly.
+		// TODO: is it ^ ?


Eventually this needs to be reworked to select from the peers that are not currently connected and are not being connected.
Also, the node could be running with more than a thousands slots, so it should not try to connect all of them at the same time.

gavofyork · 2018-07-18T11:55:08Z

substrate/network-libp2p/src/transport.rs

+
+	TransportTimeout::new(base, Duration::from_secs(20))
+		.map_err(|err| {
+			debug!(target: "sub-libp2p", "Error in base transport \


unneeded run-on line.

gavofyork

some style nits, but they can be fixed later

* Crude attempt at finding open ports (paritytech#346) * added example test for frontier (paritytech#343) * added example test for frontier * added npm run non-ci-test * move staking test spec to node/chainspec (paritytech#335) * init * hide staking test spec behind test-staking feature flag * clean * comment staking test genesis config for readability * move test spec to separate file * Crude attempt at finding open ports * npx -w * Add missing file * Adds random port and parallel test execution * Fixed ethers test * Adds delay to node start in test Co-authored-by: Antoine Estienne <estienne.antoine@gmail.com> Co-authored-by: Amar Singh <asinghchrony@protonmail.com> Co-authored-by: Crystalin <alan@purestake.com> * Crystalin testweek watch (paritytech#347) * Crude attempt at finding open ports * npx -w * Add missing file * Adds random port and parallel test execution * Fixed ethers test * Adds delay to node start in test * Adds support for test watch * Cleaned package.json * Prevent test to run at start of watch * Restore optimization for cargo build Co-authored-by: Stephen Shelton <steve@brewcraft.org> * Unskip tests (paritytech#357) * Unskip `fetch genesis block by hash` * Remove timeouts * Unskip block gas limit tests * Remove `opt-level = 0` from Cargo.toml * Unskip * Test contract factory (paritytech#351) * added deployContractByName and contractSources * added getcompiled function * increase timeout for testfilterapi * change contract name and add sourcecode * Fix fixture requirement * update package and json Co-authored-by: Crystalin <alan@purestake.com> * Gorka remove intra test dependencies (paritytech#355) * Crude attempt at finding open ports * npx -w * Add missing file * Adds random port and parallel test execution * Fixed ethers test * Adds delay to node start in test * Adds support for test watch * Cleaned package.json * Prevent test to run at start of watch * Restore optimization for cargo build * Allow for non-finalization and ParentHash specification in createAndFinalizeBlock * Make test-block tests non-dependent by using non-finalized chains * Separate tests to make them non-dependent and parametrize tx hashes and nonces in test-txpool * it instead of step in test-version * setup providers before tests in test-ethers.ts * Create a block before tests to avoid one depend on the other in test-nonce * it for step in test-receipt * import constants for test-staking * Attemp at parametrizing test-filter-api tests * attemp at parametrizing test-revert-receipt * remove mocha-steps from tests * make test-subscription tests non-dependent * comply with prettier * Add support for retrieving the hash of the generated block * Make test-trace-filter tests non-dependent * balance-tests non-dependent * Non-dependent tests for test-polkadot-api * Add test-fork-chain tests * remove unnecesary timeouts * Use context provider for test-ether.ts * Remove currentId from test-filter-api since we are uninstalling all filters from test to test * Join balance test setups and check balance at each block * Rename variables to camelCase in tests * small introduced bug in test-pool-pending when camelCasing variables * Use a different account for test-revert-receipt deployment contract to avoid non-determinism of the nonce * shorten line because of editconfig * Separate test-subscription and test-subscription-past-events.ts * remove single letter variable in test-filter-api * Small cosmetic changes * revert changes in package.json * Remove beforeEach and substitute it for common function * change var for let or const Co-authored-by: Stephen Shelton <steve@brewcraft.org> Co-authored-by: Crystalin <alan@purestake.com> * Crystalin remove solc (paritytech#362) * Crude attempt at finding open ports * npx -w * Add missing file * Adds random port and parallel test execution * Fixed ethers test * Adds delay to node start in test * Adds support for test watch * Cleaned package.json * Prevent test to run at start of watch * Restore optimization for cargo build * Allow for non-finalization and ParentHash specification in createAndFinalizeBlock * Make test-block tests non-dependent by using non-finalized chains * Separate tests to make them non-dependent and parametrize tx hashes and nonces in test-txpool * it instead of step in test-version * setup providers before tests in test-ethers.ts * Create a block before tests to avoid one depend on the other in test-nonce * it for step in test-receipt * import constants for test-staking * Attemp at parametrizing test-filter-api tests * attemp at parametrizing test-revert-receipt * remove mocha-steps from tests * make test-subscription tests non-dependent * comply with prettier * Add support for retrieving the hash of the generated block * Make test-trace-filter tests non-dependent * balance-tests non-dependent * Non-dependent tests for test-polkadot-api * Add test-fork-chain tests * remove unnecesary timeouts * Use context provider for test-ether.ts * Remove currentId from test-filter-api since we are uninstalling all filters from test to test * Join balance test setups and check balance at each block * Rename variables to camelCase in tests * small introduced bug in test-pool-pending when camelCasing variables * Use a different account for test-revert-receipt deployment contract to avoid non-determinism of the nonce * shorten line because of editconfig * Separate test-subscription and test-subscription-past-events.ts * remove single letter variable in test-filter-api * Small cosmetic changes * revert changes in package.json * Remove beforeEach and substitute it for common function * change var for let or const * Removes solc from tests * Fixes path for contract tests * Adds formatting for compiled contracts Co-authored-by: Stephen Shelton <steve@brewcraft.org> Co-authored-by: Gorka Irazoqui <gorka.irazoki@gmail.com> * Typescript test refactoring (paritytech#364) * Crude attempt at finding open ports * npx -w * Add missing file * Adds random port and parallel test execution * Fixed ethers test * Adds delay to node start in test * Adds support for test watch * Cleaned package.json * Prevent test to run at start of watch * Restore optimization for cargo build * Allow for non-finalization and ParentHash specification in createAndFinalizeBlock * Make test-block tests non-dependent by using non-finalized chains * Separate tests to make them non-dependent and parametrize tx hashes and nonces in test-txpool * it instead of step in test-version * setup providers before tests in test-ethers.ts * Create a block before tests to avoid one depend on the other in test-nonce * it for step in test-receipt * import constants for test-staking * Attemp at parametrizing test-filter-api tests * attemp at parametrizing test-revert-receipt * remove mocha-steps from tests * make test-subscription tests non-dependent * comply with prettier * Add support for retrieving the hash of the generated block * Make test-trace-filter tests non-dependent * balance-tests non-dependent * Non-dependent tests for test-polkadot-api * Add test-fork-chain tests * remove unnecesary timeouts * Use context provider for test-ether.ts * Remove currentId from test-filter-api since we are uninstalling all filters from test to test * Join balance test setups and check balance at each block * Rename variables to camelCase in tests * small introduced bug in test-pool-pending when camelCasing variables * Use a different account for test-revert-receipt deployment contract to avoid non-determinism of the nonce * shorten line because of editconfig * Separate test-subscription and test-subscription-past-events.ts * remove single letter variable in test-filter-api * Small cosmetic changes * revert changes in package.json * Remove beforeEach and substitute it for common function * change var for let or const * Removes solc from tests * Fixes path for contract tests * Adds formatting for compiled contracts * Refactor tests * Fix formatting * Limit concurrency to half cpus for ts tests * Fixes github action typo Co-authored-by: Stephen Shelton <steve@brewcraft.org> Co-authored-by: Gorka Irazoqui <gorka.irazoki@gmail.com> * Antoine update testweek readme typos (paritytech#369) * added instructions to readme * update package lock * Add missing contracts to definition (paritytech#368) * Add Callee, Caller, Incrementer * Update test-trace * Remove unused * Move blockscout tracer to util/tracer * Test for txpool multiple transactions (paritytech#367) * Improve pending pool test * Rephrase comment * Better naming for txpool tests * Split txpool multiple test for more independance * Fixes few test expectations * Adds test for future transaction * Rename eth pool correctly * Fixes node listeners Co-authored-by: tgmichel <telmo@purestake.com> * Fixes typo in variable name * Added test for block gas in smart contract (paritytech#370) * VSCode debugger (support for TS tests) (paritytech#348) Add a VSCode debugger config with CodeLLDB. TS tests can be debugged by first starting the node in the debugger, then launching DEBUG_MODE=true npm run test-single in the test folder (modify package.json to change the test file). * Adds test for filter trace pagination (paritytech#373) * Fixes trace filter test Co-authored-by: Stephen Shelton <steve@brewcraft.org> Co-authored-by: Antoine Estienne <estienne.antoine@gmail.com> Co-authored-by: Amar Singh <asinghchrony@protonmail.com> Co-authored-by: tgmichel <telmo@purestake.com> Co-authored-by: girazoki <gorka.irazoki@gmail.com> Co-authored-by: nanocryk <6422796+nanocryk@users.noreply.github.com>

Upgrade Substrate to 2022-04-15 snapshot

tomaka added 6 commits July 18, 2018 09:43

Serve only non-empty Kademlia nodes

6216799

Use the number of custom protos to determine whether to open more

8f14944

Add timeout when connecting

6448c3e

Connect to random peers from the peer store

0506401

Various adjustements

4385b81

Typo

065f3f8

tomaka added A0-please_review Pull request needs code review. M4-core labels Jul 18, 2018

tomaka mentioned this pull request Jul 18, 2018

libp2p networking seemingly ignores bootnodes #361

Closed

svyatonik reviewed Jul 18, 2018

View reviewed changes

tomaka added 2 commits July 18, 2018 13:15

Explicitely connect to bootnodes

1e41844

Fix potential overflow

93a39a9

gavofyork reviewed Jul 18, 2018

View reviewed changes

arkpar reviewed Jul 18, 2018

View reviewed changes

arkpar added A6-mustntgrumble and removed A0-please_review Pull request needs code review. labels Jul 18, 2018

gavofyork reviewed Jul 18, 2018

View reviewed changes

gavofyork approved these changes Jul 18, 2018

View reviewed changes

gavofyork merged commit fc126da into paritytech:master Jul 18, 2018

tomaka deleted the fix-networking branch July 18, 2018 11:57

tomaka added a commit to tomaka/polkadot that referenced this pull request Jul 18, 2018

Fix style in paritytech#364

a058a5f

tomaka added a commit to tomaka/polkadot that referenced this pull request Jul 18, 2018

Fix style in paritytech#364

432eb41

tomaka added a commit to tomaka/polkadot that referenced this pull request Jul 18, 2018

Fix style in paritytech#364

f65ddf9

arkpar pushed a commit that referenced this pull request Jul 18, 2018

Fix style in #364 (#365)

2eb32a3

liuchengxu pushed a commit to autonomys/substrate that referenced this pull request Jun 3, 2022

Merge pull request paritytech#364 from subspace/substrate-upgrade

c691a2d

Upgrade Substrate to 2022-04-15 snapshot

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix the networking #364

Fix the networking #364

tomaka commented Jul 18, 2018 •

edited

Loading

svyatonik Jul 18, 2018 •

edited

Loading

gavofyork Jul 18, 2018

gavofyork Jul 18, 2018

gavofyork Jul 18, 2018

arkpar Jul 18, 2018

gavofyork Jul 18, 2018

gavofyork left a comment

Fix the networking #364

Fix the networking #364

Conversation

tomaka commented Jul 18, 2018 • edited Loading

svyatonik Jul 18, 2018 • edited Loading

Choose a reason for hiding this comment

gavofyork Jul 18, 2018

Choose a reason for hiding this comment

gavofyork Jul 18, 2018

Choose a reason for hiding this comment

gavofyork Jul 18, 2018

Choose a reason for hiding this comment

arkpar Jul 18, 2018

Choose a reason for hiding this comment

gavofyork Jul 18, 2018

Choose a reason for hiding this comment

gavofyork left a comment

Choose a reason for hiding this comment

tomaka commented Jul 18, 2018 •

edited

Loading

svyatonik Jul 18, 2018 •

edited

Loading