Increase global peer coverage

[TangMonk]

It's hard to looking peers:

base-geth-1  | INFO [03-25|07:56:33.173] Looking for peers                        peercount=0 tried=78  static=0
base-geth-1  | INFO [03-25|07:56:53.511] Looking for peers                        peercount=1 tried=79  static=0
base-geth-1  | INFO [03-25|07:57:03.647] Looking for peers                        peercount=0 tried=148 static=0
base-geth-1  | INFO [03-25|07:57:13.685] Looking for peers                        peercount=0 tried=114 static=0
base-geth-1  | INFO [03-25|07:57:23.717] Looking for peers                        peercount=0 tried=138 static=0
base-geth-1  | INFO [03-25|07:57:33.748] Looking for peers                        peercount=0 tried=92  static=0
base-geth-1  | INFO [03-25|07:57:43.799] Looking for peers                        peercount=0 tried=175 static=0
base-geth-1  | INFO [03-25|07:57:53.872] Looking for peers                        peercount=0 tried=139 static=0
base-geth-1  | INFO [03-25|07:58:04.108] Looking for peers                        peercount=0 tried=105 static=0
base-geth-1  | INFO [03-25|07:58:14.178] Looking for peers                        peercount=0 tried=117 static=0
base-geth-1  | INFO [03-25|07:58:24.263] Looking for peers                        peercount=0 tried=138 static=0
base-geth-1  | INFO [03-25|07:58:34.396] Looking for peers                        peercount=0 tried=115 static=0
base-geth-1  | INFO [03-25|07:58:44.417] Looking for peers                        peercount=0 tried=104 static=0
base-geth-1  | INFO [03-25|07:58:54.432] Looking for peers                        peercount=0 tried=89  static=0
base-geth-1  | INFO [03-25|07:59:04.459] Looking for peers                        peercount=0 tried=102 static=0
base-geth-1  | INFO [03-25|07:59:14.540] Looking for peers                        peercount=0 tried=145 static=0
base-geth-1  | INFO [03-25|07:59:24.541] Looking for peers                        peercount=0 tried=72  static=0

Very long time, still 0 peer

[mariokami]

I'm experiencing the same issue. I haven't dug into the architecture of the project (apologies) but it seems that 9222 tcp/udp are the discovery ports for p2p.

I've opened these up on the node but I'm still experiencing the same issue as @TangMonk - long periods of 0 peers.

[roberto-bayardo]

Thanks, will take a look. We set up some default public peers but perhaps they don't have enough capacity.

As for the question in the title of this issue, yes we have the static peers that should be provided by the bootnodes in the default config here. This is a recent update, so make sure your config has this too:

node/.env.mainnet

Line 3 in 18a9591

OP_GETH_BOOTNODES=enode://87a32fd13bd596b2ffca97020e31aef4ddcc1bbd4b95bb633d16c1329f654f34049ed240a36b449fda5e5225d70fe40bc667f53c304b71f8e68fc9d448690b51@3.231.138.188:30301,enode://ca21ea8f176adb2e229ce2d700830c844af0ea941a1d8152a9513b966fe525e809c3a6c73a2c18a12b74ed6ec4380edf91662778fe0b79f6a591236e49e176f9@184.72.129.189:30301,enode://acf4507a211ba7c1e52cdf4eef62cdc3c32e7c9c47998954f7ba024026f9a6b2150cd3f0b734d9c78e507ab70d59ba61dfe5c45e1078c7ad0775fb251d7735a2@3.220.145.177:30301,enode://8a5a5006159bf079d06a04e5eceab2a1ce6e0f721875b2a9c96905336219dbe14203d38f70f3754686a6324f786c2f9852d8c0dd3adac2d080f4db35efc678c5@3.231.11.52:30301,enode://cdadbe835308ad3557f9a1de8db411da1a260a98f8421d62da90e71da66e55e98aaa8e90aa7ce01b408a54e4bd2253d701218081ded3dbe5efbbc7b41d7cef79@54.198.153.150:30301

[wbnns]

@TangMonk @mariokami Can you all also please make sure TCP / UDP for P2P is forwarded for 30303 and let us know the results?

[TangMonk]

@TangMonk* @mariokami* Can you all also please make sure TCP / UDP for P2P is forwarded for 30303 and let us know the results?

$ sudo docker ps
CONTAINER ID   IMAGE                         COMMAND                  CREATED             STATUS                 PORTS                                                                                                                                                                                                                   NAMES
2c85eb19d42b   base-node                     "bash ./op-node-entr…"   4 hours ago         Up 4 hours             0.0.0.0:6060->6060/tcp, :::6060->6060/tcp, 0.0.0.0:7300->7300/tcp, :::7300->7300/tcp, 0.0.0.0:9222->9222/tcp, :::9222->9222/tcp, 0.0.0.0:9222->9222/udp, :::9222->9222/udp, 0.0.0.0:7545->8545/tcp, :::7545->8545/tcp   base-node-1
add7fe3cae97   base-geth                     "bash ./geth-entrypo…"   4 hours ago         Up About a minute      0.0.0.0:8545-8546->8545-8546/tcp, :::8545-8546->8545-8546/tcp, 0.0.0.0:30303->30303/tcp, :::30303->30303/tcp, 0.0.0.0:30303->30303/udp, :::30303->30303/udp, 0.0.0.0:7301->6060/tcp, :::7301->6060/tcp                  base-geth-1

and no firewall enabled on my machine

[TangMonk]

Thanks, will take a look. We set up some default public peers but perhaps they don't have enough capacity.

As for the question in the title of this issue, yes we have the static peers that should be provided by the bootnodes in the default config here. This is a recent update, so make sure your config has this too:

node/.env.mainnet

Line 3 in 18a9591

OP_GETH_BOOTNODES=enode://87a32fd13bd596b2ffca97020e31aef4ddcc1bbd4b95bb633d16c1329f654f34049ed240a36b449fda5e5225d70fe40bc667f53c304b71f8e68fc9d448690b51@3.231.138.188:30301,enode://ca21ea8f176adb2e229ce2d700830c844af0ea941a1d8152a9513b966fe525e809c3a6c73a2c18a12b74ed6ec4380edf91662778fe0b79f6a591236e49e176f9@184.72.129.189:30301,enode://acf4507a211ba7c1e52cdf4eef62cdc3c32e7c9c47998954f7ba024026f9a6b2150cd3f0b734d9c78e507ab70d59ba61dfe5c45e1078c7ad0775fb251d7735a2@3.220.145.177:30301,enode://8a5a5006159bf079d06a04e5eceab2a1ce6e0f721875b2a9c96905336219dbe14203d38f70f3754686a6324f786c2f9852d8c0dd3adac2d080f4db35efc678c5@3.231.11.52:30301,enode://cdadbe835308ad3557f9a1de8db411da1a260a98f8421d62da90e71da66e55e98aaa8e90aa7ce01b408a54e4bd2253d701218081ded3dbe5efbbc7b41d7cef79@54.198.153.150:30301

All above of your provided static peers node are from USA. Not convenient for people from other areas

Could u please add some Asian static peers node, Hong Kong, Singapore, Japan is fine. I am living in China, Syncing node is so sloooow.

[wbnns]

Thanks for the feedback; leaving this issue open to track the need for increasing global peer coverage to facilitate easier discovery.

[mariokami]

@wbnns thanks for the response - 30303 is forwarded on this box, and I'll keep you posted with updates. I wasn't thinking about 30303 because it says in the docker-compose.yml file :

      - 30403:30303     # P2P TCP (currently unused)
      - 30403:30303/udp # P2P UDP (currently unused)

@TangMonk the bootnodes are just for discovery (ie: to provide addresses for other nodes via Kademlia DHT) so their geographic location shouldn't make too much of a difference. I'd be interested if there was something to suggest otherwise.

[rcastellaj]

Is it 30303 (conflicting with Ethereum) or is it 30301 ? The static bootnodes list has 30301 for all their ports...

[parsdextra]

Are the boot nodes still overloaded? Same issue here with no connected nodes, no firewall.

[rcastellaj]

Took about 1 hour for my node to find one peer, after that the whole chain sync took 1 hour as my L1 beacon and exec are local to the infrastructure.

To be fair it kind of feels like you dont need more than 1 peer to start syncing. Would it be too hard to provide a reliable list of 10 or so static nodes?

[lmcapp]

My server is in Germany, and all provided static nodes cannot be pinged. Looking for a peer for more than a day and the result is still 0.

[mariokami]

The boot nodes don't (shouldn't) participate in peering iirc - they just provide pointers to additional peers via Kademlia DHT lookups.

If anyone has a moment and is interested there is Nebula which can crawl DHT networks for analysis. It would be interesting to see what the properties of the Base network are like.

[roberto-bayardo]

We do provide a set of nodes available for peering, but apparently not enough. We will look into increasing capacity.

[smowden]

been running for days without being able to find a single peer

[pikansj0s]

We do provide a set of nodes available for peering, but apparently not enough. We will look into increasing capacity.

So.... any new peers?

[roberto-bayardo]

We cranked up our snapshot peering this week, let me know if you still have issues.

[MindlessSteel]

I thought this was the basic goal of base eth.

[rcastellaj]

We cranked up our snapshot peering this week, let me know if you still have issues.

Yes, it's still an issue. I've been trying to resync after updating to .4 for about 4 days now. Its stuck at peercount=2 but makes no progress.

[roberto-bayardo]

We have added one more machine with additional 500 peer handling capacity, though I am not sure if that will help, as the issue may be discovery related. Still digging into this.

One thing I found that helps with peer connectivity is to make sure inbound connections to 30303 are open, and you specify your external IP address appropriately (geth --nat=extip:[your external ip address]). Is that something your setup would allow?

[rcastellaj]

We have added one more machine with additional 500 peer handling capacity, though I am not sure if that will help, as the issue may be discovery related. Still digging into this.

One thing I found that helps with peer connectivity is to make sure inbound connections to 30303 are open, and you specify your external IP address appropriately (geth --nat=extip:[your external ip address]). Is that something your setup would allow?

Wait, why 30303 ? That's Ethereum.

[roberto-bayardo]

right op-geth is just a fork of geth, so it defaults to 30303 for p2p. You can override it to whatever port you prefer with the --port flag however.

[rcastellaj]

now im in a situation where my chain finds 1-2 peers quick enough but it never syncs. just stays in looking for peers. why?

[rcastellaj]

INFO [06-17|11:44:14.386] Looking for peers                        peercount=4 tried=95  static=0
INFO [06-17|11:44:24.467] Looking for peers                        peercount=4 tried=167 static=0
INFO [06-17|11:44:34.568] Looking for peers                        peercount=4 tried=160 static=0
INFO [06-17|11:44:44.766] Looking for peers                        peercount=4 tried=115 static=0
INFO [06-17|11:44:54.796] Looking for peers                        peercount=4 tried=156 static=0
INFO [06-17|11:45:04.845] Looking for peers                        peercount=4 tried=133 static=0

I'm here, and it's not syncing. Doesn't matter if I start with an empty data dir or with a snapshot. Using the 0.8.4 release.

[roberto-bayardo]

Can you try opening inbound connections to port 30303 and specifying the HOST_IP appropriately?

[rcastellaj]

Both are done and it's not starting the sync

[roberto-bayardo]

Can you confirm you can "telnet <HOST_IP> 30303" (or whatever port you're using) to the node from an outside machine? I am having trouble replicating so wondering if your port is still blocked somehow. We're still working on allowing more peers to be found even if the port is blocked. Thanks for your patience.

[rcastellaj]

I can confirm, yes. It's a kube cluster but the NodePort 30303 is being proxied through to the box, and the HOST_IP indeed can be telnet'd into 30303 from an external machine, and it arrives at the right pod. Also I dont have issues finding peers. It's just not starting the sync.

[rcastellaj]

I have 2 peers already within 2 minutes of restarting. Now running 0.8.3 I think. Same old.

[roberto-bayardo]

Would you mind sharing your enode id? (Feel free to e-mail...) I can see if I can peer my node with it and see if that gets it syncing.

[rcastellaj]

I also have 100+ peers connected now but it's not trying to catch up the chain. Can you advise of current configuration that might make this weird?

[roberto-bayardo]

Interesting so now peering is better but still no sync. assuming you already uncomment the flags here?

node/.env.mainnet

Line 41 in 2dbf4a4

# SNAP SYNC

Is your config otherwise standard?

[rcastellaj]

Ah, no, I didn't realize there were new envs there, and we are supplying them via a cm so didn't pick up the file automatically. Will try with that and report back. Thanks.

[rcastellaj]

@roberto-bayardo yea, even with 0.9.0, the updated env, testing changing between local L1 rpc/beacon to a chainstack archive one, it still doesn't start syncing.

data dir is wiped, it says starting snap sync mode and that the database is empty, but up to 10 nodes now and it doesn't start any sync -- what am I missing? :(

[walkerlala]

I am having the same issue here. Long period of 'peercount=0' . Already changed the 'HOST_IP' for NAT (though I have a public ip) and added bootnodes.

[walkerlala]

Can anyone share more bootnodes address here?

[roberto-bayardo]

Can anyone share more bootnodes address here?

node/.env.mainnet

Line 45 in 8e7460e

# OP_GETH_BOOTNODES=enode://87a32fd13bd596b2ffca97020e31aef4ddcc1bbd4b95bb633d16c1329f654f34049ed240a36b449fda5e5225d70fe40bc667f53c304b71f8e68fc9d448690b51@3.231.138.188:30301,enode://ca21ea8f176adb2e229ce2d700830c844af0ea941a1d8152a9513b966fe525e809c3a6c73a2c18a12b74ed6ec4380edf91662778fe0b79f6a591236e49e176f9@184.72.129.189:30301,enode://acf4507a211ba7c1e52cdf4eef62cdc3c32e7c9c47998954f7ba024026f9a6b2150cd3f0b734d9c78e507ab70d59ba61dfe5c45e1078c7ad0775fb251d7735a2@3.220.145.177:30301,enode://8a5a5006159bf079d06a04e5eceab2a1ce6e0f721875b2a9c96905336219dbe14203d38f70f3754686a6324f786c2f9852d8c0dd3adac2d080f4db35efc678c5@3.231.11.52:30301,enode://cdadbe835308ad3557f9a1de8db411da1a260a98f8421d62da90e71da66e55e98aaa8e90aa7ce01b408a54e4bd2253d701218081ded3dbe5efbbc7b41d7cef79@54.198.153.150:30301

[ssnickolay]

I had the same issue and solved it by deleting everything from geth folder after unzipping the snapshot archive (except chaindata folder)

root@full-2tb ~/base/blockchain # ls geth
chaindata  LOCK  nodekey  nodes  transactions.rl

I believe the root issue is nodekey which is the same for all of the users who downloaded the official snapshot => peers rejected connection since the key already exists in the peer network.

If you have trouble with downloading the state (after reaching some peers) please check local optimism node logs (op-node); In my case, I forgot to change OP_NODE_L1_ETH_RPC and quickly reached the limits:

lvl=warn msg="Engine temporary error" err="temp: failed to find L1 block info by number, at origin 0xe125aab359435b0f224d3ab6f9603201aae967c95821bd9b715cedb8f13349b5:20625586 next 20625587: failed to fetch header by num 20625587: Exceeded the quota usage"

[Creamers158]

@roberto-bayardo yea, even with 0.9.0, the updated env, testing changing between local L1 rpc/beacon to a chainstack archive one, it still doesn't start syncing.

data dir is wiped, it says starting snap sync mode and that the database is empty, but up to 10 nodes now and it doesn't start any sync -- what am I missing? :(

Did you find a solution? Got the exact same here. Peers but no blocks.

*EDIT: on my end it appeared that my rpc/beacon endpoint did have all blobs. Used a 3rd party rpc for both and it started syncing again.

[roberto-bayardo]

I had the same issue and solved it by deleting everything from geth folder after unzipping the snapshot archive (except chaindata folder)

root@full-2tb ~/base/blockchain # ls geth
chaindata  LOCK  nodekey  nodes  transactions.rl

I believe the root issue is nodekey which is the same for all of the users who downloaded the official snapshot => peers rejected connection since the key already exists in the peer network.

Thank you for pointing this out, we have fixed this with our snapshots.

Next up we're going to make the peer IDs of our snap-sync capable nodes persistent so they can be hard coded in a config.