Node synchronization slow #325

yqZhang711 · 2021-07-26T08:55:52Z

Node synchronization is slow, always more than 2000 blocks away from the block browser, what is the solution?

yqZhang711 · 2021-07-26T09:24:04Z

@fjl @guagualvcha @j75689 Node synchronization is slow, always more than 2000 blocks away from the block browser, what is the solution?

wenchaopeng · 2021-07-26T14:01:58Z

@yqZhang711 me too,how is going of your side?

vae520283995 · 2021-07-26T14:58:58Z

In the past two days, it is often found that the synchronization cannot catch up with the block generation

vietthang207 · 2021-07-26T15:36:37Z

Me too, I am using v1.0.7-ht.3-deploy

yqZhang711 · 2021-07-26T16:04:54Z

@yqZhang711 me too,how is going of your side?

I'm using the latest node (V1.1.0), but it's still unresolved！

yqZhang711 · 2021-07-26T16:08:11Z

In the past two days, it is often found that the synchronization cannot catch up with the block generation

Me too, I started yesterday, node synchronization speed is slower than block speed! Resulting in more and more block height difference

vae520283995 · 2021-07-26T16:35:15Z

@guagualvcha need your help

luoxiandong · 2021-07-27T01:35:59Z

me too, I am using v1.1.0

wo4zhuzi · 2021-07-27T01:37:13Z

Me too. I need the minimum conditions to catch up with the block.

such as:
cpu
disk
Memory

wenchaopeng · 2021-07-27T02:05:41Z

my node is still 1237 blocks behind, and the synchronization speed cannot keep up with the block generation speed

jun0tpyrc · 2021-07-27T04:47:26Z

Me too. I need the minimum conditions to catch up with the block.

such as:
cpu
disk
Memory

full mode: (with heavy rpc, 4 cores won't suffice) 8-core-cpu + nvme

archive mode: no idea, bottleneck seems on p2p for head blocks, raid0 on aws gp3 ssd can catch up for old blocks but not from last day, running 16-core-cpu machiness

ghost · 2021-07-27T05:16:15Z

Still can't catch the latest height with about 100 blocks. my node With 8-core + 32g + 640G nvme

vshybalov · 2021-07-27T14:09:25Z

Had the same issue after update to the 1.1.0-7822e9e26, the resolution was to add the --txlookuplimit=0 option while starting the daemon. Seems like after update, it tried to do some unindexing job which, slowed down the sync.

Once I added the option, after approximately one hour the sync started to run faster and was able to complete it.

dracmic · 2021-07-28T08:35:32Z

Taking in consideration that I have similar issue even with latest 1.1.0 release by using --txlookuplimit=0 ... I still have the issue with 100 blocks behind in fast syncmode ... I don't know what happen but this is happening from last week. I tried 2-3 different server providers all with NVME and a lot of RAM and different location same issue: eth.blockNumber on 0 and pivot getting stalled so I get the issue with 3 minutes back blocks (~100blocks).

vae520283995 · 2021-07-28T08:37:40Z

I am already 5,000 blocks behind

zy2d · 2021-07-28T09:17:25Z

from 1.0.6 update to 1.1.0 but 5719 blocks behind
--txlookuplimit=0 is default and syncmode full and fast has try but nothing

j75689 · 2021-07-28T09:32:17Z

In the past few days, TPS has grown very fast, and the original hardware performance may not be able to meet the current speed.

We have also increased GasLimit to increase TPS, which may cause the block size to become larger, resulting in synchronization difficulties.

We provide several operations for reference:

Use the latest binary version.
Prune the data to reduce the amount of data or download the snapshot we provide.
Use the recommended AWS m5zn.3xlarge.

We will discuss how to improve synchronization performance asap.
If the community has any good ideas, you are welcome to submit your ideas.

Thanks all

dracmic · 2021-07-28T09:53:10Z

@j75689 I am using: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz with 18000 cache on running command (32 GB ram total ram), 2 x NVME Samsung put in raid0 ( to increase space) still no luck .... to get synced ... Also I've used a VPS with 10 core of AMD EPYC 7282 16-Core Processor with 60GB of total system RAM and same cache setting I suppose NVME SSD which worked ... I have enough bandwithd. So can you suggest some settings ...

dwjpeng · 2021-07-28T09:54:37Z

In the past few days, TPS has grown very fast, and the original hardware performance may not be able to meet the current speed.

We have also increased GasLimit to increase TPS, which may cause the block size to become larger, resulting in synchronization difficulties.

We provide several operations for reference:

Use the latest binary version.

Prune the data to reduce the amount of data or download the snapshot we provide.

Use the recommended AWS m5zn.3xlarge.

We will discuss how to improve synchronization performance asap.
If the community has any good ideas, you are welcome to submit your ideas.

Thanks all

Geth
Version: 1.1.0
Git Commit: 7822e9e
Architecture: amd64
Go Version: go1.16.3
Operating System: ubuntu 18.04(alyun),
storage: essd 2.0T
net: 100M/S
cpu: 16C

syncing slower three days ago, I tried different syncmodes like snap and fast, it didn't work. now 9000 blocks behind.

seonggwonyoon · 2021-07-29T04:45:29Z

https://github.com/binance-chain/docs-site/commit/db2b4be7a7c784ff3c19f3e3679f1f0cb0b5a42e

The recommended system requirements for BSC nodes have been updated today.
see this commit

jun0tpyrc · 2021-07-29T04:56:35Z

binance-chain/docs-site@db2b4be

The recommended system requirements for BSC nodes have been updated today.
see this commit

could we add suggestion/estimation for archive node requirement too? thanks

jun0tpyrc · 2021-07-29T05:17:24Z

seems won't be enough to let aws gp3 catch up gap using full node full syncmode as suggested in doc, it's finalising slower than new block

c5.4xlarge
--syncmode=full --gcmode=full --snapshot=false --txlookuplimit=0 --cache.preimages
ebs gp3 2200GB  10000 iops 1000 MB/s

any tunings at filesystem / os level?
i think we still need nvme for any syncmode

dracmic · 2021-07-29T07:37:28Z

@j75689 Hey I have suggestion to talk with the guys from CryptoBlades (SKILL) guys which are doing nft fights and transactions on blockchain maybe they can move the fight calls :) https://explorer.bitquery.io/bsc/calls (https://explorer.bitquery.io/bsc/smart_contract/0x39bea96e13453ed52a734b6aceed4c41f57b2271/methods)

For me new issue at last 2-3 minutes of full sync:
t=2021-07-29T07:16:03+0000 lvl=info msg="Sidechain written to disk" start=9,568,446 end=9,568,446 sidetd=19,060,848 localtd=19,061,334
t=2021-07-29T07:16:03+0000 lvl=info msg="Sidechain written to disk" start=9,568,446 end=9,568,446 sidetd=19,060,849 localtd=19,061,334
t=2021-07-29T07:16:03+0000 lvl=info msg="Sidechain written to disk" start=9,568,448 end=9,568,448 sidetd=19,060,853 localtd=19,061,334
t=2021-07-29T07:16:03+0000 lvl=info msg="Sidechain written to disk" start=9,568,448 end=9,568,448 sidetd=19,060,852 localtd=19,061,334
t=2021-07-29T07:16:03+0000 lvl=info msg="Sidechain written to disk" start=9,568,450 end=9,568,450 sidetd=19,060,857 localtd=19,061,334
t=2021-07-29T07:16:03+0000 lvl=info msg="Sidechain written to disk" start=9,568,451 end=9,568,451 sidetd=19,060,859 localtd=19,061,334

kgcdream2019 · 2021-07-30T02:45:37Z

I have same issue, blocks are behind now 51750 ( 1 days 22hr)

kgcdream2019 · 2021-07-30T02:46:24Z

my vps is CPU 36 RAM 72 gp3 10000 IOPS 500 MB/S

seonggwonyoon · 2021-07-30T04:02:14Z

I've never had a slow block sync.
I am running 2 BSC Full Nodes.
Prerequisite is only running bsc
One is running on a dedicated server, and the other is running on AWS.

Here is my system resource and running script.
Please note

Dedicated Server

CPU: i7-9700
RAM: 64GB
DISK: RAID 0 (Samsung SSD 870, Samsung SSD 860)
1Gbps Network Speed

AWS EC2

Instance: c5.4xlarge
GP3 1800GB 6000/IOPS 500/Mbps(Throughput)
Use seoul region (ap-northeast-2)

No tunings at filesystem or os level.

And my start script is

Dedicated Server (geth version: 1.1.0 beta)

$ ./geth_linux --config ./config.toml --datadit ./node --cache 18000 --rpc.allow-unprotected-txs --txlookuplimit 0

AWS EC2 (geth version: 1.1.0)

$ ./geth_linux --config ./config.toml --datadit ./node --cache 18000 --rpc.allow-unprotected-txs --txlookuplimit 0

jun0tpyrc · 2021-07-30T04:20:10Z

We run like 15 bsc nodes, my observations being

p2p is slower for cold start, taking 2-3 hour to resume efficiency for finalising block at comparable speed before restart , even for old blocks
maybe you need to have fast and quality peer list which are in sync to catch up fast
node slow down when serving heavy rpc requests

the result is , we run some syncmode=fast/full mode on i3en.2xlarge 8C 64GB nvme, seeing some of them failing to catch up the gap, not shortening

jayboy-mabushi · 2021-07-30T05:12:02Z

did anyone manage to sync?

j75689 · 2021-07-30T08:11:19Z

Hi everyone, could you please move to #338 to discuss this issue?

The recent large number of transactions have caused a lot of problems, and we are also working hard to improve performance.

We will continue to follow issue #338.

j75689 closed this as completed Aug 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Node synchronization slow #325

Node synchronization slow #325

yqZhang711 commented Jul 26, 2021

yqZhang711 commented Jul 26, 2021

wenchaopeng commented Jul 26, 2021

vae520283995 commented Jul 26, 2021

vietthang207 commented Jul 26, 2021

yqZhang711 commented Jul 26, 2021

yqZhang711 commented Jul 26, 2021

vae520283995 commented Jul 26, 2021

luoxiandong commented Jul 27, 2021

wo4zhuzi commented Jul 27, 2021

wenchaopeng commented Jul 27, 2021

jun0tpyrc commented Jul 27, 2021

ghost commented Jul 27, 2021

vshybalov commented Jul 27, 2021 •

edited

Loading

dracmic commented Jul 28, 2021

vae520283995 commented Jul 28, 2021

zy2d commented Jul 28, 2021

j75689 commented Jul 28, 2021

dracmic commented Jul 28, 2021

dwjpeng commented Jul 28, 2021

seonggwonyoon commented Jul 29, 2021 •

edited

Loading

jun0tpyrc commented Jul 29, 2021

jun0tpyrc commented Jul 29, 2021 •

edited

Loading

dracmic commented Jul 29, 2021 •

edited

Loading

kgcdream2019 commented Jul 30, 2021

kgcdream2019 commented Jul 30, 2021

seonggwonyoon commented Jul 30, 2021 •

edited

Loading

jun0tpyrc commented Jul 30, 2021 •

edited

Loading

jayboy-mabushi commented Jul 30, 2021

j75689 commented Jul 30, 2021

Node synchronization slow #325

Node synchronization slow #325

Comments

yqZhang711 commented Jul 26, 2021

yqZhang711 commented Jul 26, 2021

wenchaopeng commented Jul 26, 2021

vae520283995 commented Jul 26, 2021

vietthang207 commented Jul 26, 2021

yqZhang711 commented Jul 26, 2021

yqZhang711 commented Jul 26, 2021

vae520283995 commented Jul 26, 2021

luoxiandong commented Jul 27, 2021

wo4zhuzi commented Jul 27, 2021

wenchaopeng commented Jul 27, 2021

jun0tpyrc commented Jul 27, 2021

ghost commented Jul 27, 2021

vshybalov commented Jul 27, 2021 • edited Loading

dracmic commented Jul 28, 2021

vae520283995 commented Jul 28, 2021

zy2d commented Jul 28, 2021

j75689 commented Jul 28, 2021

dracmic commented Jul 28, 2021

dwjpeng commented Jul 28, 2021

seonggwonyoon commented Jul 29, 2021 • edited Loading

jun0tpyrc commented Jul 29, 2021

jun0tpyrc commented Jul 29, 2021 • edited Loading

dracmic commented Jul 29, 2021 • edited Loading

kgcdream2019 commented Jul 30, 2021

kgcdream2019 commented Jul 30, 2021

seonggwonyoon commented Jul 30, 2021 • edited Loading

Dedicated Server

AWS EC2

Dedicated Server (geth version: 1.1.0 beta)

AWS EC2 (geth version: 1.1.0)

jun0tpyrc commented Jul 30, 2021 • edited Loading

jayboy-mabushi commented Jul 30, 2021

j75689 commented Jul 30, 2021

vshybalov commented Jul 27, 2021 •

edited

Loading

seonggwonyoon commented Jul 29, 2021 •

edited

Loading

jun0tpyrc commented Jul 29, 2021 •

edited

Loading

dracmic commented Jul 29, 2021 •

edited

Loading

seonggwonyoon commented Jul 30, 2021 •

edited

Loading

jun0tpyrc commented Jul 30, 2021 •

edited

Loading