Please generate a late-April snapshot

[DefiDebauchery]

Hello,

With so many reports of sync issues, I'd personally be interested in having a known-good snapshot taken since the recent transaction explosion. I'm finding it a little difficult to believe that this is simply hardware, as even those with NVMe drives appear to be having trouble keeping synced.

I'm currently at 173M active states, and strangely slipping from a constant ~3m lag (over the last 24 hours of trying to sync this) to now 6 minutes behind and struggling. An older node I had maintained without issue until the surge only has 25M states and is only trying to catch up on blocks (though I guess that could potentially change if/when it does).

I don't want to spread false conjecture, but I do feel that there's something technical causing chain issues for new nodes. Either way, if Binance has a known-good full-sync snapshot it can provide, I'd really appreciate being able to try it out.

[chainwhisper]

Hello, @DefiDebauchery

Thanks for your report!

You should be able to sync with the updated list of bootstrap nodes:

BootstrapNodes = ["enode://1cc4534b14cfe351ab740a1418ab944a234ca2f702915eadb7e558a02010cb7c5a8c295a3b56bcefa7701c07752acd5539cb13df2aab8ae2d98934d712611443@52.71.43.172:30311","enode://28b1d16562dac280dacaaf45d54516b85bc6c994252a9825c5cc4e080d3e53446d05f63ba495ea7d44d6c316b54cd92b245c5c328c37da24605c4a93a0d099c4@34.246.65.14:30311","enode://5a7b996048d1b0a07683a949662c87c09b55247ce774aeee10bb886892e586e3c604564393292e38ef43c023ee9981e1f8b335766ec4f0f256e57f8640b079d5@35.73.137.11:30311"]

Chain data snapshot for april: https://docs.binance.org/smart-chain/developer/snapshot.html

[cbosss]

Did this work for anyone? I added this to my config.toml, but my node is still syncing very slow. I seem to always be ~4 days behind the latest block.

As far as I can tell, i'm not saturating network, disk, or CPU. It seems like the bsc code itself isn't written in a way to take full advantage of available hardware.

@huangsuyu - how on earth do we get these nodes sync'd?

[BennyTheDev]

We are having sync issues as soon as loq queries are used with websockets: #160
However, it is still bad using RPC instead but not that bad.
If your client(s) use(s) Websockets it might be worth looking at it.

[a04512]

yes once late , never catch up
now I sync 2~5 blocks per 10s, but bsc chain has 3 blocks /10s so always 4day behind the latest block
as I see geth didn't fully use all bandwidth, hard disk or other hardware resources, so i just don't know how to keep a full node as the public rpc recently has some issue

[chainwhisper]

@crosstec which sync mode did you use?

[1b1og-com]

my wechat is wu4yyy, Let's discuss it together
@crosstec @cbosss @a04512 @huangsuyu @DefiDebauchery

[DefiDebauchery]

Hello, @DefiDebauchery

Thanks for your report!

You should be able to sync with the updated list of bootstrap nodes:

BootstrapNodes = ["enode://1cc4534b14cfe351ab740a1418ab944a234ca2f702915eadb7e558a02010cb7c5a8c295a3b56bcefa7701c07752acd5539cb13df2aab8ae2d98934d712611443@52.71.43.172:30311","enode://28b1d16562dac280dacaaf45d54516b85bc6c994252a9825c5cc4e080d3e53446d05f63ba495ea7d44d6c316b54cd92b245c5c328c37da24605c4a93a0d099c4@34.246.65.14:30311","enode://5a7b996048d1b0a07683a949662c87c09b55247ce774aeee10bb886892e586e3c604564393292e38ef43c023ee9981e1f8b335766ec4f0f256e57f8640b079d5@35.73.137.11:30311"]

Chain data snapshot for april: https://docs.binance.org/smart-chain/developer/snapshot.html

Thank you for your help. Unfortunately, this did not seem to correct the problem. I added Bootstrap nodes as well as replaced the StaticNodes from another listed Issue.

Specifically, I seem to get these two errors:
t=2021-04-26T19:49:00-0400 lvl=warn msg="Pivot became stale, moving" old=6914179 new=6914244
t=2021-04-26T19:49:12-0400 lvl=warn msg="Stalling state sync, dropping peer" peer=003c26e5b9625100

This is preventing me from collecting all of the states. I'm anywhere from 60 to 110 Blocks behind, and perpetually 1 state away from all 226.5 million knownStates.

Previously, I had copied over a chain database from another node that had been running smoothly until the spike in transactions. I was able to get to HEAD, but every time I tried to submit a txn, it took ages to get a response, and I would get a warning in the log about a chain reorg.

The more people I discuss about this (which I've been approached on Discord, Telegram, and Twitter about), the more I feel that there's some pervasive issue that happened at some point in the life of the chain. I must reiterate my request for a known-good database snapshot from a Validator or other trusted node.

[a04512]

I used to have a full synced node, but sometimes get chain reorg in log , and fail to catch up the latest block from then on

[chainwhisper]

@DefiDebauchery

which snapshot did you download? please use this one:
https://s3.ap-northeast-1.amazonaws.com/dex-bin.bnbstatic.com/s3-witness-data-download/chaindata-2021-04-15.zip?AWSAccessKeyId=AKIAYINE6SBQPUZDDRRO&Expires=1644459350&Signature=a4vfmga8%2BRwNZl3boisIMEdSWbA%3D

I have use this chaindata to sync a fullnode at my end successfully

[DefiDebauchery]

@huangsuyu - In this case, I didn't use any snapshot; I did a removedb and built from genesis. I'm asking for a late-April snapshot because I still very much believe the explosion that happened after this snapshot was taken is playing hell on people's nodes.

[kiiiiwiiii]

Any solution on this issue? Seems to me like a ton of people are having issues with no real solution. I've been trying to run my node a couple of weeks with no success following the official guide provided by binance. I've upgraded my machine a couple of times and even tried out in cloud using a $500 per month machine with NVME M.2 SSD etc. I don't think hardware is a problem here. What is going on with the chain??

[heliosVII-tech]

Set syncmode=“fast” in config.toml, and increase your number of peers. It’s worked for me many times without any issues. Syncs within a couple days tops and takes up 100-120GB. Then it switches to a full node.

[DefiDebauchery]

@heliosVII-tech - This is not a definitive answer. I always run with fast syncmode. Ever since the transaction explosion a week and some ago, I and several others have been unable to get fully synced.

I did compare with a friend who does have a properly-synced node. He only has about 227M states, whereas I have 271M and rising. We both started new copies from genesis.

I remain increasingly convinced that some sort of fork or other inconsistency has happened in the chain and is causing issues. Why it happens to some and not others is a mystery to be solved.

I continue to implore @huangsuyu to provide a late-April snapshot that is known to be properly operational.

[heliosVII-tech]

Are you running master? Or a tagged branch?

[cbosss]

@heliosVII-tech - which do you recommend?

[heliosVII-tech]

Try v1.0.7 if you haven’t? Or try master if you’ve been trying 1.0.7. Definitely Build from source.

[kiiiiwiiii]

I am trying to run it using v1.0.7 built from source and like DefiDebauchery I am way up from the 227M states that other fellow node runners have. And as a sidenote, seems like geth is not using the full potential of my computer. Only about 10% CPU usage, very low Disk I/O and bandwidth.

We are not the only ones having this problem and we would really appreciate an answer, what is going on?

[koen84]

Catching up to chainhead tends to be IOPS heavy (which isn't the same as diskspeed). I see a fairly clear difference between my archive node on NVMe versus SSD.

That said, any optimisaitons to keep lag and sync variance lower would be highly welcomed.

[j75689]

Hi all.
Thank you for your report.
We have received many report of a sync issue.
You can try the latest version. If you have any probleums, please feedback #338.
We will pay attention to the issue #338 for a long time, and if there are any updates, we will explain it on this issue.

Thanks.