BSC node crashes and can't recover (Looking for peers)

[APshenkin]

Hi there,

Please note that this is an issue tracker reserved for bug reports and feature requests.

For general questions please use discord or the Ethereum stack exchange at https://ethereum.stackexchange.com.

System information

Geth version: geth version 1.0.7
OS & Version: linux

Actual behaviour

Nodes crashes with BAD BLOCK error. After restarting in could not find peers

INFO [04-21|12:24:20.121] Looking for peers                        peercount=1 tried=113 static=23
INFO [04-21|12:24:32.053] Looking for peers                        peercount=0 tried=99  static=23
INFO [04-21|12:24:42.337] Looking for peers                        peercount=0 tried=74  static=23

Backtrace

ERROR[04-21|11:56:46.796]
Chain config: {ChainID: 56 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Ramanujan: 0, Niels: 0, MirrorSync: 5184000, Engine: parlia}

########## BAD BLOCK #########
Number: 6758576
Hash: 0x9e256fd0f982a4d65d4d06fadfdaed99fdd0bb479c05981a3dd44ed65b00b819


Error: nonce too high

WARN [04-21|11:56:46.796] Synchronisation failed, dropping peer    peer=acfcfb046811caac err="retrieved hash chain is invalid"
##############################

panic: expected tx hash 0x03ce627042c03c8a1e16c9b44c7335eb04a1845e7bc0294cf3fa1d3595c0da09, get 0x56f9f75f8ce5da3f096a2b136afcbef60aedc4e76ddbeda1c8133e816e3b3256

github.com/ethereum/go-ethereum/consensus/parlia.(*Parlia).Finalize(0xc000115c00, 0x19a4100, 0xc00d18a000, 0xc1819ef680, 0xc0f0f238c0, 0xc329271f60, 0x2411cd0, 0x0, 0x0, 0xc329271ec0, ...)
	/go-ethereum/consensus/parlia/parlia.go:707 +0x1139
goroutine 774312364 [running]:
github.com/ethereum/go-ethereum/core.(*StateProcessor).Process(0xc000450620, 0xc15de4bc20, 0xc0f0f238c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go-ethereum/core/blockchain.go:1726 +0xccc
	/go-ethereum/core/blockchain.go:1543 +0xcab
github.com/ethereum/go-ethereum/eth/fetcher.(*BlockFetcher).insert.func1(0xc00015b6c0, 0xce8607e9411b8962, 0xc7e37974c4389f02, 0x45e2c5df946fa84b, 0xcbee712eea2f00a6, 0xc15de4bc20, 0xc296cd1e80, 0x10)
created by github.com/ethereum/go-ethereum/eth/fetcher.(*BlockFetcher).insert
	/go-ethereum/core/state_processor.go:94 +0xa02
github.com/ethereum/go-ethereum/core.(*BlockChain).insertChain(0xc00d18a000, 0xc2e45e2c50, 0x1, 0x1, 0xc11def0c01, 0x0, 0x0, 0x0)
github.com/ethereum/go-ethereum/core.(*BlockChain).InsertChain(0xc00d18a000, 0xc2e45e2c50, 0x1, 0x1, 0x0, 0x0, 0x0)
	/go-ethereum/eth/fetcher/block_fetcher.go:667 +0x2be
github.com/ethereum/go-ethereum/eth.NewProtocolManager.func3(0xc2e45e2c50, 0x1, 0x1, 0x23c6e01, 0xc09eebc900, 0x1304c20)
	/go-ethereum/eth/handler.go:185 +0x745
	/go-ethereum/eth/fetcher/block_fetcher.go:694 +0x82f

[APshenkin]

Seems this crash corrupts chain state. Synchronisation started but from genesis block.

INFO [04-21|12:01:17.249] Loaded most recent local fast block      number=0 hash=d4e567…cb8fa3 td=17179869184 age=52y2w4d
INFO [04-21|12:01:17.249] Loaded most recent local full block      number=0 hash=d4e567…cb8fa3 td=17179869184 age=52y2w4d
INFO [04-21|12:01:17.249] Loaded most recent local header          number=0 hash=d4e567…cb8fa3 td=17179869184 age=52y2w4d

So the best option is to recover from snapshot?

[unclezoro]

The data is corrupt, please check ethereum/go-ethereum#23613
You can use geth snapshot verify-state to double check.

[unclezoro]

Will kindly close this issue.