Synchronisation failed, retrying & Rolled back chain segment .. reason="syncing canceled (requested)"

[ocracy]

System information

Geth version: 1.1.7-74f6b613
OS & Version: Ubuntu 20.04

Expected behaviour

Finish eth.syncing

Actual behaviour

Still syncing for 5 days
{
currentBlock: 13333752,
highestBlock: 13335820,
knownStates: 776715822,
pulledStates: 776715821,
startingBlock: 13292646
}

Steps to reproduce the behaviour

Start command
./geth_linux --config ./config.toml --datadir ./node --cache 20000 --rpc.allow-unprotected-txs --txlookuplimit 0 --ws --ws.origins '*' --ws.port 8546

Backtrace

t=2021-12-09T21:51:05+0000 lvl=info msg="Imported new block headers"            count=129  elapsed=13.268ms    number=13,333,805 hash=0x40f513b37e69da4f8652d63da32d4de9704fb832879a37412f97de333f8bf6f4 age=1h38m12s  ignored=63
t=2021-12-09T21:51:05+0000 lvl=info msg="Imported new block headers"            count=576  elapsed=38.680ms    number=13,334,381 hash=0x3c213c7203d49b6dfce65e572817563db95f14ca1d122f0d69bc53160ae4d2f7 age=1h9m23s
t=2021-12-09T21:51:05+0000 lvl=info msg="Imported new block headers"            count=960  elapsed=67.171ms    number=13,335,341 hash=0x5a82b7bd4d5c3f29c4e4aa8a0970a4ca915df50afab2d20c0096eb50a5b9e4d0 age=21m23s
t=2021-12-09T21:51:05+0000 lvl=info msg="Imported new block headers"            count=192  elapsed=11.751ms    number=13,335,533 hash=0x0e4b0e2ac32b000d9f3a522060778af02e694d8824a21d390eeb86b13bd4b419 age=11m47s
t=2021-12-09T21:51:05+0000 lvl=info msg="Imported new block headers"            count=192  elapsed=11.636ms    number=13,335,725 hash=0xc0199934416131da4727426f0cb5df838ab4573aa3e264d0c2f50ba43cdd5626 age=2m11s
t=2021-12-09T21:51:06+0000 lvl=info msg="Imported new block receipts"           count=6    elapsed=16.939ms    number=13,333,619 hash=0xb0bff9b22a371a7d5b838796c967083bf2f4343329304c64e68f79a2e422c013 age=1h47m31s  size="1.24 MiB"
t=2021-12-09T21:51:06+0000 lvl=info msg="Imported new block headers"            count=41   elapsed=6.821ms     number=13,335,766 hash=0xc370012989dfa91a67dd88717936a6c18eaf276ef75a96cfdca5902419b5d303
t=2021-12-09T21:51:06+0000 lvl=info msg="Imported new block receipts"           count=58   elapsed=159.629ms   number=13,333,677 hash=0x4958670517d6db6efa983732968f27f5ad38c34c9a6abec8f7e326b377020744 age=1h44m37s  size="12.07 MiB"
t=2021-12-09T21:51:06+0000 lvl=info msg="Imported new block receipts"           count=3    elapsed=7.514ms     number=13,333,680 hash=0xfd0b1bbd15450d3405c1e9a9af447c534fb99e0fb1928d115a2507b642d8eae6 age=1h44m28s  size="555.41 KiB"
t=2021-12-09T21:51:06+0000 lvl=warn msg="Node data write error"                 err="trie node 507547..81a1a2 failed with all peers (11 tries, 11 peers)"
t=2021-12-09T21:51:06+0000 lvl=info msg="Imported new block receipts"           count=74   elapsed=196.124ms   number=13,333,754 hash=0x9442ad563bf22e3daec37972efc50c4cf09d2522d68f4fe4d9261886e6c52933 age=1h40m46s  size="14.78 MiB"
t=2021-12-09T21:51:06+0000 lvl=warn msg="Rewinding blockchain"                  target=13,333,717
t=2021-12-09T21:51:07+0000 lvl=info msg="Loaded most recent local header"       number=13,333,717 hash=0x73533a39f5f4cdea35fb1db91bad740671bc563927d46e76bd3b87e601cd4690 td=26,482,505 age=1h42m38s
t=2021-12-09T21:51:07+0000 lvl=info msg="Loaded most recent local full block"   number=0          hash=0x0d21840abff46b96c84b2ac9e10e4f5cdaeb5693cb665db62a2f3b02d2d57b5b td=1          age=1y7mo4w
t=2021-12-09T21:51:07+0000 lvl=info msg="Loaded most recent local fast block"   number=13,333,717 hash=0x73533a39f5f4cdea35fb1db91bad740671bc563927d46e76bd3b87e601cd4690 td=26,482,505 age=1h42m38s
t=2021-12-09T21:51:07+0000 lvl=info msg="Loaded last fast-sync pivot marker"    number=13,335,697
t=2021-12-09T21:51:07+0000 lvl=warn msg="Rolled back chain segment"             header=13335766->13333717 fast=13333754->13333717 block=0->0 reason="syncing canceled (requested)"
t=2021-12-09T21:51:07+0000 lvl=warn msg="Synchronisation failed, retrying"      err="trie node 507547..81a1a2 failed with all peers (11 tries, 11 peers)"
t=2021-12-09T21:51:09+0000 lvl=info msg="Disabling direct-ancient mode"         origin=13,333,717 ancient=13,178,881
t=2021-12-09T21:51:09+0000 lvl=info msg="Imported new block headers"            count=576  elapsed=38.127ms    number=13,334,293 hash=0x500747475741f0daac668a7d0413593d557d6e63cac9c518ba598b7c811de99f age=1h13m51s
t=2021-12-09T21:51:09+0000 lvl=info msg="Imported new block headers"            count=1344 elapsed=80.773ms    number=13,335,637 hash=0x3e766919b5cf2d9204c72c30a44743c2160c4e9e152e2e70b507e422903282bb age=6m39s
t=2021-12-09T21:51:10+0000 lvl=info msg="Imported new block receipts"           count=14   elapsed=39.107ms    number=13,333,731 hash=0x1213ffee210125362841d4f04aa5ccdc0e50aeae7de669f53b0db78910b827d6 age=1h41m59s  size="2.75 MiB"
t=2021-12-09T21:51:10+0000 lvl=info msg="Imported new block receipts"           count=2    elapsed=5.702ms     number=13,333,733 hash=0x802450ddc999e55694eb20704831fdf41e2f70b6f2607d3722fe8f5a64201e67 age=1h41m53s  size="429.23 KiB"
t=2021-12-09T21:51:10+0000 lvl=warn msg="Node data write error"                 err="trie node 507547..81a1a2 failed with all peers (11 tries, 11 peers)"
t=2021-12-09T21:51:10+0000 lvl=warn msg="Rewinding blockchain"                  target=13,333,588
t=2021-12-09T21:51:10+0000 lvl=info msg="Loaded most recent local header"       number=13,333,588 hash=0xe17978e2c08464dafb08c30227cf300ebcce09a047fecbf6099e956d330dbdb7 td=26,482,247 age=1h49m8s
t=2021-12-09T21:51:10+0000 lvl=info msg="Loaded most recent local full block"   number=0          hash=0x0d21840abff46b96c84b2ac9e10e4f5cdaeb5693cb665db62a2f3b02d2d57b5b td=1          age=1y7mo4w
t=2021-12-09T21:51:10+0000 lvl=info msg="Loaded most recent local fast block"   number=13,333,588 hash=0xe17978e2c08464dafb08c30227cf300ebcce09a047fecbf6099e956d330dbdb7 td=26,482,247 age=1h49m8s
t=2021-12-09T21:51:10+0000 lvl=info msg="Loaded last fast-sync pivot marker"    number=13,335,697
t=2021-12-09T21:51:10+0000 lvl=warn msg="Rolled back chain segment"             header=13335637->13333588 fast=13333733->13333588 block=0->0 reason="syncing canceled (requested)"
t=2021-12-09T21:51:10+0000 lvl=warn msg="Synchronisation failed, retrying"      err="trie node 507547..81a1a2 failed with all peers (11 tries, 11 peers)"
t=2021-12-09T21:51:12+0000 lvl=info msg="Disabling direct-ancient mode"         origin=13,333,588 ancient=13,178,881
t=2021-12-09T21:51:13+0000 lvl=info msg="Imported new block headers"            count=192  elapsed=13.412ms    number=13,333,780 hash=0x50ea5de4431c90e6e7607062e49ebd90fde70fb7212464319643b1ac28149ceb age=1h39m35s
t=2021-12-09T21:51:13+0000 lvl=info msg="Imported new block headers"            count=384  elapsed=23.767ms    number=13,334,164 hash=0x707daa7ed11cbab7b2f29957daf0806a83a0a8dc92421cf10a4d5ec361551a58 age=1h20m22s
t=2021-12-09T21:51:13+0000 lvl=info msg="Imported new block headers"            count=384  elapsed=23.937ms    number=13,334,548 hash=0xc4d0ec3ff4db420a692ee45f20e5fd049dec0bd772845f434935ccd627ca4f51 age=1h1m10s
t=2021-12-09T21:51:13+0000 lvl=info msg="Imported new block receipts"           count=2    elapsed=5.447ms     number=13,333,590 hash=0x781584b2d3d59297162a0ecafb6a673f9544f7d71e05b02bb388c23b5041f22a age=1h49m5s   size="326.27 KiB"
t=2021-12-09T21:51:13+0000 lvl=info msg="Imported new block headers"            count=1152 elapsed=72.198ms    number=13,335,700 hash=0x92910c465b7362ba589558b34086b7584be359c0e00031ba322bb3372dde7d9b age=3m34s
t=2021-12-09T21:51:13+0000 lvl=info msg="Imported new block receipts"           count=20   elapsed=56.160ms    number=13,333,610 hash=0xb1418960b5c6fbe0ee738c6fcc4c7c0d087d5ff2482f5dc3a105349e19cbca3f age=1h48m5s   size="3.97 MiB"
t=2021-12-09T21:51:13+0000 lvl=warn msg="Node data write error"                 err="trie node 507547..81a1a2 failed with all peers (11 tries, 11 peers)"
t=2021-12-09T21:51:13+0000 lvl=warn msg="Rewinding blockchain"                  target=13,333,651
t=2021-12-09T21:51:14+0000 lvl=info msg="Loaded most recent local header"       number=13,333,651 hash=0xc99cac47042b8c731f3ec56e9ab64c03209dfb47ed2fa627f8f1a4b6870b2fff td=26,482,373 age=1h46m3s
t=2021-12-09T21:51:14+0000 lvl=info msg="Loaded most recent local full block"   number=0          hash=0x0d21840abff46b96c84b2ac9e10e4f5cdaeb5693cb665db62a2f3b02d2d57b5b td=1          age=1y7mo4w
t=2021-12-09T21:51:14+0000 lvl=info msg="Loaded most recent local fast block"   number=13,333,610 hash=0xb1418960b5c6fbe0ee738c6fcc4c7c0d087d5ff2482f5dc3a105349e19cbca3f td=26,482,291 age=1h48m6s
t=2021-12-09T21:51:14+0000 lvl=info msg="Loaded last fast-sync pivot marker"    number=13,335,697
t=2021-12-09T21:51:14+0000 lvl=warn msg="Rolled back chain segment"             header=13335700->13333651 fast=13333610->13333610 block=0->0 reason="syncing canceled (requested)"
t=2021-12-09T21:51:14+0000 lvl=warn msg="Synchronisation failed, retrying"      err="trie node 507547..81a1a2 failed with all peers (11 tries, 11 peers)"

Ubuntu running on 16core CPU and 32GB RAM

Is it normal that getting Node data write error Synchronisation failed, retrying

[RumeelHussainbnb]

#338

Please proceed to join our Discord channel for more discussion at https://discord.com/invite/binancesmartchain

[gaurav-s-cisin]

Hi @ocracy
Did you find some solution to this? I am also stuck on the same. Tried restarting the node multiple times, even removed the data and restarted, but always facing these errors.