testnet geth snapshot is 1 month old

[cshintov]

INFO [07-04|11:10:14.728] Loaded most recent local block           number=40,748,996 hash=67ccdc..6ebb98 root=fb6427..b00f11 td=81,272,498 age=1mo6d4h

Even though the file name is testnet-geth-pbss-20240607.tar.lz4 the data is of May 29th!

And when I start it with v1.4.11 it rewinds to genesis!

"
INFO [07-04|11:10:14.728] Loaded most recent local block           number=40,748,996 hash=67ccdc..6ebb98 root=fb6427..b00f11 td=81,272,498 age=1mo6d4h
INFO [07-04|11:10:14.731] Loaded most recent local finalized block number=40,748,994 hash=b47fa9..b43345 root=e8601a..615f64 td=81,272,494 age=1mo6d4h
WARN [07-04|11:10:14.898] Head state missing, repairing            number=40,748,996 hash=67ccdc..6ebb98 diskRoot=8d3dc4..02fca9
INFO [07-04|11:10:15.210] Snapshot loaded                          diskRoot=8d3dc4..02fca9 root=fb6427..b00f11
INFO [07-04|11:10:15.218] Rewound to block with state              number=40,748,664 hash=8b0425..d0ae92
WARN [07-04|11:10:23.978] Snapshot is not continuous with chain    snaproot=fb6427..b00f11 chainroot=8d3dc4..02fca9
INFO [07-04|11:10:23.978] Snapshot loaded                          diskRoot=8d3dc4..02fca9 root=8d3dc4..02fca9
INFO [07-04|11:10:23.978] Rewound to block with state              number=40,748,664 hash=8b0425..d0ae92
INFO [07-04|11:10:24.018] Loaded most recent local block           number=40,748,664 hash=8b0425..d0ae92 root=8d3dc4..02fca9 td=81,271,834 age=1mo6d4h
INFO [07-04|11:10:24.019] Loaded most recent local finalized block number=40,748,662 hash=a96e85..558649 root=3fb3d3..684a4c td=81,271,830 age=1mo6d4h
WARN [07-04|11:10:24.057] Enabling snapshot recovery               chainhead=40,748,664 diskbase=40,748,664
WARN [07-04|11:10:24.344] Snapshot is not continuous with chain    snaproot=fb6427..b00f11 chainroot=8d3dc4..02fca9
INFO [07-04|11:10:24.344] Snapshot loaded                          diskRoot=8d3dc4..02fca9 root=8d3dc4..02fca9
WARN [07-04|11:10:24.344] Rewinding chain to upgrade configuration err="mismatching Haber fork timestamp in database (have timestamp nil, want timestamp 1716962820, rewindto timestamp 1716962819)"
WARN [07-04|11:10:24.344] Rewinding blockchain to timestamp        target=1,716,962,819
WARN [07-04|11:10:24.646] Snapshot is not continuous with chain    snaproot=fb6427..b00f11 chainroot=a00fd9..0572cd
INFO [07-04|11:10:24.646] Snapshot loaded                          diskRoot=8d3dc4..02fca9 root=a00fd9..0572cd
INFO [07-04|11:10:26.804] Rewinding limit reached, resetting to genesis number=40,658,662 hash=909c9a..2f8ead limit=40,658,663
INFO [07-04|11:10:26.804] Chain is stateless, wait state sync      number=0          hash=6d3c66..0fbe34
WARN [07-04|11:10:26.804] Force rewinding till ancient limit       head=0

[STdevK]

The team is checking on the snapshot

[zzzckck]

INFO [07-04|11:10:14.728] Loaded most recent local block           number=40,748,996 hash=67ccdc..6ebb98 root=fb6427..b00f11 td=81,272,498 age=1mo6d4h

Even though the file name is testnet-geth-pbss-20240607.tar.lz4 the data is of May 29th!

And when I start it with v1.4.11 it rewinds to genesis!

"
INFO [07-04|11:10:14.728] Loaded most recent local block           number=40,748,996 hash=67ccdc..6ebb98 root=fb6427..b00f11 td=81,272,498 age=1mo6d4h
INFO [07-04|11:10:14.731] Loaded most recent local finalized block number=40,748,994 hash=b47fa9..b43345 root=e8601a..615f64 td=81,272,494 age=1mo6d4h
WARN [07-04|11:10:14.898] Head state missing, repairing            number=40,748,996 hash=67ccdc..6ebb98 diskRoot=8d3dc4..02fca9
INFO [07-04|11:10:15.210] Snapshot loaded                          diskRoot=8d3dc4..02fca9 root=fb6427..b00f11
INFO [07-04|11:10:15.218] Rewound to block with state              number=40,748,664 hash=8b0425..d0ae92
WARN [07-04|11:10:23.978] Snapshot is not continuous with chain    snaproot=fb6427..b00f11 chainroot=8d3dc4..02fca9
INFO [07-04|11:10:23.978] Snapshot loaded                          diskRoot=8d3dc4..02fca9 root=8d3dc4..02fca9
INFO [07-04|11:10:23.978] Rewound to block with state              number=40,748,664 hash=8b0425..d0ae92
INFO [07-04|11:10:24.018] Loaded most recent local block           number=40,748,664 hash=8b0425..d0ae92 root=8d3dc4..02fca9 td=81,271,834 age=1mo6d4h
INFO [07-04|11:10:24.019] Loaded most recent local finalized block number=40,748,662 hash=a96e85..558649 root=3fb3d3..684a4c td=81,271,830 age=1mo6d4h
WARN [07-04|11:10:24.057] Enabling snapshot recovery               chainhead=40,748,664 diskbase=40,748,664
WARN [07-04|11:10:24.344] Snapshot is not continuous with chain    snaproot=fb6427..b00f11 chainroot=8d3dc4..02fca9
INFO [07-04|11:10:24.344] Snapshot loaded                          diskRoot=8d3dc4..02fca9 root=8d3dc4..02fca9
WARN [07-04|11:10:24.344] Rewinding chain to upgrade configuration err="mismatching Haber fork timestamp in database (have timestamp nil, want timestamp 1716962820, rewindto timestamp 1716962819)"
WARN [07-04|11:10:24.344] Rewinding blockchain to timestamp        target=1,716,962,819
WARN [07-04|11:10:24.646] Snapshot is not continuous with chain    snaproot=fb6427..b00f11 chainroot=a00fd9..0572cd
INFO [07-04|11:10:24.646] Snapshot loaded                          diskRoot=8d3dc4..02fca9 root=a00fd9..0572cd
INFO [07-04|11:10:26.804] Rewinding limit reached, resetting to genesis number=40,658,662 hash=909c9a..2f8ead limit=40,658,663
INFO [07-04|11:10:26.804] Chain is stateless, wait state sync      number=0          hash=6d3c66..0fbe34
WARN [07-04|11:10:26.804] Force rewinding till ancient limit       head=0

1.the file name is the date that the .tar.gz was generated, not the block number's timestamp, we can improve it in the future.
2.how do you start the node, with flag "--pruneancient", sorry, there is a bug on the latest release for this flag, it has been fixed in bnb-chain/bsc#2562, will included in next release.

[jingjunLi]

Hi, @cshintov , There is an error message:
Rewinding chain to upgrade configuration err="mismatching Haber fork timestamp in database (have timestamp nil, want timestamp 1716962820, rewindto timestamp 1716962819)"
Could you please check if the HaberTime parameter you set during startup is consistent with the other nodes? An incorrect setting for this parameter is causing a rewind to genesis.

[cshintov]

@zzzckck

I run it as

        command = "/usr/local/bin/geth-bsc"
        args    = [
          "--config", "/root/local/config.toml",
          "--syncmode", "full",
          "--tries-verify-mode", "none",
          "--datadir", "/data",
          "--gcmode", "archive",
          "--cache", "8000",
          "--rpc.allow-unprotected-txs",
          "--history.transactions", "0",
          "--ws",
          "--ws.addr","0.0.0.0",
          "--ws.origins", "*",
          "--rpc.batch-response-max-size", "50000000",
        ]

@jingjunLi

I had old (March) zfs snapshots and since these snapshots were not working for me, I reverted to them and got it synced without mentioning any HaberTime parameter?

I didn't know the fork timestamp has to be mentioned to run the node. Where can I get more info?

[zzzckck]

"--tries-verify-mode", "none",

"--tries-verify-mode none", and "--gcmode archive", could be 2 conflicted options.
could you remove "--gcmode", "archive",?

[leoenes]

The same situation happened to me. In my case, I didn't put --gcmode archive

[zzzckck]

sorry, confirmed the issue, we will check

[zzzckck]

sorry, it really has the issue,the Haber hard fork was triggered right after the fork time, the snapshot was generated after Haber but it is not used.
we will update the testnet snapshot soon

[zzzckck]

@cshintov @leoenes testnet snapshot was just updated, pls retry testnet-geth-pbss-20240711.tar.lz4
thx for your feedback.