Cannot restart syncing archive node

[AltOrder]

Starting and stopping a syncing archive node fails for Kusama and Polkadot. This happens both in Full and Fast mode.

• Docker image

soramitsu/kagome                       latest    f896e533d8c6   29 hours ago   464MB

• Reproduce with Full mode on Kusama:

docker run -d -v /root:/kagome --name kagome soramitsu/kagome:latest kagome --chain kusama --base-path kusama-node-1 --sync Full --state-pruning archive
docker restart kagome

This hangs kagome and needs to be killed manually.

docker logs kagome -f
...
...
23.07.14 19:30:28.992889  kagome           Info      BlockExecutor  Imported block #484 (0x8e38…b0c1) within 28 ms. (lag 1324 days)
23.07.14 19:30:29.022523  kagome           Info      BlockStorage  Added block #485 (0x92f7…62ac) as child of #484 (0x8e38…b0c1)
23.07.14 19:30:29.022667  kagome           Info      BlockExecutor  Imported block #485 (0x92f7…62ac) within 28 ms. (lag 1324 days)
Fatal: callImport: unknown import: env.��]^AV
23.07.14 19:30:47.467629  kagome           Info      AppConfiguration  Soramitsu Kagome started. Version: 0.9.0-70
23.07.14 19:30:47.468121  kagome           Info      AppConfiguration  Automatically added IPv6 listen address /ip6/::/tcp/30363
23.07.14 19:30:47.468154  kagome           Info      AppConfiguration  Automatically added IPv4 listen address /ip4/0.0.0.0/tcp/30363
23.07.14 19:30:47.468341  kagome           Info      AppConfiguration  Listen addresses are set. The p2p port value would be ignored then.
23.07.14 19:30:47.526633  kagome           Warning   chain_spec  Field 'chainType' was not specified in the chain spec. 'Live' by default.
23.07.14 19:30:47.526691  kagome           Warning   chain_spec  A non-empty set of 'badBlocks' encountered! They might not be taken into account!
23.07.14 19:30:47.836981  kagome           Warning   Injector  Can not obtain a libp2p keypair from crypto storage. A unique one will be generated
23.07.14 19:30:47.847528  kagome           Info      BlockTree  Highest block: #485 (0x92f7…62ac), Last finalized: #0 (0xb0a8…dafe)
23.07.14 19:30:47.856044  kagome           Warning   BlockTree  Can't add existing non-finalized block #0 (0xb0a8…dafe) to block tree: block, which should have been inserted, already exists in the tree
23.07.14 19:30:47.952965  kagome           Warning   PrunerStateRecovery  Running pruner on a non-empty non-pruned storage may lead to skipping some stored states.

• For Fast mode, kagome segfaults on restart (with clean database):

...
...
23.07.14 19:34:12.666719  kagome           Info      BlockTree  Finalized block #1024 (0x3f17…fd7f)
23.07.14 19:34:13.680902  kagome           Info      BlockTree  Finalized block #1536 (0x55d1…fb04)
23.07.14 19:34:14.556288  kagome           Info      AppConfiguration  Soramitsu Kagome started. Version: 0.9.0-70
23.07.14 19:34:14.556729  kagome           Info      AppConfiguration  Automatically added IPv6 listen address /ip6/::/tcp/30363
23.07.14 19:34:14.556764  kagome           Info      AppConfiguration  Automatically added IPv4 listen address /ip4/0.0.0.0/tcp/30363
23.07.14 19:34:14.556998  kagome           Info      AppConfiguration  Listen addresses are set. The p2p port value would be ignored then.
23.07.14 19:34:14.615219  kagome           Warning   chain_spec  Field 'chainType' was not specified in the chain spec. 'Live' by default.
23.07.14 19:34:14.615279  kagome           Warning   chain_spec  A non-empty set of 'badBlocks' encountered! They might not be taken into account!
23.07.14 19:34:14.885920  kagome           Warning   Injector  Can not obtain a libp2p keypair from crypto storage. A unique one will be generated
23.07.14 19:34:14.890650  kagome           Info      BlockTree  Highest block: #1665 (0xcddc…e084), Last finalized: #1536 (0x55d1…fb04)
23.07.14 19:34:14.893108  kagome           Warning   BlockTree  Can't add existing non-finalized block #1536 (0x55d1…fb04) to block tree: block, which should have been inserted, already exists in the tree
23.07.14 19:34:14.976922  kagome           Warning   PrunerStateRecovery  Running pruner on a non-empty non-pruned storage may lead to skipping some stored states.
23.07.14 19:34:14.978201  kagome           Info      Application  Start as node version '0.9.0-70' named as 'ebaefdc1-eccb-40d0-910d-4e3bb0b9e3f7' with PID 1
23.07.14 19:34:14.978240  kagome           Info      Application  Chain path is "kusama-node-1/chains/ksmcc3", storage backend is RocksDB
terminate called after throwing an instance of 'boost::wrapexcept<std::system_error>'
  what():  the block must contain at least BABE header and seal digests
Stack trace (most recent call last):
#19   Object "kagome", at 0x555e349c32c0, in
#18   Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f57d5008784, in __libc_start_main
#17   Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f57d50086c9, in
#16   Object "kagome", at 0x555e348c79af, in
#15   Object "kagome", at 0x555e349cce50, in
#14   Object "kagome", at 0x555e4072faa8, in
#13   Object "kagome", at 0x555e4072fe8a, in
#12   Object "kagome", at 0x555e3d87bcf0, in
#11   Object "kagome", at 0x555e347cf2ea, in
#10   Object "kagome", at 0x555e349c68de, in
#9    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f57d53522b7, in __cxa_throw
#8    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f57d5352064, in std::terminate()
#7    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f57d5351ff9, in
#6    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f57d53429da, in
#5    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f57d50074b1, in abort
#4    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f57d501d471, in gsignal
#3    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f57d506b0fb, in
#2    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f57d501d50f, in
#1    Object "kagome", at 0x555e349c9fee, in
#0    Object "kagome", at 0x555e349c9046, in
Aborted (Signal sent by tkill() 1 0)
watf? exit

[Harrm]

This happens due to some problems with the new trie pruner, when #1702 is merged you will be able to run the node without it, while it's still unstable.