node crash and when start again, it always panic.

[yaanhyy]

It look like that the data which stored in db is not identical with the data stored in remote node db?
here is the log:
2019-09-10 10:05:26 Syncing 0.0 bps, target=#174602 (4 peers), best: #170051 (0xb14e…0cf7), finalized #158442 (0x25eb…9840), ⬇ 31.8kiB/s ⬆ 22.1kiB/s
Hash: given=edac50c8e697c899ce2bf388686bdd33b9db3983d51a98190ab903a4bec8ce7c, expected=67e81b16f1d35c35c64f08f35a4e9939bd8820d20a57c04f3f102a4a56dfd7e1

====================

stack backtrace:
0: substrate_panic_handler::set::{{closure}}::h6c64ce3a7889097c (0x55f2b5b2ed9e)
1: std::panicking::rust_panic_with_hook::h2e01ca08bfa50f6c (0x55f2b5cb863c)
at src/libstd/panicking.rs:481
2: std::panicking::begin_panic::h7a7473a1c7cf749c (0x55f2b514b8d5)
3: <node_runtime::Runtime as substrate_client::runtime_api::runtime_decl_for_Core::Core<sr_primitives::generic::block::Block<sr_primitives::generic::header::Header<u32,sr_primitives::traits::BlakeTwo256>,sr_primitives::generic::unchecked_extrinsic::UncheckedExtrinsic<<srml_indices::Module<node_runtime::Runtime> as sr_primitives::traits::StaticLookup>::Source,node_runtime::Call,sr_primitives::AnySignature,(srml_system::CheckVersion<node_runtime::Runtime>, srml_system::CheckGenesis<node_runtime::Runtime>, srml_system::CheckEra<node_runtime::Runtime>, srml_system::CheckNonce<node_runtime::Runtime>, srml_system::CheckWeight<node_runtime::Runtime>, srml_balances::TakeFees<node_runtime::Runtime>)>>>>::execute_block::h6bb511705724960c (0x55f2b5158733)
4: node_runtime::api::dispatch::hc8f40040c6da3def (0x55f2b5142f2b)
5: std::panicking::try::do_call::h61ac906458cbbf61 (0x55f2b511efa2)
6: __rust_maybe_catch_panic (0x55f2b5cbd4da)
at src/libpanic_unwind/lib.rs:80
7: substrate_executor::native_executor::safe_call::h7180b5bf22fac95f (0x55f2b511ee48)
8: std::thread::local::LocalKey::with::h8b8a8233423e320f (0x55f2b511e9a1)
9: <node_executor::Executor as substrate_executor::native_executor::NativeExecutionDispatch>::dispatch::h1082ec9f7949505d (0x55f2b511e5ad)
10: <substrate_executor::native_executor::NativeExecutor as substrate_state_machine::CodeExecutor<substrate_primitives::hasher::blake2::Blake2Hasher>>::call::h0c922e3dd8b142f9 (0x55f2b5048d09)
11: substrate_state_machine::StateMachine<H,N,B,T,O,Exec>::execute_aux::h11eb3e74471b1e34 (0x55f2b49de0ed)
12: substrate_state_machine::StateMachine<H,N,B,T,O,Exec>::execute_call_with_native_else_wasm_strategy::h89153c3d1cd03cf3 (0x55f2b4a368d2)
13: substrate_state_machine::StateMachine<H,N,B,T,O,Exec>::execute_using_consensus_failure_handler::h69c8f0915363d871 (0x55f2b4a0cf8c)
14: <substrate_client::call_executor::LocalCallExecutor<B,E> as substrate_client::call_executor::CallExecutor<Block,substrate_primitives::hasher::blake2::Blake2Hasher>>::call_at_state::h4a2d77a60fa515d7 (0x55f2b508ca30)
15: <&substrate_client::client::Client<B,E,Block,RA> as substrate_consensus_common::block_import::BlockImport>::import_block::hca6e9f968d1f90cf (0x55f2b4c7fce4)
16: <substrate_finality_grandpa::import::GrandpaBlockImport<B,E,Block,RA,PRA,SC> as substrate_consensus_common::block_import::BlockImport>::import_block::h9504021bf191d156 (0x55f2b4b10c15)
17: <substrate_consensus_babe::BabeBlockImport<B,E,Block,I,RA,PRA> as substrate_consensus_common::block_import::BlockImport>::import_block::ha0258999720fd213 (0x55f2b5041a1b)
18: substrate_consensus_common::import_queue::import_single_block::h3672434bc23e31ea (0x55f2b4cc77ff)
19: <futures_util::future::poll_fn::PollFn as core::future::future::Future>::poll::h960f28dc0322aa67 (0x55f2b4a5765b)
20: <futures_util::future::then::Then<Fut1,Fut2,F> as core::future::future::Future>::poll::h2a8efd5371c09d24 (0x55f2b4dfbc1b)
21: <futures_util::future::poll_fn::PollFn as core::future::future::Future>::poll::h3545e72c8b248bc5 (0x55f2b4a56380)
22: futures_executor::thread_pool::PoolState::work::hcd1ab1cee5baf1b2 (0x55f2b5af7e26)
23: std::sys_common::backtrace::__rust_begin_short_backtrace::h1c1d75c7aabaf887 (0x55f2b5af6ee4)
24: std::panicking::try::do_call::hcec89d19f193cf42 (0x55f2b5af57b5)
25: __rust_maybe_catch_panic (0x55f2b5cbd4da)
at src/libpanic_unwind/lib.rs:80
26: core::ops::function::FnOnce::call_once{{vtable.shim}}::he565f49584b2e31c (0x55f2b5af584a)
27: <alloc::boxed::Box as core::ops::function::FnOnce>::call_once::hf92c7ee34ebffad1 (0x55f2b5ca6e8f)
at /rustc/521d78407471cb78e9bbf47160f6aa23047ac499/src/liballoc/boxed.rs:922
28: <alloc::boxed::Box as core::ops::function::FnOnce>::call_once::ha94d3a20f0e8dd18 (0x55f2b5cbc820)
at /rustc/521d78407471cb78e9bbf47160f6aa23047ac499/src/liballoc/boxed.rs:922
std::sys_common::thread::start_thread::h4ab30187e03381f3
at src/libstd/sys_common/thread.rs:13
std::sys::unix::thread::Thread::new::thread_start::hb63d4d1c8078a565
at src/libstd/sys/unix/thread.rs:79
29: start_thread (0x7fe8b4f876ba)
30: clone (0x7fe8b4aa741d)
31: (0x0)

Thread 'import-queue-worker-0' panicked at 'Storage root must match that calculated.', /mnt/.cargo/git/checkouts/substrate-7e08433d4c370a21/d2e7d66/srml/executive/src/lib.rs:319

This is a bug. Please report it at:

https://github.com/paritytech/substrate/issues/new

Hash not equal
edac50c8e697c899ce2bf388686bdd33b9db3983d51a98190ab903a4bec8ce7c
67e81b16f1d35c35c64f08f35a4e9939bd8820d20a57c04f3f102a4a56dfd7e1
panicked at 'Storage root must match that calculated.', /mnt/.cargo/git/checkouts/substrate-7e08433d4c370a21/d2e7d66/srml/executive/src/lib.rs:319:3
2019-09-10 10:05:26 Block import error:
Execution(Wasmi(Trap(Trap { kind: Unreachable })))
2019-09-10 10:05:26 Invalid justification provided by QmcQG5Jqi3Q6ymdwK28fzc9rCvEaJiuM4VMmGMgKg51AZc for #0x82de…7555

[carumusan]

+1. Saw this when running on the edgeware testnet.

[kianenigma]

I am not sure but I think this error has recently been identified quite a few times. But due to the nature of it, it is rather hard to pin it down to what exactly caused the corrupt state root.

[yjhmelody]

@carumusan I think if you alter your code and complie it to native code, you need to purge-chain.

[marcio-diaz]

Closed by #3989