Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Panic on reorg #419

Closed
soc1c opened this issue Sep 6, 2019 · 6 comments
Closed

Panic on reorg #419

soc1c opened this issue Sep 6, 2019 · 6 comments

Comments

@soc1c
Copy link
Contributor

soc1c commented Sep 6, 2019 •
edited
Loading

Validator node crashed with:

2019-09-06 10:27:00 Imported #198718 (0x5062…ceed)                                                                     
2019-09-06 10:27:00 Error running offchain workers at Number(198718): UnknownBlock("Unknown state for block Number(198718)")
                                                                                                                       
====================                                                                                                   
                                                                                                                       
stack backtrace:                                                                                                       
   0: substrate_panic_handler::set::{{closure}}::h82cae3535dcc52cb (0x561d84a830a0)                                    
   1: std::panicking::rust_panic_with_hook::hffcefc09751839d1 (0x561d84bc06a9)                                                                                                                                                                
             at src/libstd/panicking.rs:481                                                                            
   2: std::panicking::continue_panic_fmt::hc0f142c930c846fc (0x561d84bc0142)                                           
             at src/libstd/panicking.rs:384                                                                            
   3: rust_begin_unwind (0x561d84bc0026)                                                                                                                                                                                                      
             at src/libstd/panicking.rs:311                                                                            
   4: core::panicking::panic_fmt::h2daf88b2616ca2b2 (0x561d84be1edd)                                                   
             at src/libcore/panicking.rs:85                                                                            
   5: core::result::unwrap_failed::h62dbd5a8e5560593 (0x561d83df1cee)                                                  
   6: <substrate_state_machine::ext::Ext<H,N,B,T,O> as substrate_state_machine::Externalities<H>>::storage::hb493a5df659ab9f3 (0x561d83abf960)
   7: <substrate_executor::wasm_executor::FunctionExecutor<E> as wasmi::host::Externals>::invoke_index::hb45ea024086571d1 (0x561d8399a25d)
   8: wasmi::func::FuncInstance::invoke::h5b09d54c708b1b31 (0x561d8385aa9a)                                            
   9: wasmi::runner::Interpreter::start_execution::h4858fcd74c407050 (0x561d83bbe408)
  10: wasmi::func::FuncInstance::invoke::h5b09d54c708b1b31 (0x561d8385ab2c)                                            
  11: substrate_executor::wasm_executor::WasmExecutor::call_in_wasm_module::h07b2a5c593c7addf (0x561d839c4757)
  12: <substrate_executor::native_executor::NativeExecutor<D> as substrate_state_machine::CodeExecutor<substrate_primitives::hasher::blake2::Blake2Hasher>>::call::heef7ff72239b90f2 (0x561d83c4d61b)
  13: substrate_state_machine::StateMachine<H,N,B,T,O,Exec>::execute_aux::h40e0fef5aad580f3 (0x561d83d75990)
  14: substrate_state_machine::StateMachine<H,N,B,T,O,Exec>::execute_using_consensus_failure_handler::h260d5fc78a88d573 (0x561d83d93cf0)
  15: <substrate_client::call_executor::LocalCallExecutor<B,E> as substrate_client::call_executor::CallExecutor<Block,substrate_primitives::hasher::blake2::Blake2Hasher>>::contextual_call::hc9d14c9a3c6e6a24 (0x561d83c6bc6b)
  16: <substrate_client::client::Client<B,E,Block,RA> as substrate_client::runtime_api::CallRuntimeAt<Block>>::call_api_at::hbf93113da3de111a (0x561d838cb86a)
  17: <polkadot_runtime::RuntimeApiImpl<RuntimeApiImplCall> as substrate_client::runtime_api::Core<<polkadot_runtime::Runtime as sr_primitives::traits::GetNodeBlockType>::NodeBlock>>::Core_initialize_block_runtime_api_impl::hb7bdde40663ba
0bc (0x561d83826239)
  18: substrate_client::runtime_api::Core::initialize_block::hb74ada721fa5e557 (0x561d837eb6d7)
  19: <substrate_client::call_executor::LocalCallExecutor<B,E> as substrate_client::call_executor::CallExecutor<Block,substrate_primitives::hasher::blake2::Blake2Hasher>>::contextual_call::h646c30bb0a86b123 (0x561d83c615c8)
  20: <substrate_client::client::Client<B,E,Block,RA> as substrate_client::runtime_api::CallRuntimeAt<Block>>::call_api_at::h6f50441f47597513 (0x561d838c842f)
  21: polkadot_primitives::parachain::runtime_decl_for_ParachainHost::validators_call_api_at::hee4b5cbc355a5bcd (0x561d83d641ce)
  22: polkadot_primitives::parachain::ParachainHost::validators::h5e101c52657c0562 (0x561d83806edb)
  23: polkadot_validation::ParachainValidation<C,N,P>::get_or_instantiate::he6ddef975679aaa6 (0x561d838f1044)
  24: <futures::stream::for_each::ForEach<S,F,U> as futures::future::Future>::poll::h989eff7555f1e838 (0x561d83eac5e8)
  25: <futures::future::select::Select<A,B> as futures::future::Future>::poll::h072fc464b56269e6 (0x561d83be6ff5)
  26: futures::future::chain::Chain<A,B,C>::poll::h31357d6fbbcb8b36 (0x561d83f3a679)
  27: futures::task_impl::std::set::h8ed5be9f01d42e6d (0x561d83fcb7b2)                                                 
  28: tokio_current_thread::CurrentRunner::set_spawn::h5abebce908ba8889 (0x561d83825a6b)
  29: tokio_current_thread::scheduler::Scheduler<U>::tick::h997a8e2f57321237 (0x561d83effdb9)
  30: tokio_current_thread::Entered<P>::block_on::hf670e5584a8dc800 (0x561d83825c66)
  31: std::thread::local::LocalKey<T>::with::hc4580ec2b7427ee3 (0x561d83cab224)                                                                                                                                                               
  32: std::thread::local::LocalKey<T>::with::hedef1bfa9db36c0b (0x561d83cae6fb)                                                                                                                                                               
  33: std::thread::local::LocalKey<T>::with::h5f5d9fce2829a137 (0x561d83ca090b)                                                                                                                                                               
  34: std::thread::local::LocalKey<T>::with::h80852b028e491352 (0x561d83ca4961)                                                                                                                                                               
  35: tokio::runtime::current_thread::runtime::Runtime::block_on::h26dab3a938557965 (0x561d83a3a815)
  36: std::sys_common::backtrace::__rust_begin_short_backtrace::hbddf5d9de90b8ce0 (0x561d83916c74)
  37: std::panicking::try::do_call::hf4bbd8c887fb36ec (0x561d83e755c9)                                                 
  38: __rust_maybe_catch_panic (0x561d84bc95ca)                                                                        
             at src/libpanic_unwind/lib.rs:82                                                                          
  39: core::ops::function::FnOnce::call_once{{vtable.shim}}::h68fd22c3217e11e2 (0x561d83a1dff4)
  40: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h42806b83647d4c79 (0x561d84bae44f)
             at /rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/liballoc/boxed.rs:746
  41: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h83c921c8e826dd1d (0x561d84bc8950)
             at /rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/liballoc/boxed.rs:746
      std::sys_common::thread::start_thread::h2613204ce513782e                                                         
             at src/libstd/sys_common/thread.rs:13                                                                     
      std::sys::unix::thread::Thread::new::thread_start::h4570080769500bcd                                             
             at src/libstd/sys/unix/thread.rs:79                                                                       
  42: start_thread (0x7fe2cef4e57f)                        
  43: __clone (0x7fe2cee620e3)                             
  44: <unknown> (0x0)                                      

Thread '<unnamed>' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Invalid state root: 0xc0d103959902b69de4784e4820776e06f50f1987730c493b8e8c6fc31a3d1eb3"', src/libcore/result.rs:999

This is a bug. Please report it at:                        

        https://github.com/paritytech/polkadot/issues/new

Running latest 0.5.1: 60155e8

[user@host ~]$ polkadot --version
polkadot 0.5.1-60155e89-x86_64-linux-gnu

[user@host ~]$ uname -a
Linux 5.2.11-arch1-1-ARCH #1 SMP PREEMPT Thu Aug 29 08:09:36 UTC 2019 x86_64 GNU/Linux
@bkchr
Copy link
Member

bkchr commented Sep 6, 2019

Was there a reorg before?

@soc1c
Copy link
Contributor Author

soc1c commented Sep 6, 2019

None I can see:

2019-09-06 10:26:03 Idle (26 peers), best: #198708 (0x50a8…969c), finalized #198707 (0x89b5…bd3d), ⬇ 32.8kiB/s ⬆ 43.0kiB/s                                                                                                                    
2019-09-06 10:26:06 Imported #198709 (0x1f7f…8852)                                                                                                                                                                                            
2019-09-06 10:26:06 Starting parachain attestation session on top of parent 0x1f7f0f970e829af4e5f363640e4d6de0f5f8425c0aeb0c8a9e180afe06bd8852. Local parachain duty is None                                                                  
2019-09-06 10:26:08 Idle (27 peers), best: #198709 (0x1f7f…8852), finalized #198708 (0x50a8…969c), ⬇ 37.8kiB/s ⬆ 53.9kiB/s                                                                                                                    
2019-09-06 10:26:13 Idle (23 peers), best: #198709 (0x1f7f…8852), finalized #198709 (0x1f7f…8852), ⬇ 35.8kiB/s ⬆ 49.3kiB/s                                                                                                                    
2019-09-06 10:26:13 Imported #198710 (0x50ef…f297)                                                                                                                                                                                            
2019-09-06 10:26:13 Starting parachain attestation session on top of parent 0x50eff1faf6b038561a2717a062e8bf1c45c59b3c9a6f503f4dd98d2ad7f8f297. Local parachain duty is None                                                                  
2019-09-06 10:26:18 Idle (26 peers), best: #198710 (0x50ef…f297), finalized #198710 (0x50ef…f297), ⬇ 37.9kiB/s ⬆ 50.8kiB/s                                                                                                                    
2019-09-06 10:26:18 Imported #198711 (0xe896…772f)                                                                                                                                                                                            
2019-09-06 10:26:18 Starting parachain attestation session on top of parent 0xe896d002005cb6fcc658fbb63f8454fd475a586b9234118dc63c639ec144772f. Local parachain duty is None                                                                  
2019-09-06 10:26:23 Idle (26 peers), best: #198711 (0xe896…772f), finalized #198711 (0xe896…772f), ⬇ 33.2kiB/s ⬆ 43.0kiB/s                                                                                                                    
2019-09-06 10:26:25 Imported #198712 (0x3c8a…4c3e)                                                                                                                                                                                            
2019-09-06 10:26:25 Starting parachain attestation session on top of parent 0x3c8a5038a1cd42b1f94628dd20621452820c33286e2493e568d4cc7953ed4c3e. Local parachain duty is None                                                                  
2019-09-06 10:26:28 Idle (23 peers), best: #198712 (0x3c8a…4c3e), finalized #198712 (0x3c8a…4c3e), ⬇ 33.5kiB/s ⬆ 45.4kiB/s                                                                                                                    
2019-09-06 10:26:30 Imported #198713 (0xbae3…9a95)                                                                                                                                                                                            
2019-09-06 10:26:30 Starting parachain attestation session on top of parent 0xbae308902d4955b4cddeaaa9fa0bf7603894427732d61e61928b6bfcd1f09a95. Local parachain duty is None                                                                  
2019-09-06 10:26:31 Imported #198713 (0x993c…8a05)                                                                                                                                                                                            
2019-09-06 10:26:33 Idle (26 peers), best: #198713 (0xbae3…9a95), finalized #198712 (0x3c8a…4c3e), ⬇ 29.0kiB/s ⬆ 36.2kiB/s                                                                                                                    
2019-09-06 10:26:38 Idle (24 peers), best: #198713 (0xbae3…9a95), finalized #198712 (0x3c8a…4c3e), ⬇ 24.4kiB/s ⬆ 24.8kiB/s                                                                                                                    
2019-09-06 10:26:38 Imported #198714 (0x477c…03ae)                                                                                                                                                                                            
2019-09-06 10:26:38 Starting parachain attestation session on top of parent 0x477cfa3a97a2c7737a920be25f156bd975a24653ad82eeee8c4e431a9f2a03ae. Local parachain duty is None                                                                  
2019-09-06 10:26:43 Idle (26 peers), best: #198714 (0x477c…03ae), finalized #198714 (0x477c…03ae), ⬇ 41.9kiB/s ⬆ 50.7kiB/s                                                                                                                    
2019-09-06 10:26:43 Imported #198715 (0x131b…a115)                                                                                                                                                                                            
2019-09-06 10:26:43 Starting parachain attestation session on top of parent 0x131bbe03db1f54bcb70dc3692a5b7261b7a41f2dc86c7151f0a9e2167944a115. Local parachain duty is None                                                                  
2019-09-06 10:26:48 Idle (28 peers), best: #198715 (0x131b…a115), finalized #198715 (0x131b…a115), ⬇ 36.9kiB/s ⬆ 50.4kiB/s                                                                                                                    
2019-09-06 10:26:48 Imported #198716 (0x18e2…2fd9)                                                                                                                                                                                            
2019-09-06 10:26:49 Starting parachain attestation session on top of parent 0x18e282ccb60f623a200439ad76faf1cfbe0de32d9117aadacbba5f6440c52fd9. Local parachain duty is None                                                                  
2019-09-06 10:26:49 Imported #198716 (0x471f…bb39)                                                                                                                                                                                            
2019-09-06 10:26:53 Idle (26 peers), best: #198716 (0x18e2…2fd9), finalized #198715 (0x131b…a115), ⬇ 18.5kiB/s ⬆ 26.9kiB/s                                                                                                                    
2019-09-06 10:26:55 Imported #198717 (0x9201…aba9)                                                                                                                                                                                            
2019-09-06 10:26:55 Starting parachain attestation session on top of parent 0x920161561076f7f7bd6184ee8938fafe615319e5484fce4ae534042c3179aba9. Local parachain duty is None
2019-09-06 10:26:57 Reorg from #198717,0x9201…aba9 to #198717,0x95dd…0258, common ancestor #198716,0x18e2…2fd9                                                                                                                                
2019-09-06 10:26:57 Imported #198717 (0x95dd…0258)                                                                                                                                                                                            
2019-09-06 10:26:57 Starting parachain attestation session on top of parent 0x95dd1950a38159dd723fae7992b197d8d9419d677abc733d6c9ad127498c0258. Local parachain duty is None                                                                  
2019-09-06 10:26:58 Idle (24 peers), best: #198717 (0x95dd…0258), finalized #198716 (0x18e2…2fd9), ⬇ 23.1kiB/s ⬆ 24.2kiB/s                                                                                                                    
2019-09-06 10:27:00 Imported #198718 (0x5062…ceed)                                                                                                                                                                                            
2019-09-06 10:27:00 Error running offchain workers at Number(198718): UnknownBlock("Unknown state for block Number(198718)")

@soc1c
Copy link
Contributor Author

soc1c commented Sep 6, 2019

My validator node panicked once again:

2019-09-06 17:46:57 Idle (18 peers), best: #203104 (0x4756…913e), finalized #203103 (0xfdfe…19b9), ⬇ 28.4kiB/s ⬆ 37.7kiB/s
2019-09-06 17:47:01 Imported #203105 (0xe31f…f822)
2019-09-06 17:47:01 Starting parachain attestation session on top of parent 0xe31f4627bcf0c9569e606db58c94d376bce70f5a12c3220f033f94a3ae90f822. Local parachain duty is None
2019-09-06 17:47:02 Idle (24 peers), best: #203105 (0xe31f…f822), finalized #203104 (0x4756…913e), ⬇ 39.4kiB/s ⬆ 43.2kiB/s
2019-09-06 17:47:06 Imported #203106 (0xdc74…9e2b)
2019-09-06 17:47:06 Starting parachain attestation session on top of parent 0xdc74ce1715a0cb0bc8376805e02ac05be51cd1eba0b4d6c69bc18ccf53869e2b. Local parachain duty is None
2019-09-06 17:47:07 Reorg from #203106,0xdc74…9e2b to #203106,0xaf17…07ae, common ancestor #203105,0xe31f…f822
2019-09-06 17:47:07 Imported #203106 (0xaf17…07ae)
2019-09-06 17:47:07 Starting parachain attestation session on top of parent 0xaf17a3f3c8691c538a77c59500ea268ae2bffe7ecbf8c9415108560db92b07ae. Local parachain duty is None
2019-09-06 17:47:07 Idle (20 peers), best: #203106 (0xaf17…07ae), finalized #203105 (0xe31f…f822), ⬇ 29.1kiB/s ⬆ 44.2kiB/s
2019-09-06 17:47:12 Idle (17 peers), best: #203106 (0xaf17…07ae), finalized #203105 (0xe31f…f822), ⬇ 13.8kiB/s ⬆ 15.8kiB/s
2019-09-06 17:47:12 Imported #203107 (0xa2ba…64c8)
2019-09-06 17:47:12 Error running offchain workers at Number(203107): UnknownBlock("Unknown state for block Number(203107)")

====================

stack backtrace:
   0: substrate_panic_handler::set::{{closure}}::h82cae3535dcc52cb (0x55bc0b5ef0a0)
   1: std::panicking::rust_panic_with_hook::hffcefc09751839d1 (0x55bc0b72c6a9)
             at src/libstd/panicking.rs:481
   2: std::panicking::continue_panic_fmt::hc0f142c930c846fc (0x55bc0b72c142)
             at src/libstd/panicking.rs:384
   3: rust_begin_unwind (0x55bc0b72c026)
             at src/libstd/panicking.rs:311
   4: core::panicking::panic_fmt::h2daf88b2616ca2b2 (0x55bc0b74dedd)
             at src/libcore/panicking.rs:85
   5: core::result::unwrap_failed::h62dbd5a8e5560593 (0x55bc0a95dcee)
   6: <substrate_state_machine::ext::Ext<H,N,B,T,O> as substrate_state_machine::Externalities<H>>::storage::hb493a5df659ab9f3 (0x55bc0a62b960)
   7: <substrate_executor::wasm_executor::FunctionExecutor<E> as wasmi::host::Externals>::invoke_index::hb45ea024086571d1 (0x55bc0a50625d)
   8: wasmi::func::FuncInstance::invoke::h5b09d54c708b1b31 (0x55bc0a3c6a9a)
   9: wasmi::runner::Interpreter::start_execution::h4858fcd74c407050 (0x55bc0a72a408)
  10: wasmi::func::FuncInstance::invoke::h5b09d54c708b1b31 (0x55bc0a3c6b2c)
  11: substrate_executor::wasm_executor::WasmExecutor::call_in_wasm_module::h07b2a5c593c7addf (0x55bc0a530757)
  12: <substrate_executor::native_executor::NativeExecutor<D> as substrate_state_machine::CodeExecutor<substrate_primitives::hasher::blake2::Blake2Hasher>>::call::heef7ff72239b90f2 (0x55bc0a7b961b)
  13: substrate_state_machine::StateMachine<H,N,B,T,O,Exec>::execute_aux::h40e0fef5aad580f3 (0x55bc0a8e1990)
  14: substrate_state_machine::StateMachine<H,N,B,T,O,Exec>::execute_using_consensus_failure_handler::h260d5fc78a88d573 (0x55bc0a8ffcf0)
  15: <substrate_client::call_executor::LocalCallExecutor<B,E> as substrate_client::call_executor::CallExecutor<Block,substrate_primitives::hasher::blake2::Blake2Hasher>>::contextual_call::hc9d14c9a3c6e6a24 (0x55bc0a7d7c6b)
  16: <substrate_client::client::Client<B,E,Block,RA> as substrate_client::runtime_api::CallRuntimeAt<Block>>::call_api_at::hbf93113da3de111a (0x55bc0a43786a)
  17: <polkadot_runtime::RuntimeApiImpl<RuntimeApiImplCall> as substrate_client::runtime_api::Core<<polkadot_runtime::Runtime as sr_primitives::traits::GetNodeBlockType>::NodeBlock>>::Core_initialize_block_runtime_api_impl::hb7bdde40663ba0bc (0x55bc0a392239)
  18: substrate_client::runtime_api::Core::initialize_block::hb74ada721fa5e557 (0x55bc0a3576d7)
  19: <substrate_client::call_executor::LocalCallExecutor<B,E> as substrate_client::call_executor::CallExecutor<Block,substrate_primitives::hasher::blake2::Blake2Hasher>>::contextual_call::h646c30bb0a86b123 (0x55bc0a7cd5c8)
  20: <substrate_client::client::Client<B,E,Block,RA> as substrate_client::runtime_api::CallRuntimeAt<Block>>::call_api_at::h6f50441f47597513 (0x55bc0a43442f)
  21: polkadot_primitives::parachain::runtime_decl_for_ParachainHost::validators_call_api_at::hee4b5cbc355a5bcd (0x55bc0a8d01ce)
  22: polkadot_primitives::parachain::ParachainHost::validators::h5e101c52657c0562 (0x55bc0a372edb)
  23: polkadot_validation::ParachainValidation<C,N,P>::get_or_instantiate::he6ddef975679aaa6 (0x55bc0a45d044)
  24: <futures::stream::for_each::ForEach<S,F,U> as futures::future::Future>::poll::h989eff7555f1e838 (0x55bc0aa185e8)
  25: <futures::future::select::Select<A,B> as futures::future::Future>::poll::h072fc464b56269e6 (0x55bc0a752ff5)
  26: futures::future::chain::Chain<A,B,C>::poll::h31357d6fbbcb8b36 (0x55bc0aaa6679)
  27: futures::task_impl::std::set::h8ed5be9f01d42e6d (0x55bc0ab377b2)
  28: tokio_current_thread::CurrentRunner::set_spawn::h5abebce908ba8889 (0x55bc0a391a6b)
  29: tokio_current_thread::scheduler::Scheduler<U>::tick::h997a8e2f57321237 (0x55bc0aa6bdb9)
  30: tokio_current_thread::Entered<P>::block_on::hf670e5584a8dc800 (0x55bc0a391c66)
  31: std::thread::local::LocalKey<T>::with::hc4580ec2b7427ee3 (0x55bc0a817224)
  32: std::thread::local::LocalKey<T>::with::hedef1bfa9db36c0b (0x55bc0a81a6fb)
  33: std::thread::local::LocalKey<T>::with::h5f5d9fce2829a137 (0x55bc0a80c90b)
  34: std::thread::local::LocalKey<T>::with::h80852b028e491352 (0x55bc0a810961)
  35: tokio::runtime::current_thread::runtime::Runtime::block_on::h26dab3a938557965 (0x55bc0a5a6815)
  36: std::sys_common::backtrace::__rust_begin_short_backtrace::hbddf5d9de90b8ce0 (0x55bc0a482c74)
  37: std::panicking::try::do_call::hf4bbd8c887fb36ec (0x55bc0a9e15c9)
  38: __rust_maybe_catch_panic (0x55bc0b7355ca)
             at src/libpanic_unwind/lib.rs:82
  39: core::ops::function::FnOnce::call_once{{vtable.shim}}::h68fd22c3217e11e2 (0x55bc0a589ff4)
  40: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h42806b83647d4c79 (0x55bc0b71a44f)
             at /rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/liballoc/boxed.rs:746
  41: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h83c921c8e826dd1d (0x55bc0b734950)
             at /rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/liballoc/boxed.rs:746
      std::sys_common::thread::start_thread::h2613204ce513782e
             at src/libstd/sys_common/thread.rs:13
      std::sys::unix::thread::Thread::new::thread_start::h4570080769500bcd
             at src/libstd/sys/unix/thread.rs:79
  42: start_thread (0x7f396c61c57f)
  43: __clone (0x7f396c5300e3)
  44: <unknown> (0x0)

Thread '<unnamed>' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Invalid state root: 0xc6c6c87518a09803b2d809f4a4ed9b6f152d39c939f0430c25b37605de9c1331"', src/libcore/result.rs:999

This is a bug. Please report it at:

        https://github.com/paritytech/polkadot/issues/new

@soc1c
Copy link
Contributor Author

soc1c commented Sep 6, 2019

Different node, same crash

2019-09-06 18:29:46 Idle (21 peers), best: #203531 (0xdf56…8783), finalized #203530 (0x47b6…7a0d), ⬇ 23.5kiB/s ⬆ 33.1kiB/s
2019-09-06 18:29:49 Imported #203532 (0x545e…7ad0)
2019-09-06 18:29:49 Starting parachain attestation session on top of parent 0x545e340503679dbb0547f029fa8802dc3db5b89983a83cd9292d276f5af47ad0. Local parachain duty is None
2019-09-06 18:29:50 Imported #203532 (0x5a06…c561)
2019-09-06 18:29:51 Idle (19 peers), best: #203532 (0x545e…7ad0), finalized #203531 (0xdf56…8783), ⬇ 30.7kiB/s ⬆ 41.7kiB/s
2019-09-06 18:29:55 Imported #203533 (0xf2d7…a7fc)
2019-09-06 18:29:55 Starting parachain attestation session on top of parent 0xf2d76f63568346aaf19872fbe06d2496be524879b063cf125a21f6299577a7fc. Local parachain duty is None
2019-09-06 18:29:55 Imported #203533 (0x43a0…2d75)
2019-09-06 18:29:56 Idle (16 peers), best: #203533 (0xf2d7…a7fc), finalized #203531 (0xdf56…8783), ⬇ 24.2kiB/s ⬆ 20.2kiB/s
2019-09-06 18:30:01 Idle (17 peers), best: #203533 (0xf2d7…a7fc), finalized #203532 (0x545e…7ad0), ⬇ 22.0kiB/s ⬆ 27.7kiB/s
2019-09-06 18:30:01 Imported #203534 (0x50d3…c8b0)
2019-09-06 18:30:02 Error running offchain workers at Number(203534): UnknownBlock("Number(203534)")
2019-09-06 18:30:02 Discovered new external address for our node: /ip4/10.12.4.43/tcp/34567/p2p/QmU4ahXCJBvZ8adrFGdEES2iyfQkjVwUEh5dWNrvaCt48H

====================

stack backtrace:
   0: substrate_panic_handler::set::{{closure}}::h82cae3535dcc52cb (0x5591f589c0a0)
   1: std::panicking::rust_panic_with_hook::hffcefc09751839d1 (0x5591f59d96a9)
             at src/libstd/panicking.rs:481
   2: std::panicking::continue_panic_fmt::hc0f142c930c846fc (0x5591f59d9142)
             at src/libstd/panicking.rs:384
   3: rust_begin_unwind (0x5591f59d9026)
             at src/libstd/panicking.rs:311
   4: core::panicking::panic_fmt::h2daf88b2616ca2b2 (0x5591f59faedd)
             at src/libcore/panicking.rs:85
   5: core::result::unwrap_failed::h62dbd5a8e5560593 (0x5591f4c0acee)
   6: <substrate_state_machine::ext::Ext<H,N,B,T,O> as substrate_state_machine::Externalities<H>>::storage::h8fc321cc53d883d5 (0x5591f48d8960)
   7: <substrate_executor::wasm_executor::FunctionExecutor<E> as wasmi::host::Externals>::invoke_index::hb45ea024086571d1 (0x5591f47b325d)
   8: wasmi::func::FuncInstance::invoke::h5b09d54c708b1b31 (0x5591f4673a9a)
   9: wasmi::runner::Interpreter::start_execution::h4858fcd74c407050 (0x5591f49d7408)
  10: wasmi::func::FuncInstance::invoke::h5b09d54c708b1b31 (0x5591f4673b2c)
  11: substrate_executor::wasm_executor::WasmExecutor::call_in_wasm_module::h07b2a5c593c7addf (0x5591f47dd757)
  12: <substrate_executor::native_executor::NativeExecutor<D> as substrate_state_machine::CodeExecutor<substrate_primitives::hasher::blake2::Blake2Hasher>>::call::heef7ff72239b90f2 (0x5591f4a6661b)
  13: substrate_state_machine::StateMachine<H,N,B,T,O,Exec>::execute_aux::h40e0fef5aad580f3 (0x5591f4b8e990)
  14: substrate_state_machine::StateMachine<H,N,B,T,O,Exec>::execute_using_consensus_failure_handler::h260d5fc78a88d573 (0x5591f4baccf0)
  15: <substrate_client::call_executor::LocalCallExecutor<B,E> as substrate_client::call_executor::CallExecutor<Block,substrate_primitives::hasher::blake2::Blake2Hasher>>::contextual_call::hc9d14c9a3c6e6a24 (0x5591f4a84c6b)
  16: <substrate_client::client::Client<B,E,Block,RA> as substrate_client::runtime_api::CallRuntimeAt<Block>>::call_api_at::hbf93113da3de111a (0x5591f46e486a)
  17: <polkadot_runtime::RuntimeApiImpl<RuntimeApiImplCall> as substrate_client::runtime_api::Core<<polkadot_runtime::Runtime as sr_primitives::traits::GetNodeBlockType>::NodeBlock>>::Core_initialize_block_runtime_api_impl::hb7bdde40663ba0bc (0x5591f463f239)
  18: substrate_client::runtime_api::Core::initialize_block::hb74ada721fa5e557 (0x5591f46046d7)
  19: <substrate_client::call_executor::LocalCallExecutor<B,E> as substrate_client::call_executor::CallExecutor<Block,substrate_primitives::hasher::blake2::Blake2Hasher>>::contextual_call::h5a3f600994c9dc21 (0x5591f4a79008)
  20: <substrate_client::client::Client<B,E,Block,RA> as substrate_client::runtime_api::CallRuntimeAt<Block>>::call_api_at::h59fe8550150a9186 (0x5591f46e07af)
  21: polkadot_primitives::parachain::runtime_decl_for_ParachainHost::duty_roster_call_api_at::h7d8a7e7b941f36f4 (0x5591f4b7d63e)
  22: polkadot_primitives::parachain::ParachainHost::duty_roster::h90bf5a611c8b7102 (0x5591f46201a7)
  23: polkadot_validation::ParachainValidation<C,N,P>::get_or_instantiate::he6ddef975679aaa6 (0x5591f470a1aa)
  24: <futures::stream::for_each::ForEach<S,F,U> as futures::future::Future>::poll::h989eff7555f1e838 (0x5591f4cc55e8)
  25: <futures::future::select::Select<A,B> as futures::future::Future>::poll::h072fc464b56269e6 (0x5591f49ffff5)
  26: futures::future::chain::Chain<A,B,C>::poll::h31357d6fbbcb8b36 (0x5591f4d53679)
  27: futures::task_impl::std::set::h8ed5be9f01d42e6d (0x5591f4de47b2)
  28: tokio_current_thread::CurrentRunner::set_spawn::h5abebce908ba8889 (0x5591f463ea6b)
  29: tokio_current_thread::scheduler::Scheduler<U>::tick::h997a8e2f57321237 (0x5591f4d18db9)
  30: tokio_current_thread::Entered<P>::block_on::hf670e5584a8dc800 (0x5591f463ec66)
  31: std::thread::local::LocalKey<T>::with::hc4580ec2b7427ee3 (0x5591f4ac4224)
  32: std::thread::local::LocalKey<T>::with::hedef1bfa9db36c0b (0x5591f4ac76fb)
  33: std::thread::local::LocalKey<T>::with::h5f5d9fce2829a137 (0x5591f4ab990b)
  34: std::thread::local::LocalKey<T>::with::h80852b028e491352 (0x5591f4abd961)
  35: tokio::runtime::current_thread::runtime::Runtime::block_on::h26dab3a938557965 (0x5591f4853815)
  36: std::sys_common::backtrace::__rust_begin_short_backtrace::hbddf5d9de90b8ce0 (0x5591f472fc74)
  37: std::panicking::try::do_call::hf4bbd8c887fb36ec (0x5591f4c8e5c9)
  38: __rust_maybe_catch_panic (0x5591f59e25ca)
             at src/libpanic_unwind/lib.rs:82
  39: core::ops::function::FnOnce::call_once{{vtable.shim}}::h68fd22c3217e11e2 (0x5591f4836ff4)
  40: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h42806b83647d4c79 (0x5591f59c744f)
             at /rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/liballoc/boxed.rs:746
  41: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h83c921c8e826dd1d (0x5591f59e1950)
             at /rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/liballoc/boxed.rs:746
      std::sys_common::thread::start_thread::h2613204ce513782e
             at src/libstd/sys_common/thread.rs:13
      std::sys::unix::thread::Thread::new::thread_start::h4570080769500bcd
             at src/libstd/sys/unix/thread.rs:79
  42: start_thread (0x7f304aae657f)
  43: __clone (0x7f304a9fa0e3)
  44: <unknown> (0x0)

Thread '<unnamed>' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Invalid state root: 0x86e5c5d70bbb53707f744effc2ad811fd411434868639473f9e243c7bf33f275"', src/libcore/result.rs:999

This is a bug. Please report it at:

        https://github.com/paritytech/polkadot/issues/new

@bkchr
Copy link
Member

bkchr commented Sep 6, 2019

Yeah there as a reorg "Reorg from #198717,0x9201…aba9 to #198717".

@soc1c soc1c mentioned this issue Sep 6, 2019
Closed
@iammelea
Copy link

iammelea commented Sep 6, 2019

why happens this reorg? thanks

@soc1c soc1c mentioned this issue Sep 9, 2019
Closed
This was referenced Sep 26, 2019
Closed
Closed
Closed
Closed
This was referenced Oct 29, 2019
Closed
Closed
@gavofyork gavofyork changed the title Error running offchain workers at Number(198718) Panic on reorg Nov 13, 2019
@bkchr bkchr closed this as completed May 26, 2021
imstar15 pushed a commit to imstar15/polkadot that referenced this issue Aug 25, 2021
@gavofyork
Fix condition (paritytech#424)
42c18b8 
Closes paritytech#419
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@gavofyork @bkchr @iammelea @soc1c