Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

State sync unexpectedly fail #3042

Closed
bowenwang1996 opened this issue Jul 26, 2020 · 0 comments · Fixed by #3043
Closed

State sync unexpectedly fail #3042

bowenwang1996 opened this issue Jul 26, 2020 · 0 comments · Fixed by #3043
Assignees
Labels
C-bug Category: This is a bug P-critical Priority: critical

Comments

@bowenwang1996
Copy link
Collaborator

On betanet we observe that state sync fail with

Jul 26 14:33:54.650 ERROR sync: Sync: Unexpected error: DB Not Found Error: 3fqj33NedbEA4fMLfVxU7odK2bouHuRgFBRhryAc1Drm
 Cause: Unknown
 Backtrace:    0: failure::backtrace::Backtrace::new
   1: <near_chain::error::Error as core::convert::From<near_primitives::errors::EpochError>>::from
   2: <neard::runtime::NightshadeRuntime as near_chain::types::RuntimeAdapter>::get_epoch_id_from_prev_block
   3: near_client::sync::StateSync::request_shard
   4: near_client::sync::StateSync::sync_shards_status
   5: near_client::sync::StateSync::run
   6: near_client::client_actor::ClientActor::sync
   7: <F as actix::utils::TimerFuncBox<A>>::call
   8: <actix::utils::TimerFunc<A> as actix::fut::ActorFuture>::poll
   9: <actix::contextimpl::ContextFut<A,C> as core::future::future::Future>::poll
  10: tokio::runtime::task::raw::poll
  11: tokio::task::local::LocalSet::tick
  12: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  13: actix_rt::runtime::Runtime::block_on
  14: near::main
  15: std::rt::lang_start_internal::{{closure}}::{{closure}}
             at src/libstd/rt.rs:52
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at src/libstd/sys_common/backtrace.rs:130
  16: main
  17: __libc_start_main
  18: _start

It appears that information needed for state sync is already garbage collected.

@bowenwang1996 bowenwang1996 self-assigned this Jul 26, 2020
@bowenwang1996 bowenwang1996 added C-bug Category: This is a bug P-critical Priority: critical labels Jul 26, 2020
bowenwang1996 added a commit that referenced this issue Jul 26, 2020
#3043)

When we do `reset_data_pre_state_sync` before running state sync, we garbage collect everything up to sync block, which includes the prev block of the sync block. However, when we run state sync, we use `prev_hash` of sync block inside `possible_targets` [here](https://github.com/nearprotocol/nearcore/blob/f65d2441d66fa13f7933c23ba833311677ce1e96/chain/client/src/sync.rs#L801). Therefore, we cannot garbage collect `prev_hash` of sync block. Fixes #3042.

Test plan
---------
`test_data_reset_before_state_sync`
mhalambek pushed a commit that referenced this issue Aug 3, 2020
#3043)

When we do `reset_data_pre_state_sync` before running state sync, we garbage collect everything up to sync block, which includes the prev block of the sync block. However, when we run state sync, we use `prev_hash` of sync block inside `possible_targets` [here](https://github.com/nearprotocol/nearcore/blob/f65d2441d66fa13f7933c23ba833311677ce1e96/chain/client/src/sync.rs#L801). Therefore, we cannot garbage collect `prev_hash` of sync block. Fixes #3042.

Test plan
---------
`test_data_reset_before_state_sync`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Category: This is a bug P-critical Priority: critical
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant