channel recv_timeout assert failure

[UncP]

thread 'snapshot worker::test_node_conf_change_safe' panicked at 'assertion failed: `(left == right)`
  left: `139821685563168`,
 right: `0`', libstd/sync/mpsc/shared.rs:253:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:68
             at libstd/sys_common/backtrace.rs:57
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:380
   3: std::panicking::default_hook
             at libstd/panicking.rs:396
   4: tikv::util::panic_hook::track_hook::{{closure}}
             at src/util/panic_hook.rs:59
   5: <std::thread::local::LocalKey<T>>::try_with
             at /checkout/src/libstd/thread/local.rs:377
   6: <std::thread::local::LocalKey<T>>::with
             at /checkout/src/libstd/thread/local.rs:288
   7: tikv::util::panic_hook::track_hook
             at src/util/panic_hook.rs:53
   8: core::ops::function::Fn::call
             at /checkout/src/libcore/ops/function.rs:73
   9: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:577
  10: std::panicking::begin_panic
             at libstd/panicking.rs:537
  11: std::panicking::begin_panic_fmt
             at libstd/panicking.rs:521
  12: <std::sync::mpsc::shared::Packet<T>>::decrement
             at /checkout/src/libstd/macros.rs:72
  13: <std::sync::mpsc::shared::Packet<T>>::recv
             at /checkout/src/libstd/sync/mpsc/shared.rs:232
  14: <std::sync::mpsc::Receiver<T>>::recv_deadline
             at /checkout/src/libstd/sync/mpsc/mod.rs:1387
  15: <std::sync::mpsc::Receiver<T>>::recv_timeout
             at /checkout/src/libstd/sync/mpsc/mod.rs:1300
  16: tikv::util::worker::fill_task_batch
             at ./src/util/worker/mod.rs:314
  17: tikv::util::worker::poll
             at ./src/util/worker/mod.rs:280
  18: <tikv::util::worker::Worker<T>>::start_with_timer::{{closure}}
             at ./src/util/worker/mod.rs:365

in 15, the receiver had an assert failure when recv_timeout.

rust version:

rustc 1.25.0-nightly (73ac5d6a8 2018-01-11)

[oli-obk]

is this randomly happening or is there a reproducible example?

[gorup]

I can reproduce this myself. using version rustc 1.27.0-nightly (ac3c2288f 2018-04-18)

My stack trace:

thread 'toby_snd_34eb687f-749c-4412-99df-e92080069107' panicked at 'assertion failed: `(left == right)`
  left: `4399890432`,
 right: `0`', libstd/sync/mpsc/shared.rs:253:13


stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:205
   3: std::panicking::default_hook
             at libstd/panicking.rs:221
   4: <std::panicking::begin_panic::PanicPayload<A> as core::panic::BoxMeUp>::get
             at libstd/panicking.rs:457
   5: std::panicking::try::do_call
             at libstd/panicking.rs:344
   6: <core::fmt::Arguments<'a> as core::clone::Clone>::clone
             at /Users/travis/build/rust-lang/rust/src/libstd/macros.rs:75
   7: <core::fmt::Arguments<'a> as core::clone::Clone>::clone
             at /Users/travis/build/rust-lang/rust/src/libstd/sync/mpsc/shared.rs:232
   8: <std::sys::unix::time::inner::Instant as core::cmp::PartialOrd>::ge
             at /Users/travis/build/rust-lang/rust/src/libstd/sync/mpsc/mod.rs:1387
   9: <std::sys::unix::time::inner::Instant as core::cmp::PartialOrd>::ge
             at /Users/travis/build/rust-lang/rust/src/libstd/sync/mpsc/mod.rs:1300
  10: tobytcp::TobySender::send_data
             at /Users/gorup/.cargo/registry/src/github.com-1ecc6299db9ec823/tobytcp-0.9.3/src/lib.rs:45
  11: tobytcp::TobyMessenger::start::{{closure}}
             at /Users/gorup/.cargo/registry/src/github.com-1ecc6299db9ec823/tobytcp-0.9.3/src/lib.rs:257
  12: std::sys_common::mutex::Mutex::destroy
             at /Users/travis/build/rust-lang/rust/src/libstd/sys_common/backtrace.rs:136
  13: core::sync::atomic::fence
             at /Users/travis/build/rust-lang/rust/src/libstd/thread/mod.rs:409
  14: core::ptr::swap_nonoverlapping_bytes
             at /Users/travis/build/rust-lang/rust/src/libstd/panic.rs:296
  15: std::sync::mpsc::blocking::SignalToken::cast_from_usize
             at /Users/travis/build/rust-lang/rust/src/libstd/panicking.rs:304
  16: panic_unwind::dwarf::eh::read_encoded_pointer
             at libpanic_unwind/lib.rs:105
  17: std::sync::mpsc::blocking::SignalToken::cast_from_usize
             at /Users/travis/build/rust-lang/rust/src/libstd/panicking.rs:283
  18: std::sync::mpsc::blocking::SignalToken::cast_from_usize
             at /Users/travis/build/rust-lang/rust/src/libstd/panic.rs:361
  19: core::sync::atomic::fence
             at /Users/travis/build/rust-lang/rust/src/libstd/thread/mod.rs:408
  20: core::str::from_utf8_unchecked_mut
             at /Users/travis/build/rust-lang/rust/src/liballoc/boxed.rs:637
  21: std::sys_common::thread::start_thread
             at /Users/travis/build/rust-lang/rust/src/liballoc/boxed.rs:647
             at libstd/sys_common/thread.rs:24
  22: std::sys::unix::thread::Thread::new::thread_start
             at libstd/sys/unix/thread.rs:90
  23: _pthread_body
  24: _pthread_start

The code that is calling recv_timeout is here [1]. The code using this library that I can reproduce this bug with is in a private repo, but I could give you access and you could run it yourself if you'd like. Let me know!

[1] https://github.com/gorup/tobytcp/blob/master/src/lib.rs#L45

[UncP]

#48927
Thanks, I have already filed an issue in February.