Double panic when trying to run `cargo test` on Win10 #249

ErichDonGubler · 2018-10-01T18:15:16Z

For some reason tests inside of src/dynamic_honey_badger/votes.rs are throwing panics inside of panics. This causes an immediate abort, and blocks tests from running. This seems specific to my Windows 10 machine, since master seems to pass just fine in Travis (a *nix environment).

Output of cargo test:

C:\Users\egubler\workspace\personal\pull-requests\hbbft>cargo test
    Finished dev [unoptimized + debuginfo] target(s) in 0.53s
     Running target\debug\deps\hbbft-0cddedd180c2030e.exe

running 3 tests
test broadcast::merkle::tests::test_merkle ... ok
test dynamic_honey_badger::votes::tests::test_pending_votes ... FAILED
thread panicked while panicking. aborting.
error: test failed, to rerun pass '--lib'

Output of rustup show:

active toolchain
----------------

stable-x86_64-pc-windows-gnu (default)
rustc 1.29.0 (aa3ca1994 2018-09-11)

Output of RUST_BACKTRACE=1 cargo test in MSYS2 environment

~/…/pull-requests/hbbft master$ RUST_BACKTRACE=1 cargo test
    Finished dev [unoptimized + debuginfo] target(s) in 0.57s
     Running target\debug\deps\hbbft-0cddedd180c2030e.exe

running 3 tests
test broadcast::merkle::tests::test_merkle ... ok
test dynamic_honey_badger::votes::tests::test_committed_votes ... FAILED
test dynamic_honey_badger::votes::tests::test_pending_votes ... FAILED

failures:

---- dynamic_honey_badger::votes::tests::test_committed_votes stdout ----
thread 'dynamic_honey_badger::votes::tests::test_committed_votes' panicked at 'Failed to drop `SecretKey`: Failed to `munlock` 32 bytes starting at address: 0x1a90c30', C:\Users\egubler\.cargo\git\checkouts\threshold_crypto-3605c1a1492f376b\02109b5\src\lib.rs:308:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::windows::backtrace::unwind_backtrace
             at libstd\sys\windows\backtrace/mod.rs:104
             at libstd\sys\windows\backtrace/mod.rs:91
   1: std::sys_common::backtrace::print
             at libstd\sys_common/backtrace.rs:71
             at libstd\sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:221
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:475
   5: std::panicking::continue_panic_fmt
             at libstd/panicking.rs:390
   6: std::panicking::begin_panic_fmt
             at libstd/panicking.rs:345
   7: <threshold_crypto::SecretKey as core::ops::drop::Drop>::drop
             at C:\Users\egubler\.cargo\git\checkouts\threshold_crypto-3605c1a1492f376b\02109b5\src/lib.rs:308
   8: core::ptr::drop_in_place
             at C:\projects\rust\src\libcore/ptr.rs:59
   9: core::ptr::drop_in_place
             at C:\projects\rust\src\libcore/ptr.rs:59
  10: core::mem::drop
             at C:\projects\rust\src\libcore/mem.rs:795
  11: core::ops::function::FnMut::call_mut
             at C:\projects\rust\src\libcore\ops/function.rs:146
  12: core::iter::iterator::Iterator::for_each::{{closure}}
             at C:\projects\rust\src\libcore\iter/iterator.rs:553
  13: core::iter::iterator::Iterator::fold::{{closure}}
             at C:\projects\rust\src\libcore\iter/iterator.rs:1634
  14: core::iter::iterator::Iterator::try_fold
             at C:\projects\rust\src\libcore\iter/iterator.rs:1522
  15: core::iter::iterator::Iterator::fold
             at C:\projects\rust\src\libcore\iter/iterator.rs:1634
  16: core::iter::iterator::Iterator::for_each
             at C:\projects\rust\src\libcore\iter/iterator.rs:553
  17: <alloc::collections::btree::map::IntoIter<K, V> as core::ops::drop::Drop>::drop
             at C:\projects\rust\src\liballoc\collections\btree/map.rs:1313
  18: core::ptr::drop_in_place
             at C:\projects\rust\src\libcore/ptr.rs:59
  19: core::mem::drop
             at C:\projects\rust\src\libcore/mem.rs:795
  20: <alloc::collections::btree::map::BTreeMap<K, V> as core::ops::drop::Drop>::drop
             at C:\projects\rust\src\liballoc\collections\btree/map.rs:144
  21: core::ptr::drop_in_place
             at C:\projects\rust\src\libcore/ptr.rs:59
  22: <hbbft::messaging::NetworkInfo<N>>::generate_map
             at src/messaging.rs:401
  23: hbbft::dynamic_honey_badger::votes::tests::setup
             at src\dynamic_honey_badger/votes.rs:203
  24: hbbft::dynamic_honey_badger::votes::tests::test_committed_votes
             at src\dynamic_honey_badger/votes.rs:279
  25: hbbft::__test::TESTS::{{closure}}
             at src\dynamic_honey_badger/votes.rs:275
  26: core::ops::function::FnOnce::call_once
             at C:\projects\rust\src\libcore\ops/function.rs:223
  27: <F as alloc::boxed::FnBox<A>>::call_box
             at libtest/lib.rs:1451
             at C:\projects\rust\src\libcore\ops/function.rs:223
             at C:\projects\rust\src\liballoc/boxed.rs:642
  28: _rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:105

---- dynamic_honey_badger::votes::tests::test_pending_votes stdout ----
thread 'dynamic_honey_badger::votes::tests::test_pending_votes' panicked at 'Failed to drop `SecretKey`: Failed to `munlock` 32 bytes starting at address: 0x1aa0620', C:\Users\egubler\.cargo\git\checkouts\threshold_crypto-3605c1a1492f376b\02109b5\src\lib.rs:308:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::windows::backtrace::unwind_backtrace
             at libstd\sys\windows\backtrace/mod.rs:104
             at libstd\sys\windows\backtrace/mod.rs:91
   1: std::sys_common::backtrace::print
             at libstd\sys_common/backtrace.rs:71
             at libstd\sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:221
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:475
   5: std::panicking::continue_panic_fmt
             at libstd/panicking.rs:390
   6: std::panicking::begin_panic_fmt
             at libstd/panicking.rs:345
   7: <threshold_crypto::SecretKey as core::ops::drop::Drop>::drop
             at C:\Users\egubler\.cargo\git\checkouts\threshold_crypto-3605c1a1492f376b\02109b5\src/lib.rs:308
   8: core::ptr::drop_in_place
             at C:\projects\rust\src\libcore/ptr.rs:59
   9: core::ptr::drop_in_place
             at C:\projects\rust\src\libcore/ptr.rs:59
  10: core::mem::drop
             at C:\projects\rust\src\libcore/mem.rs:795
  11: core::ops::function::FnMut::call_mut
             at C:\projects\rust\src\libcore\ops/function.rs:146
  12: core::iter::iterator::Iterator::for_each::{{closure}}
             at C:\projects\rust\src\libcore\iter/iterator.rs:553
  13: core::iter::iterator::Iterator::fold::{{closure}}
             at C:\projects\rust\src\libcore\iter/iterator.rs:1634
  14: core::iter::iterator::Iterator::try_fold
             at C:\projects\rust\src\libcore\iter/iterator.rs:1522
  15: core::iter::iterator::Iterator::fold
             at C:\projects\rust\src\libcore\iter/iterator.rs:1634
  16: core::iter::iterator::Iterator::for_each
             at C:\projects\rust\src\libcore\iter/iterator.rs:553
  17: <alloc::collections::btree::map::IntoIter<K, V> as core::ops::drop::Drop>::drop
             at C:\projects\rust\src\liballoc\collections\btree/map.rs:1313
  18: core::ptr::drop_in_place
             at C:\projects\rust\src\libcore/ptr.rs:59
  19: core::mem::drop
             at C:\projects\rust\src\libcore/mem.rs:795
  20: <alloc::collections::btree::map::BTreeMap<K, V> as core::ops::drop::Drop>::drop
             at C:\projects\rust\src\liballoc\collections\btree/map.rs:144
  21: core::ptr::drop_in_place
             at C:\projects\rust\src\libcore/ptr.rs:59
  22: <hbbft::messaging::NetworkInfo<N>>::generate_map
             at src/messaging.rs:401
  23: hbbft::dynamic_honey_badger::votes::tests::setup
             at src\dynamic_honey_badger/votes.rs:203
  24: hbbft::dynamic_honey_badger::votes::tests::test_pending_votes
             at src\dynamic_honey_badger/votes.rs:226
  25: hbbft::__test::TESTS::{{closure}}
             at src\dynamic_honey_badger/votes.rs:222
  26: core::ops::function::FnOnce::call_once
             at C:\projects\rust\src\libcore\ops/function.rs:223
  27: <F as alloc::boxed::FnBox<A>>::call_box
             at libtest/lib.rs:1451
             at C:\projects\rust\src\libcore\ops/function.rs:223
             at C:\projects\rust\src\liballoc/boxed.rs:642
  28: _rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:105


failures:
    dynamic_honey_badger::votes::tests::test_committed_votes
    dynamic_honey_badger::votes::tests::test_pending_votes

test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--lib'

If there's any other information I can provide to you, please let me know!

The text was updated successfully, but these errors were encountered:

ErichDonGubler · 2018-10-01T18:56:38Z

<threshold_crypto::SecretKey as core::ops::drop::Drop>::drop in the stacktrace seems to be the culprit -- the panic message states 'Failed to drop `SecretKey`: Failed to `munlock` 32 bytes starting at address: 0x1aa0620. Is this a hole in Windows support for threshold_crypto?

c0gent · 2018-10-01T19:10:32Z

Yes. Unfortunately we haven't been testing on Windows at all. You shouldn't have any problems on MinGW (though no promises). Linux is our only first-class platform for the moment. This is something that badly needs to be addressed.

ErichDonGubler · 2018-10-01T19:15:50Z

@c0gent: In my report above I've noted that the target triple is stable-x86_64-pc-windows-gnu -- msvc doesn't even compile because unistd.h is missing. So, this problem is happening specifically in the MinGW environment.

c0gent · 2018-10-01T19:43:57Z

My mistake, I was on my phone and I missed that.

vkomenda · 2018-10-01T19:54:14Z

@ErichDonGubler, please retry with

MLOCK_SECRETS=false cargo test --release

as described in the README. Let know how it went.

ErichDonGubler · 2018-10-01T21:16:14Z

@vkomenda: MLOCK_SECRETS=false resolves my immediate issue (running tests), independent of the --release flag. I have not found a way to work around the issue as originally reported, but thank you for the tip!

afck · 2018-10-02T09:59:47Z

Does it work with test-threads=1? I wonder whether too many pages get locked at the same time.

Maybe we should consider making MLOCK_SECRETS=false the default until we have addressed this issue in general (see discussion in poanetwork/threshold_crypto#34). I wonder whether it's worse on Windows (e.g. lower limit for locked pages, or a tendency to lock more pages because of different allocation).

ErichDonGubler · 2018-10-03T01:37:30Z

@afck: This is the error output I'm getting trying to use cargo test --release --test-threads 1:

$ cargo test --release -- --test-threads 1
   Compiling hbbft v0.1.0 (file:///C:/msys64/home/K0RYU/workspace/interviews/poa-networks/hbbft)
    Finished release [optimized] target(s) in 1m 42s
     Running target\release\deps\hbbft-efd7f2c9ff217d71.exe

running 3 tests
test broadcast::merkle::tests::test_merkle ... ok
test dynamic_honey_badger::votes::tests::test_committed_votes ... FAILED
test dynamic_honey_badger::votes::tests::test_pending_votes ... FAILED

failures:

---- dynamic_honey_badger::votes::tests::test_committed_votes stdout ----
thread 'dynamic_honey_badger::votes::tests::test_committed_votes' panicked at 'Failed to drop `SecretKey`: Failed to `munlock` 32 bytes starting at address: 0xbdd70', C:\Users\K0RYU\.cargo\git\checkouts\threshold_crypto-3605c1a1492f376b\e1c9158\src\lib.rs:316:13
note: Run with `RUST_BACKTRACE=1` for a backtrace.

---- dynamic_honey_badger::votes::tests::test_pending_votes stdout ----
thread 'dynamic_honey_badger::votes::tests::test_pending_votes' panicked at 'Failed to drop `SecretKey`: Failed to `munlock` 32 bytes starting at address: 0xbda10', C:\Users\K0RYU\.cargo\git\checkouts\threshold_crypto-3605c1a1492f376b\e1c9158\src\lib.rs:316:13


failures:
    dynamic_honey_badger::votes::tests::test_committed_votes
    dynamic_honey_badger::votes::tests::test_pending_votes

test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--lib'

afck · 2018-10-23T10:01:17Z

All mlock/munlock code has been removed from threshold_crypto for now. We have poanetwork/threshold_crypto#24 and related issues to add it back in in the future, but this time with a dedicated allocator to avoid these kinds of problems.

(Please reopen if you can still reproduce the panic with current master.)

ErichDonGubler mentioned this issue Oct 1, 2018

Make Subset optimization optional #250

Merged

1 task

afck closed this as completed Oct 23, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Double panic when trying to run `cargo test` on Win10 #249

Double panic when trying to run `cargo test` on Win10 #249

ErichDonGubler commented Oct 1, 2018

ErichDonGubler commented Oct 1, 2018 •

edited

Loading

c0gent commented Oct 1, 2018 •

edited

Loading

ErichDonGubler commented Oct 1, 2018

c0gent commented Oct 1, 2018 •

edited

Loading

vkomenda commented Oct 1, 2018

ErichDonGubler commented Oct 1, 2018

afck commented Oct 2, 2018

ErichDonGubler commented Oct 3, 2018

afck commented Oct 23, 2018

Double panic when trying to run cargo test on Win10 #249

Double panic when trying to run cargo test on Win10 #249

Comments

ErichDonGubler commented Oct 1, 2018

ErichDonGubler commented Oct 1, 2018 • edited Loading

c0gent commented Oct 1, 2018 • edited Loading

ErichDonGubler commented Oct 1, 2018

c0gent commented Oct 1, 2018 • edited Loading

vkomenda commented Oct 1, 2018

ErichDonGubler commented Oct 1, 2018

afck commented Oct 2, 2018

ErichDonGubler commented Oct 3, 2018

afck commented Oct 23, 2018

Double panic when trying to run `cargo test` on Win10 #249

Double panic when trying to run `cargo test` on Win10 #249

ErichDonGubler commented Oct 1, 2018 •

edited

Loading

c0gent commented Oct 1, 2018 •

edited

Loading

c0gent commented Oct 1, 2018 •

edited

Loading