Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ex_MQ_serialization1 should be stricter #1286

Closed
dennisklein opened this issue Nov 19, 2022 · 7 comments
Closed

ex_MQ_serialization1 should be stricter #1286

dennisklein opened this issue Nov 19, 2022 · 7 comments
Assignees
@rbx
Labels
bug
Milestone
v18.8

Comments

@dennisklein
Copy link
Member

Logs: https://cdash.gsi.de/testDetails.php?test=12812982&build=356755

Error in <TExMap::Remove>: key 1949127801 not found at 286
Warning in <TMessage::CheckObject>: reference to object of unavailable class TObject, offset=1949127801 pointer will be 0

 *** Break *** segmentation violation

Another line that looks suspicious is:

[#0] WARNING:InputArguments -- The parameter 'fSigmaT' with range [-1e+30, 1e+30] of the RooGaussian 'fGaussT' exceeds the safe range of (0, inf). Advise to limit its range.

 ===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================

Thread 6 (Thread 0x7f517f478640 (LWP 35583) "ex-serializatio"):
#0  0x00007f51949259d9 in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1  0x00007f51949287c7 in pthread_cond_clockwait
GLIBC_2.30 () from /lib64/libc.so.6
#2  0x00007f51977d74ea in std::__condvar::wait_until (__abs_time=..., __clock=1, __m=..., this=<optimized out>) at /usr/include/c++/12/bits/std_mutex.h:123
#3  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__lock=..., __atime=..., this=<optimized out>) at /usr/include/c++/12/condition_variable:200
#4  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=<optimized out>) at /usr/include/c++/12/condition_variable:110
#5  std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long int, std::ratio<1, 1000000000> >, fair::mq::StateMachine::WaitForPendingStateFor(int) const::<lambda()> > (__p=..., __atime=..., __lock=..., this=<optimized out>) at /usr/include/c++/12/condition_variable:151
#6  std::condition_variable::wait_for<long int, std::ratio<1, 1000>, fair::mq::StateMachine::WaitForPendingStateFor(int) const::<lambda()> > (__rtime=..., __p=..., __lock=..., this=<optimized out>) at /usr/include/c++/12/condition_variable:174
#7  fair::mq::StateMachine::WaitForPendingStateFor (this=this
entry=0x2877dd0, durationInMs=durationInMs
entry=1000) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/StateMachine.cxx:362
#8  0x00007f5197a7b9e1 in fair::mq::Device::WaitFor<long, std::ratio<1l, 1l> > (duration=..., this=<optimized out>) at /usr/include/c++/12/bits/chrono.h:505
#9  fair::mq::Device::LogSocketRates (this=<optimized out>) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/Device.cxx:727
#10 0x00007f5194c77b03 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#11 0x00007f5194928ded in start_thread () from /lib64/libc.so.6
#12 0x00007f51949ad524 in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f517fc79640 (LWP 35581) "ZMQbg/IO/0"):
#0  0x00007f51949ad94e in epoll_wait () from /lib64/libc.so.6
#1  0x00007f5194821aa8 in zmq::epoll_t::loop (this=0x287f220) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/zeromq/src/epoll.cpp:184
#2  0x00007f5194853349 in thread_routine (arg_=0x287f278) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/zeromq/src/thread.cpp:257
#3  0x00007f5194928ded in start_thread () from /lib64/libc.so.6
#4  0x00007f51949ad524 in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f518047a640 (LWP 35580) "ZMQbg/Reaper"):
#0  0x00007f51949ad94e in epoll_wait () from /lib64/libc.so.6
#1  0x00007f5194821aa8 in zmq::epoll_t::loop (this=0x28ef450) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/zeromq/src/epoll.cpp:184
#2  0x00007f5194853349 in thread_routine (arg_=0x28ef4a8) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/zeromq/src/thread.cpp:257
#3  0x00007f5194928ded in start_thread () from /lib64/libc.so.6
#4  0x00007f51949ad524 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f5180c7b640 (LWP 35579) "ex-serializatio"):
#0  0x00007f5194974635 in clock_nanosleep
GLIBC_2.2.5 () from /lib64/libc.so.6
#1  0x00007f5194978ea7 in nanosleep () from /lib64/libc.so.6
#2  0x00007f5197b805b4 in std::this_thread::sleep_for<long, std::ratio<1l, 1000l> > (__rtime=...) at /usr/include/c++/12/bits/this_thread_sleep.h:80
#3  fair::mq::plugins::Control::SignalHandler (this=0x2901380) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/plugins/control/Control.cxx:438
#4  0x00007f5194c77b03 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#5  0x00007f5194928ded in start_thread () from /lib64/libc.so.6
#6  0x00007f51949ad524 in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f518147c640 (LWP 35578) "ex-serializatio"):
#0  0x00007f51949259d9 in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1  0x00007f51949281d0 in pthread_cond_wait

GLIBC_2.3.2 () from /lib64/libc.so.6
#2  0x00007f5194c71060 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib64/libstdc++.so.6
#3  0x00007f5197b80373 in std::condition_variable::wait<fair::mq::StateQueue::WaitForNextOrCustom<fair::mq::plugins::Control::StaticMode()::<lambda()> >(fair::mq::plugins::Control::StaticMode()::<lambda()>&&)::<lambda()> > (__p=..., __lock=..., this=0x29014b8) at /usr/include/c++/12/condition_variable:102
#4  fair::mq::StateQueue::WaitForNextOrCustom<fair::mq::plugins::Control::StaticMode()::<lambda()> > (customPredicate=..., this=0x2901440) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/StateQueue.h:50
#5  fair::mq::plugins::Control::StaticMode (this=0x2901380) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/plugins/control/Control.cxx:405
#6  0x00007f5194c77b03 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#7  0x00007f5194928ded in start_thread () from /lib64/libc.so.6
#8  0x00007f51949ad524 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f5191b42f00 (LWP 35360) "ex-serializatio"):
#0  0x00007f5194978c5f in wait4 () from /lib64/libc.so.6
#1  0x00007f51948e79db in do_system () from /lib64/libc.so.6
#2  0x00007f519629765c in TUnixSystem::Exec (shellcmd=<optimized out>, this=0x1130200) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/root/core/unix/src/TUnixSystem.cxx:2108
#3  TUnixSystem::StackTrace (this=0x1130200) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/root/core/unix/src/TUnixSystem.cxx:2399
#4  0x00007f5196297004 in TUnixSystem::DispatchSignals (this=0x1130200, sig=kSigSegmentationViolation) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/root/core/unix/src/TUnixSystem.cxx:3619
#5  <signal handler called>
#6  0x00007f519620579d in TObjArray::GetAbsLast (this=this
entry=0x0) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/root/core/cont/src/TObjArray.cxx:545
#7  0x0000000000424984 in TObjArray::GetEntriesFast (this=0x0) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/root/TObjArray.h:65
#8  Ex1Processor::FindHits (this=0x2877d10, digis=...) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/FairRoot_v18.6_patches/examples/MQ/serialization/1-simple/Ex1Processor.h:62
#9  Ex1Processor::Run (this=0x2877d10) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/FairRoot_v18.6_patches/examples/MQ/serialization/1-simple/Ex1Processor.h:42
#10 0x00007f5197a78d89 in fair::mq::Device::RunWrapper (this=0x2877d10) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/Device.cxx:486
#11 0x00007f51977dbc5d in std::function<void (fair::mq::State)>::operator()(fair::mq::State) const (__args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/12/bits/std_function.h:591
#12 boost::detail::function::void_function_obj_invoker1<std::function<void (fair::mq::State)>, void, fair::mq::State>::invoke(boost::detail::function::function_buffer&, fair::mq::State) (function_obj_ptr=..., a0=<optimized out>) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/function/function_template.hpp:158
#13 0x00007f51977e5479 in boost::function1<void, fair::mq::State>::operator() (a0=<optimized out>, this=<optimized out>) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/function/function_template.hpp:763
#14 boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::m_invoke<boost::function<void (fair::mq::State)>, 0u, fair::mq::State&>(boost::function<void (fair::mq::State)>&, boost::signals2::detail::unsigned_meta_array<0u>, std::tuple<fair::mq::State&> const&, boost::enable_if<boost::is_void<boost::function<void (fair::mq::State)>::result_type>, void>::type*) const (args=std::tuple containing = {...}, func=..., this=<optimized out>) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/signals2/detail/variadic_slot_invoker.hpp:105
#15 boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::operator()<boost::function<void (fair::mq::State)>, fair::mq::State&, 1ul>(boost::function<void (fair::mq::State)>&, std::tuple<fair::mq::State&> const&, mpl_::size_t<1ul>) const (args=std::tuple containing = {...}, func=..., this=<optimized out>) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/signals2/detail/variadic_slot_invoker.hpp:90
#16 boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>::operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >(boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > const&) const (connectionBody=..., this=0x7ffc53023e00) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/signals2/detail/variadic_slot_invoker.hpp:133
#17 boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >::dereference() const (this=0x7ffc53023c30) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/signals2/detail/slot_call_iterator.hpp:110
#18 boost::iterators::iterator_core_access::dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > const&) (f=...) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/iterator/iterator_facade.hpp:550
#19 boost::iterators::detail::iterator_facade_base<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >, boost::signals2::detail::void_type, boost::iterators::single_pass_traversal_tag, boost::signals2::detail::void_type const&, long, false, false>::operator*() const (this=0x7ffc53023c30) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/iterator/iterator_facade.hpp:656
#20 boost::signals2::optional_last_value<void>::operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >) const (this=<optimized out>, last=..., first=...) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/signals2/optional_last_value.hpp:57
#21 boost::signals2::detail::combiner_invoker<void>::operator()<boost::signals2::optional_last_value<void>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >(boost::signals2::optional_last_value<void>&, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >) const (last=..., first=..., combiner=..., this=<optimized out>) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/signals2/detail/result_type_wrapper.hpp:64
#22 boost::signals2::detail::signal_impl<void (fair::mq::State), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (fair::mq::State)>, boost::function<void (boost::signals2::connection const&, fair::mq::State)>, boost::signals2::mutex>::operator()(fair::mq::State) (this=0x2879730, args#0=<optimized out>, args#0
entry=fair::mq::State::Running) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/signals2/detail/signal_template.hpp:243
#23 0x00007f51977e5ce7 in boost::signals2::signal<void (fair::mq::State), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (fair::mq::State)>, boost::function<void (boost::signals2::connection const&, fair::mq::State)>, boost::signals2::mutex>::operator()(fair::mq::State) const (args#0=fair::mq::State::Running, this=0x2879548) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/smart_ptr/shared_ptr.hpp:775
#24 fair::mq::fsm::Machine_::CallStateHandler (state=fair::mq::State::Running, this=0x28794c0) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/StateMachine.cxx:156
#25 fair::mq::fsm::Machine_::CallStateHandler (state=fair::mq::State::Running, this=0x28794c0) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/StateMachine.cxx:153
#26 fair::mq::fsm::Machine_::ProcessWork (this=0x28794c0) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/StateMachine.cxx:202
#27 0x00007f51977d9f0d in fair::mq::StateMachine::ProcessWork (this=<optimized out>) at /usr/include/c++/12/bits/shared_ptr_base.h:1665
#28 0x00007f5197a876ed in fair::mq::Device::RunStateMachine (this=<optimized out>) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/Device.h:418
#29 fair::mq::DeviceRunner::Run (this=this
entry=0x7ffc53024e20) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/DeviceRunner.cxx:182
#30 0x00000000004180fa in main (argc=<optimized out>, argv=<optimized out>) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/fairmq/runDevice.h:48
===========================================================


The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum https://root.cern/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#6  0x00007f519620579d in TObjArray::GetAbsLast (this=this
entry=0x0) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/root/core/cont/src/TObjArray.cxx:545
#7  0x0000000000424984 in TObjArray::GetEntriesFast (this=0x0) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/root/TObjArray.h:65
#8  Ex1Processor::FindHits (this=0x2877d10, digis=...) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/FairRoot_v18.6_patches/examples/MQ/serialization/1-simple/Ex1Processor.h:62
#9  Ex1Processor::Run (this=0x2877d10) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/FairRoot_v18.6_patches/examples/MQ/serialization/1-simple/Ex1Processor.h:42
#10 0x00007f5197a78d89 in fair::mq::Device::RunWrapper (this=0x2877d10) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/Device.cxx:486
#11 0x00007f51977dbc5d in std::function<void (fair::mq::State)>::operator()(fair::mq::State) const (__args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/12/bits/std_function.h:591
#12 boost::detail::function::void_function_obj_invoker1<std::function<void (fair::mq::State)>, void, fair::mq::State>::invoke(boost::detail::function::function_buffer&, fair::mq::State) (function_obj_ptr=..., a0=<optimized out>) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/function/function_template.hpp:158
#13 0x00007f51977e5479 in boost::function1<void, fair::mq::State>::operator() (a0=<optimized out>, this=<optimized out>) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/function/function_template.hpp:763
#14 boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::m_invoke<boost::function<void (fair::mq::State)>, 0u, fair::mq::State&>(boost::function<void (fair::mq::State)>&, boost::signals2::detail::unsigned_meta_array<0u>, std::tuple<fair::mq::State&> const&, boost::enable_if<boost::is_void<boost::function<void (fair::mq::State)>::result_type>, void>::type*) const (args=std::tuple containing = {...}, func=..., this=<optimized out>) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/signals2/detail/variadic_slot_invoker.hpp:105
#15 boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::operator()<boost::function<void (fair::mq::State)>, fair::mq::State&, 1ul>(boost::function<void (fair::mq::State)>&, std::tuple<fair::mq::State&> const&, mpl_::size_t<1ul>) const (args=std::tuple containing = {...}, func=..., this=<optimized out>) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/signals2/detail/variadic_slot_invoker.hpp:90
#16 boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>::operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >(boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > const&) const (connectionBody=..., this=0x7ffc53023e00) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/signals2/detail/variadic_slot_invoker.hpp:133
#17 boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >::dereference() const (this=0x7ffc53023c30) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/signals2/detail/slot_call_iterator.hpp:110
#18 boost::iterators::iterator_core_access::dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > const&) (f=...) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/iterator/iterator_facade.hpp:550
#19 boost::iterators::detail::iterator_facade_base<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >, boost::signals2::detail::void_type, boost::iterators::single_pass_traversal_tag, boost::signals2::detail::void_type const&, long, false, false>::operator*() const (this=0x7ffc53023c30) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/iterator/iterator_facade.hpp:656
#20 boost::signals2::optional_last_value<void>::operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >) const (this=<optimized out>, last=..., first=...) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/signals2/optional_last_value.hpp:57
#21 boost::signals2::detail::combiner_invoker<void>::operator()<boost::signals2::optional_last_value<void>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >(boost::signals2::optional_last_value<void>&, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >) const (last=..., first=..., combiner=..., this=<optimized out>) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/signals2/detail/result_type_wrapper.hpp:64
#22 boost::signals2::detail::signal_impl<void (fair::mq::State), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (fair::mq::State)>, boost::function<void (boost::signals2::connection const&, fair::mq::State)>, boost::signals2::mutex>::operator()(fair::mq::State) (this=0x2879730, args#0=<optimized out>, args#0
entry=fair::mq::State::Running) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/signals2/detail/signal_template.hpp:243
#23 0x00007f51977e5ce7 in boost::signals2::signal<void (fair::mq::State), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (fair::mq::State)>, boost::function<void (boost::signals2::connection const&, fair::mq::State)>, boost::signals2::mutex>::operator()(fair::mq::State) const (args#0=fair::mq::State::Running, this=0x2879548) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/boost/smart_ptr/shared_ptr.hpp:775
#24 fair::mq::fsm::Machine_::CallStateHandler (state=fair::mq::State::Running, this=0x28794c0) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/StateMachine.cxx:156
#25 fair::mq::fsm::Machine_::CallStateHandler (state=fair::mq::State::Running, this=0x28794c0) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/StateMachine.cxx:153
#26 fair::mq::fsm::Machine_::ProcessWork (this=0x28794c0) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/StateMachine.cxx:202
#27 0x00007f51977d9f0d in fair::mq::StateMachine::ProcessWork (this=<optimized out>) at /usr/include/c++/12/bits/shared_ptr_base.h:1665
#28 0x00007f5197a876ed in fair::mq::Device::RunStateMachine (this=<optimized out>) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/Device.h:418
#29 fair::mq::DeviceRunner::Run (this=this
entry=0x7ffc53024e20) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/Source/fairmq/fairmq/DeviceRunner.cxx:182
#30 0x00000000004180fa in main (argc=<optimized out>, argv=<optimized out>) at /tmp/jenkins-FairRootGroup-FairSoft-PR-486-2.zxk/build/install/include/fairmq/runDevice.h:48
===========================================================
@dennisklein dennisklein added this to the v18.8 milestone Nov 19, 2022
ChristianTackeGSI added a commit to ChristianTackeGSI/FairRoot that referenced this issue Dec 24, 2022
@ChristianTackeGSI
fix(Ex1Processor): Handle nullptr, cleanup
9f984e5 
fix part of FairRootGroup#1286: The result of ROOT's deserializer should
be checked.
@ChristianTackeGSI ChristianTackeGSI mentioned this issue Dec 24, 2022
Merged
ChristianTackeGSI added a commit to ChristianTackeGSI/FairRoot that referenced this issue Dec 24, 2022
@ChristianTackeGSI
fix(Ex1Processor): Handle nullptr, cleanup
aab45c4 
fix part of FairRootGroup#1286: The result of ROOT's deserializer should
be checked.
karabowi pushed a commit that referenced this issue Jan 18, 2023
@ChristianTackeGSI @karabowi
fix(Ex1Processor): Handle nullptr, cleanup
edb7657 
fix part of #1286: The result of ROOT's deserializer should
be checked.
ChristianTackeGSI added a commit to ChristianTackeGSI/FairRoot that referenced this issue Jan 20, 2023
@ChristianTackeGSI
fix(Ex1Processor): Handle nullptr, cleanup
c5d0d3d 
fix part of FairRootGroup#1286: The result of ROOT's deserializer should
be checked.

(cherry picked from commit edb7657)
@ChristianTackeGSI
Copy link
Member

Can we close this, once #1325 is merged?

karabowi pushed a commit that referenced this issue Jan 23, 2023
@ChristianTackeGSI @karabowi
fix(Ex1Processor): Handle nullptr, cleanup
88d26d2 
fix part of #1286: The result of ROOT's deserializer should
be checked.

(cherry picked from commit edb7657)
@dennisklein
Copy link
Member Author

Hm, as you wish. The proposed fix, however, does not really understand why there are non-deserializable messages. So, in this aspect the example is still broken.

One could make the argument that it should continue to fail (although preferably not via a nullptr deref ;) which is fixed now)

@ChristianTackeGSI
Copy link
Member

ChristianTackeGSI commented Jan 26, 2023 •
edited
Loading

Good point.

Well, in theory the thing should check for "all messages were received that were expected". So in that regard the thing is probably also broken.

Retitled to follow the new target of this issue. Hope that's okay for you for the time being?

@ChristianTackeGSI ChristianTackeGSI changed the title segfault in ex_MQ_serialization1 (deref of nullptr) ex_MQ_serialization1 should be stricter Jan 26, 2023
@rbx rbx self-assigned this Jan 31, 2023
@rbx
Copy link
Member

rbx commented Feb 3, 2023 •
edited
Loading

Possibly #1295 is related. For now I'll assume it is the same underlying issue.
Also I cannot reproduce it myself, so just thinking what it could be for now.

I don't see any issue in the example devices.
There is no race in the lifetime of the devices.
Also, each event is processed individually, so it's not a matter of not seeing all the messages (there is also no evidence that this ever occurs).

Well, in theory the thing should check for "all messages were received that were expected". So in that regard the thing is probably also broken.

What "thing"? The test checks for the "Received 100 messages!" string, which is printed by the sink upon success. And each message is processed independently by the processor, so the devices don't care if there are not all messages.

So, I think the issue is either in the data generation, or in the ROOT serializer. I cannot judge what the data generation is doing - it was written by Nicolas 9 years ago and from a brief look I have no idea how it works.

To narrow it down I propose to run the same algorithm which is done by the processor, also in the Sampler. That way if Sampler also crashes we will know that data generation is at fault. And if only the Processor keeps crashing - then it has to be de-/serialization.
I think Serializer is less likely to be at fault, since it is also in use elsewhere, where this specific issue does not occur, although, of course, it could be just not showing the symptoms.

If data generation is at fault, we either debug it further or replace it with something simpler - since it is not the purpose of this example to do anything special on the generation side. if the generation code is valueable, it should be extracted to its own example.

@ChristianTackeGSI ChristianTackeGSI mentioned this issue Feb 3, 2023
Closed
@ChristianTackeGSI
Copy link
Member

What "thing"? The test checks for the "Received 100 messages!" string, which is printed by the sink upon success. And each message is processed independently by the processor, so the devices don't care if there are not all messages.

thing := the test.

I had assumed, that we don't have enough checks in place. My fault! Please excuse me.

That said, I am no longer sure, what the target of this issue currently is.

@rbx
Copy link
Member

rbx commented Feb 3, 2023

That said, I am no longer sure, what the target of this issue currently is.

I've seen the checks that you've added in the example, and they may surely help to avoid the crash.
But the interesting part is still why it is failing there at all.
It looks like the data is broken somehow. So it is either bad generation, or bad serialization or the file gets corrupted somehow - perhaps there is a race in the file I/O. One more thing I will look at is the RootOutFileManager - wanted to clean that up since a long time anyway - maybe there is a race there on writing, but I doubt it.

@rbx rbx mentioned this issue Feb 7, 2023
Merged
@karabowi
Copy link
Collaborator

Closing as the examples had been rewritten in #1341 .
Many of the MQ test failures are probably connected to running tests in parallel,
which is no good for hardcoded IP addresses.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rbx rbx

Labels
bug
Projects
None yet
Milestone
v18.8
Development

No branches or pull requests
4 participants
@dennisklein @rbx @karabowi @ChristianTackeGSI