EntityByComponents causes a crash on exit

[iche033]

Environment

• OS Version: Ubuntu Bionic
• Source or binary build?
  source build, ign-gazebo6 branch

Description

When calling EntityByComponents with certain combination of components as filters inside a gui plugin, ign-gazebo will crash on exit.

The EntityByComponents call creates a new View that's stored in the EntityComponentManager class (this->dataPtr->views). The cause of the crash seems to be related to deletion of the views when the unique pointer goes out of scope.

The following will cause a crash in the ECM / BaseView default destructor

  _ecm.EntityByComponents(
        components::Link(), components::Name("some_name"));
  _ecm.EntityByComponents(
        components::Visual(), components::Name("some_name"));
  _ecm.EntityByComponents(
        components::World(), components::Name("some_name"));

You can replace components::Name above with with any other component and it'll still crash.

Interestingly, the calls below do not cause a crash on exit:

  // components::Model() does not cause a crash!
  _ecm.EntityByComponents(
        components::Model(), components::Name("some_name"));

  // query entities using just one single component at a time
  _ecm.EntityByComponents(components::Link());
  _ecm.EntityByComponents(components::Visual());
  _ecm.EntityByComponents(components::World());
  _ecm.EntityByComponents(components::Name("some_name"));

Steps to reproduce

1. Put the above code in Entity Tree's Update function
2. Build and launch ign-gazebo with Entity Tree gui plugin loaded
3. Click the x button on top right to exit, and confirm exit -> crash

[adlarkin]

With the new ign-gazebo6.3.0 release, I'm starting to see crashes on exit almost every time 😥 here's what I see when I exit after running ign gazebo shapes.sdf:

Stack trace (most recent call last):
#31   Object "/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5", at 0x7eff19d0a41d, in ruby_run_node
#30   Object "/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5", at 0x7eff19d07f3c, in ruby_exec_node
#29   Object "/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5", at 0x7eff19d060b3, in 
#28   Object "/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5", at 0x7eff19e2bad3, in 
#27   Object "/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5", at 0x7eff19e25cf4, in 
#26   Object "/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5", at 0x7eff19e2f862, in 
#25   Object "/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5", at 0x7eff19e21338, in 
#24   Object "/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5", at 0x7eff19d8e99d, in 
#23   Object "/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5", at 0x7eff19d08b10, in rb_protect
#22   Object "/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5", at 0x7eff19e32b9b, in rb_yield
#21   Object "/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5", at 0x7eff19e2bad3, in 
#20   Object "/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5", at 0x7eff19e25cf4, in 
#19   Object "/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5", at 0x7eff19e2f862, in 
#18   Object "/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5", at 0x7eff19e21338, in 
#17   Object "/usr/lib/x86_64-linux-gnu/ruby/2.5.0/fiddle.so", at 0x7eff17dce68c, in 
#16   Object "/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5", at 0x7eff19dfac1a, in 
#15   Object "/usr/lib/x86_64-linux-gnu/ruby/2.5.0/fiddle.so", at 0x7eff17dce8f7, in 
#14   Object "/usr/lib/x86_64-linux-gnu/libffi.so.6", at 0x7eff17bc771e, in ffi_call
#13   Object "/usr/lib/x86_64-linux-gnu/libffi.so.6", at 0x7eff17bc7dad, in ffi_call_unix64
#12   Object "/usr/lib/x86_64-linux-gnu/libignition-gazebo6-ign.so.6.3.0", at 0x7eff167b514c, in runGui
#11   Object "/usr/lib/x86_64-linux-gnu/libignition-gazebo6-gui.so.6", at 0x7eff163c469b, in ignition::gazebo::v6::gui::runGui(int&, char**, char const*, char const*)
#10   Object "/usr/lib/x86_64-linux-gnu/libignition-gui6.so.6", at 0x7eff13bc2e08, in ignition::gui::Application::~Application()
#9    Object "/usr/lib/x86_64-linux-gnu/libignition-gui6.so.6", at 0x7eff13bc2acc, in ignition::gui::Application::~Application()
#8    Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5", at 0x7eff134a0298, in QApplication::~QApplication()
#7    Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7eff140a99f5, in QCoreApplication::~QCoreApplication()
#6    Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7eff140dd93a, in QObject::~QObject()
#5    Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7eff140d410a, in QObjectPrivate::deleteChildren()
#4    Object "/usr/lib/x86_64-linux-gnu/libignition-gazebo6-gui.so.6", at 0x7eff163cc218, in ignition::gazebo::v6::GuiRunner::~GuiRunner()
#3    Object "/usr/lib/x86_64-linux-gnu/libignition-gazebo6-gui.so.6", at 0x7eff163cc1fd, in ignition::gazebo::v6::GuiRunner::~GuiRunner()
#2    Object "/usr/lib/x86_64-linux-gnu/libignition-gazebo6-gui.so.6", at 0x7eff1640d43d, in void ignition::utils::detail::DefaultDelete<ignition::gazebo::v6::GuiRunner::Implementation>(ignition::gazebo::v6::GuiRunner::Implementation*)
#1    Object "/usr/lib/x86_64-linux-gnu/libignition-gazebo6.so.6", at 0x7eff15f660b9, in ignition::gazebo::v6::EntityComponentManager::~EntityComponentManager()
#0    Object "/usr/lib/x86_64-linux-gnu/libignition-gazebo6.so.6", at 0x7eff15f73432, in std::_Hashtable<std::vector<unsigned long, std::allocator<unsigned long> >, std::pair<std::vector<unsigned long, std::allocator<unsigned long> > const, std::pair<std::unique_ptr<ignition::gazebo::v6::detail::BaseView, std::default_delete<ignition::gazebo::v6::detail::BaseView> >, std::unique_ptr<std::mutex, std::default_delete<std::mutex> > > >, std::allocator<std::pair<std::vector<unsigned long, std::allocator<unsigned long> > const, std::pair<std::unique_ptr<ignition::gazebo::v6::detail::BaseView, std::default_delete<ignition::gazebo::v6::detail::BaseView> >, std::unique_ptr<std::mutex, std::default_delete<std::mutex> > > > >, std::__detail::_Select1st, std::equal_to<std::vector<unsigned long, std::allocator<unsigned long> > >, ignition::gazebo::v6::detail::ComponentTypeHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::clear()
Segmentation fault (Address not mapped to object [0x7efed55001d0])

[adlarkin]

I just noticed another crash that I believe is related to this when I run ign gazebo and then exit simulation. The stack trace this time around seems to give some more information that may be useful:

#12   Object "/usr/lib/x86_64-linux-gnu/libignition-gazebo6-ign.so.6.3.0", at 0x7fe8fa06114c, in runGui
#11   Object "/usr/lib/x86_64-linux-gnu/libignition-gazebo6-gui.so.6", at 0x7fe8f9c7069b, in ignition::gazebo::v6::gui::runGui(int&, char**, char const*, char const*)
#10   Object "/usr/lib/x86_64-linux-gnu/libignition-gui6.so.6", at 0x7fe8f7463e08, in ignition::gui::Application::~Application()
#9    Object "/usr/lib/x86_64-linux-gnu/libignition-gui6.so.6", at 0x7fe8f7463acc, in ignition::gui::Application::~Application()
#8    Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5", at 0x7fe8f6d41298, in QApplication::~QApplication()
#7    Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7fe8f794a9f5, in QCoreApplication::~QCoreApplication()
#6    Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7fe8f797e93a, in QObject::~QObject()
#5    Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7fe8f797510a, in QObjectPrivate::deleteChildren()
#4    Object "/usr/lib/x86_64-linux-gnu/libignition-gazebo6-gui.so.6", at 0x7fe8f9c78218, in ignition::gazebo::v6::GuiRunner::~GuiRunner()
#3    Object "/usr/lib/x86_64-linux-gnu/libignition-gazebo6-gui.so.6", at 0x7fe8f9c781fd, in ignition::gazebo::v6::GuiRunner::~GuiRunner()
#2    Object "/usr/lib/x86_64-linux-gnu/libignition-gazebo6-gui.so.6", at 0x7fe8f9cb943d, in void ignition::utils::detail::DefaultDelete<ignition::gazebo::v6::GuiRunner::Implementation>(ignition::gazebo::v6::GuiRunner::Implementation*)
#1    Object "/usr/lib/x86_64-linux-gnu/libignition-gazebo6.so.6", at 0x7fe8f98120b9, in ignition::gazebo::v6::EntityComponentManager::~EntityComponentManager()
#0    Object "/usr/lib/x86_64-linux-gnu/libignition-gazebo6.so.6", at 0x7fe8f981f432, in std::_Hashtable<std::vector<unsigned long, std::allocator<unsigned long> >, std::pair<std::vector<unsigned long, std::allocator<unsigned long> > const, std::pair<std::unique_ptr<ignition::gazebo::v6::detail::BaseView, std::default_delete<ignition::gazebo::v6::detail::BaseView> >, std::unique_ptr<std::mutex, std::default_delete<std::mutex> > > >, std::allocator<std::pair<std::vector<unsigned long, std::allocator<unsigned long> > const, std::pair<std::unique_ptr<ignition::gazebo::v6::detail::BaseView, std::default_delete<ignition::gazebo::v6::detail::BaseView> >, std::unique_ptr<std::mutex, std::default_delete<std::mutex> > > > >, std::__detail::_Select1st, std::equal_to<std::vector<unsigned long, std::allocator<unsigned long> > >, ignition::gazebo::v6::detail::ComponentTypeHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::clear()
Segmentation fault (Address not mapped to object [0x7fe8bcd931d0])

[azeey]

I spent sometime looking into this and I've found one issue (I don't know if there are multiple) that causes the crash. I have MWE in https://github.com/azeey/sandbox/tree/master/use_ecm_in_shared_libraries, but to summarize, the views created in the gui plugins have their virtual destructors stored in the plugins' shared library. A segfault occurs if the EntityComponentManger is destructed after these plugins have been unloaded, which is the case currently when exiting ign-gazebo.

[chapulina]

Wow great detective work, @azeey ! So we may see the same issue on the server too if we were actually unloading plugins there (#113).

[azeey]

Wow great detective work, @azeey ! So we may see the same issue on the server too if we were actually unloading plugins there (#113).

Yes, I believe so. I don't know what the solution is other than keeping the plugins loaded. I think it's okay to run the destructors of the plugins, but not okay to completely unload them from memory, unless we come up with a different solution.

[adlarkin]

Thanks for looking into this, @azeey. I wasn't aware of the effects of unloading shared libraries early, so I learned something new! I took a look at the MWE you shared, and read the following:

When creating the views, objects of type detail::View<...> are created and stored in a container in EntityComponentManagerPrivate. These View objects have their virtual destructors in the shared library.

I'm having a hard time understanding why the virtual destructors are in the shared library instead of the main application, since the ECM (which holds a pointer to EntityComponentManagerPrivate) is defined in main: https://github.com/azeey/sandbox/blob/aacddca7fec33ea98860bd8834dcb7638afb0329/use_ecm_in_shared_libraries/main.cc#L47

Would you mind explaining why the virtual destructor is placed in the shared library so that I can have a better understanding?

[azeey]

I'm having a hard time understanding why the virtual destructors are in the shared library instead of the main application, since the ECM (which holds a pointer to EntityComponentManagerPrivate) is defined in main:

It happens because View is a template and it gets instantiated in the shared library when EntityFromComponents(which is also a template) is run. All templates . One way to check is to run nm -C path/to/gui/plugin.so | grep vtable.

[adlarkin]

Interesting, so even though the views are stored in the EntityComponentManager via EntityComponentManagerPrivate, the destructors live in the shared library?

Okay, well if this is the issue for the crash, should we not completely unload GUI plugins from memory? Or would this have any potential drawbacks?

[azeey]

Interesting, so even though the views are stored in the EntityComponentManager via EntityComponentManagerPrivate, the destructors live in the shared library?

Yeah, the object data stored inside EntityComponentManager's containers will be on the heap, but the destructor, along with all other member functions will be located in different part of memory loaded from the .text section of the binary/shared library. The question is which shared library holds the destructor. With non-template classes, the destructor would likely be in the shared library that built from the translation unit that defines the destructor. With template classes though, each translation unit that instantiates the template would get a copy of the destructor, hence each shared library. In the end, how and where an object of the class is stored doesn't affect where the function itself is stored.

Okay, well if this is the issue for the crash, should we not completely unload GUI plugins from memory? Or would this have any potential drawbacks?

Since this will affect all plugins, not just GUI plugins, it would be great if we could come up with a solution that doesn't require us to leave plugins loaded in memory. One drawback of leaving them loaded is memory consumption. Another is how they might make it difficult to implement World Reset.

For the short term, leaving them loaded might be our solution. If we can find a way to make Views not be template classes, I think that would be a better solution.

[azeey]

@adlarkin here's a potential solution #1317. Let me know what you think.