Crash when Thread Model is set to Multi-Threaded

[ddxt]

Godot version

Godot Engine v4.0.alpha9.official.fc18891db

System information

5.15.0-33-generic #34-Ubuntu SMP Wed May 18 13:34:26 UTC 2022 x86_64 5700XT

Issue description

Projects run with Godot 4.0 alpha 9 instantly crash when Rendering>Driver>Threads>Thread Model is set to Multi-Threaded. The same project ran fine in alpha 8. Below is the output log.

handle_crash: Program crashed with signal 11
Engine version: Godot Engine v4.0.alpha9.official (fc18891dbceace03c3867ac26c15b79cfbc673df)
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f80a1d1d520] (??:0)
[2] /home/user/Downloads/Godot_v4.0-alpha9_linux.64() [0xdc77a7] (??:0)
[3] /home/user/Downloads/Godot_v4.0-alpha9_linux.64() [0x37459a0] (??:0)
[4] /home/user/Downloads/Godot_v4.0-alpha9_linux.64() [0xdaf1ed] (??:0)
[5] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f80a1d04d90] (??:0)
[6] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f80a1d04e40] (??:0)
[7] /home/user/Downloads/Godot_v4.0-alpha9_linux.64() [0xdc3aae] (??:0)
-- END OF BACKTRACE --

Steps to reproduce

Set Thread Model to Multi-Threaded and run a project. Affects existing project, as well as new projects with no other changes made.

Minimal reproduction project

minimal_crash.zip

[fire]

Can you post a minimal project? I think I have this patched in my fork, but I need to find the exact patch branch.

[ddxt]

minimal_crash.zip

[lostminds]

Tried the Rendering/Driver multi-threaded mode today in my project on 4.0.beta10 and it didn't crash on launch as described above, trying the minimal crash project about also doesn't crash for me.

However, in the case of my project it did crash quite quickly playing the game (possibly related to an explosion effect with light and 3d particles) with no errors or warnings. This does not happen using the Single-Safe thread mode. So there still seem to be some stability issues with the Multi-threaded driver mode, even if some have been fixed since this bug report.

[clayjohn]

I can't reproduce on latest master a51ca2b

@Calinou You marked this as confirmed earlier, can you still reproduce the crash on latest master?

[lostminds]

Still crashes for me when activating a 3d particle effect on macOS 13, Godot 4.0beta12 if I switch to the Multi-Threaded mode instead of Single-Thread safe. So I think there's at least some issue that still there.

[Calinou]

@Calinou You marked this as confirmed earlier, can you still reproduce the crash on latest master?

I can't reproduce this on 4.0.beta 4c5e09f (Linux, NVIDIA 525.60.11).

@lostminds Are you using CPUParticles or GPUParticles? #71359 reports the same issue when using CPUParticles. Also, what GPU model does your Mac have?

[lostminds]

@lostminds Are you using CPUParticles or GPUParticles? #71359 reports the same issue when using CPUParticles. Also, what GPU model does your Mac have?

They are CPUParticles3D spawning 50 sphere mesh particles. My GPU is a Radeon Pro 560 4 GB.
I've also tested now disabling only starting the particle emittor in the effect, and with that disabled it doesn't crah on Multi-Threaded, so I'm pretty confident emitting the particles is the problem.

[spitemonster]

Can confirm similar experience on macOS 12.6.3 with an RX580, at least with an existing project. Instantiating a CPUParticles3D in a scene causes a crash on launch. I can edit and preview particles no problem but they crash in the editor. I can also confirm this is not default behavior, as creating a new project, instantiating a CPUParticles3D node and then running the project does not cause a crash; I have not, however, investigated further.

No error message in Godot; macOS provides a crash report which I can provide, if useful. I can also confirm that switching to Single Threaded, both safe and unsafe, prevents crashes.

[vassembly]

can confirm on 4.0stable

[akien-mga]

I can't reproduce the issue myself on Linux with 4.0.stable with the provided reproduction project.

Can people reproducing the issue locally reproduce it with the project in the OP? If not, can you provide a MRP which triggers the crash reliably?

[m4nu3lf]

I can confirm too. I noticed Godot spamming these errors before crashing.

E 0:00:00:0465   particles_get_emitting: This function should never be used with threaded rendering, as it stalls the renderer.
  <C++ Error>    Condition "RenderingServerGlobals::threaded" is true. Returning: false
  <C++ Source>   servers/rendering/renderer_rd/storage_rd/particles_storage.cpp:265 @ particles_get_emitting()

[m4nu3lf]

It seems that using animated textures + multithreading causes a crash.

animated_texture_mt_crash.zip

[Foxysen]

Windows 10, GPU with no Vulkan support.

Crashes instantly on any scene I try, including those without particles or animated textures.
I run projects in Compatibility OpenGL render mode.

Of course my OpenGL drivers are a bit old right now with official support ending time ago, but I want to see if it crashes instantly for other people in OpenGL mode too.

It worked fine in Godot 3.5.

[darksylinc]

I was frustrated I can't repro the GPU errors, so this one out, randomly, which grabbed my attention (I'm on latest master 262d1ea).

The problem is that it ends up in an infinite recursion:

1  CommandQueueMT::_flush                                                                                                                                                                                          command_queue_mt.h           363  0x8a32654      
2  CommandQueueMT::flush_if_pending                                                                                                                                                                                command_queue_mt.h           404  0x8a328d9      
3  RenderingServerDefault::multimesh_set_buffer                                                                                                                                                                    rendering_server_default.h   341  0x8f42ac7      
4  CPUParticles3D::_update_render_thread                                                                                                                                                                           cpu_particles_3d.cpp         1263 0x7a86afd      
5  call_with_variant_args_helper<CPUParticles3D>(CPUParticles3D *, void (CPUParticles3D:: *)(), Variant const * *, Callable::CallError&, IndexSequence<>)                                                          binder_common.h              303  0x7aa3e09      
6  call_with_variant_args<CPUParticles3D>                                                                                                                                                                          binder_common.h              417  0x7aa3d79      
7  CallableCustomMethodPointer<CPUParticles3D>::call                                                                                                                                                               callable_method_pointer.h    104  0x7aa3c64      
8  Callable::callp                                                                                                                                                                                                 callable.cpp                 50   0x98e190a      
9  Object::emit_signalp                                                                                                                                                                                            object.cpp                   1072 0x9ca2f8e      
10 Object::emit_signal<>(StringName const&)                                                                                                                                                                        object.h                     891  0x5655b53      
11 RenderingServerDefault::_draw                                                                                                                                                                                   rendering_server_default.cpp 73   0x8f3afeb      
12 RenderingServerDefault::_thread_draw                                                                                                                                                                            rendering_server_default.cpp 341  0x8f3cb5f      
13 CommandQueueMT::Command2<RenderingServerDefault, void (RenderingServerDefault:: *)(bool, double), bool, double>::call                                                                                           command_queue_mt.h           322  0x8f90ae9      
14 CommandQueueMT::_flush                                                                                                                                                                                          command_queue_mt.h           373  0x8a326bc      
15 CommandQueueMT::flush_if_pending                                                                                                                                                                                command_queue_mt.h           404  0x8a328d9      
16 RenderingServerDefault::multimesh_set_buffer                                                                                                                                                                    rendering_server_default.h   341  0x8f42ac7      
17 CPUParticles3D::_update_render_thread                                                                                                                                                                           cpu_particles_3d.cpp         1263 0x7a86afd      
18 call_with_variant_args_helper<CPUParticles3D>(CPUParticles3D *, void (CPUParticles3D:: *)(), Variant const * *, Callable::CallError&, IndexSequence<>)                                                          binder_common.h              303  0x7aa3e09      
19 call_with_variant_args<CPUParticles3D>                                                                                                                                                                          binder_common.h              417  0x7aa3d79      
20 CallableCustomMethodPointer<CPUParticles3D>::call                                                                                                                                                               callable_method_pointer.h    104  0x7aa3c64      
21 Callable::callp                                                                                                                                                                                                 callable.cpp                 50   0x98e190a      
22 Object::emit_signalp                                                                                                                                                                                            object.cpp                   1072 0x9ca2f8e      
23 Object::emit_signal<>(StringName const&)                                                                                                                                                                        object.h                     891  0x5655b53      
24 RenderingServerDefault::_draw                                                                                                                                                                                   rendering_server_default.cpp 73   0x8f3afeb      
25 RenderingServerDefault::_thread_draw                                                                                                                                                                            rendering_server_default.cpp 341  0x8f3cb5f      
26 CommandQueueMT::Command2<RenderingServerDefault, void (RenderingServerDefault:: *)(bool, double), bool, double>::call                                                                                           command_queue_mt.h           322  0x8f90ae9      
27 CommandQueueMT::_flush                                                                                                                                                                                          command_queue_mt.h           373  0x8a326bc      
28 CommandQueueMT::wait_and_flush                                                                                                                                                                                  command_queue_mt.h           414  0x8a2c09f      
29 RenderingServerDefault::_thread_loop                                                                                                                                                                            rendering_server_default.cpp 363  0x8f3cc1d      
30 RenderingServerDefault::_thread_callback                                                                                                                                                                        rendering_server_default.cpp 350  0x8f3c5fd      
31 Thread::callback                                                                                                                                                                                                thread.cpp                   61   0x95fb09f      
32 std::__invoke_impl<void, void ( *)(unsigned long, Thread::Settings const&, void ( *)(void *), void *), unsigned long, Thread::Settings, void ( *)(void *), void *>                                              invoke.h                     60   0x95fc15a      
33 std::__invoke<void ( *)(unsigned long, Thread::Settings const&, void ( *)(void *), void *), unsigned long, Thread::Settings, void ( *)(void *), void *>                                                         invoke.h                     95   0x95fbfd1      
34 std::thread::_Invoker<std::tuple<void ( *)(unsigned long, Thread::Settings const&, void ( *)(void *), void *), unsigned long, Thread::Settings, void ( *)(void *), void *>>::_M_invoke<0ul, 1ul, 2ul, 3ul, 4ul> thread                       244  0x95fbf4d      
35 std::thread::_Invoker<std::tuple<void ( *)(unsigned long, Thread::Settings const&, void ( *)(void *), void *), unsigned long, Thread::Settings, void ( *)(void *), void *>>::operator()                         thread                       251  0x95fbeb5      
36 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void ( *)(unsigned long, Thread::Settings const&, void ( *)(void *), void *), unsigned long, Thread::Settings, void ( *)(void *), void *>>>::_M_run   thread                       195  0x95fb939      
37 execute_native_thread_routine                                                                                                                                                                                                                     0xa1c5c84      
38 start_thread                                                                                                                                                                                                    pthread_create.c             477  0x7ffff7a4c609 
39 clone                                                                                                                                                                                                           clone.S                      95   0x7ffff7822133 

multimesh_set_buffer ends up calling flush_if_pending, which ends up calling multimesh_set_buffer (and tries to flush again).

The value of command_mem.size() is 56

Update

So it appears that CommandQueueMT::_flush starts executing all commands, which ends up calling CPUParticles3D::_update_render_thread

_update_render_thread calls RS::get_singleton()->multimesh_set_buffer.

However despite being a virtual function, it doesn't seem to be overriden by anything except this:

FUNC2(multimesh_set_buffer, RID, const Vector<float> &)

Which is a macro that generates the following body:

virtual void multimesh_set_buffer( RID p1, const Vector<float> &p2 )
{
   RenderingServerDefault::redraw_request();

   if (Thread::get_caller_id() != server_thread) {
	command_queue.push(server_name, &ServerName::multimesh_set_buffer, p1, p2);
} else {
	command_queue.flush_if_pending(); // ---> we are here
	server_name->multimesh_set_buffer(p1, p2);
}   
}

[reduz]

IMO the problem is how the command queue is pushing the draw function and then using a sync command. This is very dangerous because it can create bizarre loops like the one described above.

#77004 should theoretically solve this, but it will probably not be merged until there is a decision on what to do with OpenGL.

[Listwon]

Still crashes in current 4.2.1 stable with MT rendering.
MRP with CPUParticles2D

Crashes here:

godot/core/templates/command_queue_mt.h

Line 373 in 6128206

SyncSemaphore *sync_sem = cmd->get_sync_semaphore();

So as reduz pointed out - the problem is that from whitin _flush there's a call to another _flush(). RenderingServer calls signal frame_pre_draw connected to _update_render_thread in CPUParticles2D which then calls multimesh_set_buffer. Because the calling thread (where RenderingServer is running) is the same as the server's (again RenderingServer) it jumps to the branch with command_queue.flush_if_pending() and then explodes.

[Summersay415]

Still crashes in current 4.2.1 stable with MT rendering. MRP with CPUParticles2D

Crashes here:

godot/core/templates/command_queue_mt.h

Line 373 in 6128206

SyncSemaphore *sync_sem = cmd->get_sync_semaphore();

So as reduz pointed out - the problem is that from whitin _flush there's a call to another _flush(). RenderingServer calls signal frame_pre_draw connected to _update_render_thread in CPUParticles2D which then calls multimesh_set_buffer. Because the calling thread (where RenderingServer is running) is the same as the server's (again RenderingServer) it jumps to the branch with command_queue.flush_if_pending() and then explodes.

Can't reproduce with 4.3dev6, but can with 4.3dev5, seems like fixed by #90268

[akien-mga]

Thanks for testing!