OSX/M1 window resizing crashes the engine when rendering thread model is set to multi-threaded

[lekoder]

Godot version

3.5.2

System information

MacBook Air M1 2020 8GB Sonoma 14.0

Issue description

On some M1 Mac systems, setting the rendering thread model to multi-threaded causes the engine to crash when the window size is changed.

This seems to be a race condition and gets progressively worse when the game is actually doing stuff - to the point where a game like ΔV: Rings of Saturn will crash on any attempt to resize the window or switch between full-screen and windowed modes. I have verified that adding a command line of --render-thread safe fixes the problem, so I'm confident that it's caused by the multi-threaded rendering model.

You can replicate this with an empty project without code - it just requires more vigorous mouse movements to trigger.

• Crash recording: https://imgur.com/a/ERIE01f
• OS crash report: crash-report.txt

Steps to reproduce

This replicates only on M1 Macs.

1. Create a new project
2. Set the rendering thread model to multi-threaded
3. Create an empty Node2D
4. Run it
5. Resize the window rapidly

MRP attached.

Minimal reproduction project

mrp.zip

[lekoder]

Also happening in 3.6 beta 3.

[akien-mga]

Fixed by #81442.

[lekoder]

TL;DR: Huge stability improvement, but I still get reports of crashes in production.

Unfortunately, it seems that this was not the root cause. While the #81442 does greatly reduce the rate of crashes (from "in a second" from before the patch to "in few minutes if you play with setting" after the patch), there seems to be another underlying problem that causes the instability, and disabling the live resize just makes it not immediately apparent. It seems that window resizing still has a small chance of crashing the engine, but after the patch we are just not resizing 60 times per second.

I am unable to replicate this anymore with a minimal project, but it does still happen in production. If you grab the free demo build from itch.io, enter the settings and either keep changing the fullscreen mode, or set the mode to windowed and keep resizing the window (and releasing the mouse button), it will cause a crash in a dozen of tries or so on an M1 system.

The .pck file in there should be compatible with any 3.5.3 build.

[bruvzg]

The crash seems to be caused by particles processing (deep inside the GPU driver after the RasterizerStorageGLES3::_particles_process call). So it is might be related to #72469, #72960 and #54052, If it is, I'm not sure if we can fix it on engine side, and since OpenGL is deprecated on macOS, driver won't be fixed either.

0   libsystem_kernel.dylib        	       0x18b86711c __pthread_kill + 8
1   libsystem_pthread.dylib       	       0x18b89ecc0 pthread_kill + 288
2   libsystem_c.dylib             	       0x18b7aea50 abort + 180
3   Delta-V                       	       0x102ce24b4 handle_crash(int) + 4264 (crash_handler_osx.mm:171)
4   libsystem_platform.dylib      	       0x18b8cda24 _sigtramp + 56
5   AppleMetalOpenGLRenderer      	       0x10af46aac GLDTextureRec::getTextureResource(unsigned int, bool) + 504
6   AppleMetalOpenGLRenderer      	       0x10af4faa0 GLDContextRec::buildRenderPassDescriptor() + 276
7   AppleMetalOpenGLRenderer      	       0x10af50064 GLDContextRec::beginRenderPass() + 96
8   AppleMetalOpenGLRenderer      	       0x10af660b4 gldRenderVertexArray(GLDContextRec*, unsigned int, unsigned int, int, int, unsigned int, void const*, int, void const*) + 480
9   GLEngine                      	       0x1f37b9930 glDrawArrays_GL3Exec + 232
10  Delta-V                       	       0x103bb302c RasterizerStorageGLES3::_particles_process(RasterizerStorageGLES3::Particles*, float) + 692 (rasterizer_storage_gles3.cpp:6578)
11  Delta-V                       	       0x103bb3c68 RasterizerStorageGLES3::update_particles() + 2000 (rasterizer_storage_gles3.cpp:6764)
12  Delta-V                       	       0x103bbd8a0 RasterizerStorageGLES3::update_dirty_resources() + 60 (rasterizer_storage_gles3.cpp:8533)
13  Delta-V                       	       0x103b67608 RasterizerGLES3::begin_frame(double) + 440 (rasterizer_gles3.cpp:212)
14  Delta-V                       	       0x105313ebc VisualServerRaster::draw(bool, double) + 380 (visual_server_raster.cpp:111)
15  Delta-V                       	       0x1053701a8 VisualServerWrapMT::thread_draw(bool, double) + 64 (visual_server_wrap_mt.cpp:40)
16  Delta-V                       	       0x1053cd8a0 CommandQueueMT::Command2<VisualServerWrapMT, void (VisualServerWrapMT::*)(bool, double), bool, double>::call() + 128 (command_queue_mt.h:300)
17  Delta-V                       	       0x105388a54 CommandQueueMT::flush_one(bool) + 352 (command_queue_mt.h:441)
18  Delta-V                       	       0x1053703cc CommandQueueMT::wait_and_flush_one() + 232 (command_queue_mt.h:478)
19  Delta-V                       	       0x1053702b4 VisualServerWrapMT::thread_loop() + 196 (visual_server_wrap_mt.cpp:63)
20  Delta-V                       	       0x1053701e4 VisualServerWrapMT::_thread_callback(void*) + 32 (visual_server_wrap_mt.cpp:49)
21  Delta-V                       	       0x10584ea74 Thread::callback(Thread*, Thread::Settings const&, void (*)(void*), void*) + 196 (thread.cpp:78)
22  Delta-V                       	       0x1058508c4 decltype(std::declval<void (*)(Thread*, Thread::Settings const&, void (*)(void*), void*)>()(std::declval<Thread*>(), std::declval<Thread::Settings>(), std::declval<void (*)(void*)>(), std::declval<void*>())) std::__1::__invoke[abi:v160006]<void (*)(Thread*, Thread::Settings const&, void (*)(void*), void*), Thread*, Thread::Settings, void (*)(void*), void*>(void (*&&)(Thread*, Thread::Settings const&, void (*)(void*), void*), Thread*&&, Thread::Settings&&, void (*&&)(void*), void*&&) + 72 (invoke.h:394)
23  Delta-V                       	       0x105850830 void std::__1::__thread_execute[abi:v160006]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(Thread*, Thread::Settings const&, void (*)(void*), void*), Thread*, Thread::Settings, void (*)(void*), void*, 2ul, 3ul, 4ul, 5ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(Thread*, Thread::Settings const&, void (*)(void*), void*), Thread*, Thread::Settings, void (*)(void*), void*>&, std::__1::__tuple_indices<2ul, 3ul, 4ul, 5ul>) + 96 (thread:288)
24  Delta-V                       	       0x105850068 void* std::__1::__thread_proxy[abi:v160006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(Thread*, Thread::Settings const&, void (*)(void*), void*), Thread*, Thread::Settings, void (*)(void*), void*>>(void*) + 84 (thread:299)
25  libsystem_pthread.dylib       	       0x18b89f034 _pthread_start + 136
26  libsystem_pthread.dylib       	       0x18b899e3c thread_start + 8

[Calinou]

As a workaround, I would probably disable GPUParticles rendering on macOS when not using ANGLEMetal if using the Multi-Threaded model (and print a warning).

[lekoder]

If this is caused by particles, and disabling such particles fixes the issue, an approach that seems reasonable is to blanket-disable all GPUParticles when the resize procedure starts and then re-enable them when it finishes.

I could probably implement that on the game side, but this is a nasty gotcha for anybody developing cross-platform, so it should probably be either prominently documented or handled on the engine side.