Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change MessageQueue to a page allocator to prevent overflow #35653

Closed
akien-mga opened this issue Jan 28, 2020 · 6 comments · Fixed by #75940
Closed

Change MessageQueue to a page allocator to prevent overflow #35653

akien-mga opened this issue Jan 28, 2020 · 6 comments · Fixed by #75940

Comments

@akien-mga
Copy link
Member

akien-mga commented Jan 28, 2020

Godot version:
Master (3cfb67e)

OS/device including version:
Any

Issue description:
As discussed a few times on IRC, and exemplified by many bug reports, our current fixed size for the Message Queue is problematic.

It currently defaults to 1024 (see memory/limits/message_queue/max_size_kb), and many users need to increase it to prevent overflow when handling a large amount of messages (which can happen easily in the editor when using e.g. large TileSets #29238 or Skeletons #35652).

As discussed with @reduz, Message Queue should be changed to use a page allocator so that its size can grow as needed, preventing overflows.

Situations which can cause a Message Queue overflow like #29238 or #35652 should however be fixed in the Inspector itself, as even increasing the MQ will just make things functional again, but it will still freeze a long time while populating all those controls.


Example output to help uses find this issue:

Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
TOTAL BYTES: 1048560
NULL count: 8
CALL : 3
CALL _update_callback: 37181
CALL _update_minimum_size: 5
CALL _sort_children: 6479
CALL _emit_change: 1
CALL _update_script_names: 1
CALL _update_script_connections: 1
CALL _test_update_tree: 9
CALL _node_selected: 1
CALL _update_bone_list: 1
ERROR: push_call: Message queue out of memory. Try increasing 'message_queue_size_kb' in project settings.
   At: core/message_queue.cpp:55.
Failed method: VBoxContainer:_update_callback target ID: 74982
@akien-mga
Copy link
Member Author

For the reference, I fixed the outdated setting name in the error message with cab555bb, so from now on errors will be:

Message queue out of memory. Try increasing 'memory/limits/message_queue/max_size_kb' in project settings.

@staddy
Copy link
Contributor

staddy commented Jan 28, 2020

My commit doesn't fix #35652 though.
_call_function(target, message->target, args, message->args, message->type & FLAG_SHOW_ERROR);
in MessageQueue::flush() corrupts memory; think it's not my fault, but not 100% sure.

@Xrayez
Copy link
Contributor

Xrayez commented Jul 22, 2020

How about increasing the default value to something like 2048-4096 (while we wait for page allocator to get merged/implemented)? I've set this value to 2048 in my own project and I've never got to experience those errors again. I can only guess though. Just see the number of "duplicate" bug reports with the same underlying issue (unless you're interested in collecting the use cases this limitation has on people's projects).

Some other projects which do happen to set the size to 4096: yumedev40/Yume-Visual-Novel-Editor#1

Is the value of 1024 caused by mobile limitations?

akien-mga added a commit to akien-mga/godot that referenced this issue Sep 24, 2020
1024 KB was low enough that many users seem to hit it, which can lead to the
editor freezing.

The proper fixed as described in godotengine#35653 would be to implement a page allocator
to prevent this overflow, but as a stop-gap measure, we can increase the
default value to a more lenient 4096 KB which should be high enough for the
vast majority of use cases.

The default size can be brought down again if/when godotengine#35653 is properly fixed,
and if it's actually relevant from a memory point of view.
akien-mga added a commit to akien-mga/godot that referenced this issue Sep 29, 2020
1024 KB was low enough that many users seem to hit it, which can lead to the
editor freezing.

The proper fixed as described in godotengine#35653 would be to implement a page allocator
to prevent this overflow, but as a stop-gap measure, we can increase the
default value to a more lenient 4096 KB which should be high enough for the
vast majority of use cases.

The default size can be brought down again if/when godotengine#35653 is properly fixed,
and if it's actually relevant from a memory point of view.

(cherry picked from commit 5009ba5)
@akien-mga akien-mga moved this to To Assess in 4.x Priority Issues Mar 26, 2022
@akien-mga akien-mga moved this from To Assess to Todo in 4.x Priority Issues Mar 26, 2022
reduz added a commit to reduz/godot that referenced this issue Jun 23, 2022
* Uses simple vector resizing (po2)

Fixes godotengine#35653
Supersedes godotengine#35658
@YuriSizov YuriSizov moved this from Todo to In Progress in 4.x Priority Issues Sep 12, 2022
@akien-mga akien-mga modified the milestones: 4.0, 4.x, 4.1 Jan 31, 2023
@Atlinx
Copy link
Contributor

Atlinx commented Mar 16, 2023

Hey all I'm getting this error in 3.x,

Failed method: MarginContainer:_update_callback target ID: 613365
Failed method: RichTextLabel:_update_callback target ID: 9387
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
...
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
Object was deleted while awaiting a callback
TOTAL BYTES: 4194288
NULL count: 185
CALL : 174575
CALL _update_callback: 2
ERROR: Message queue out of memory. Try increasing 'memory/limits/message_queue/max_size_kb' in project settings.

I'm not sure if this is related to this issue.

EDIT: I fixed this problem by making some of my UI code use called deferred. Maybe removing a child and then adding it back to the scene causes too many updates?

@wecybin
Copy link

wecybin commented Mar 16, 2023

Related to this issue I also posted in recently : #71929 (comment)

I started off with receiving :

ERROR: Message queue out of memory. Try increasing 'memory/limits/message_queue/max_size_kb' in project settings.

After turning on SDFGI and a few other settings, then trying to change their quality to be lower in the project settings resulted in this + an out of memory exception windows error. I ended up increasing the kb to 10,000 and it seemed to fix it, however I started getting the issue I linked above resulting in this error printing on the console over and over while godot is unresponsive :

ERROR: Vulkan: Did not create swapchain successfully. Error code: VK_NOT_READY
at: prepare_buffers (drivers/vulkan/vulkan_context.cpp:2280)
ERROR: Vulkan: Cannot submit graphics queue. Error code: VK_ERROR_DEVICE_LOST
at: swap_buffers (drivers/vulkan/vulkan_context.cpp:2357)

So fast forward 2-3 days of this error popping up, I just created a master scene with a canvas layer and crashed. Now I am unable to open the project and I'm receiving the original memory limit error in the console.
Edit: Managed to get it to open again by removing the main scene, opening and changing the kb size to 20,000 and then reopening the main scene. Something about that scene it just doesn't like for some reason..

Edit: Removed 2nd comment as it ended up being unrelated, apologies.

@akien-mga
Copy link
Member Author

Fixed in 3.6 by #75527.

@github-project-automation github-project-automation bot moved this from In Progress to Done in 4.x Priority Issues Apr 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment