Large projects seem frozen/crashed for several minutes on startup with poor editor feedback

[smix8]

Godot version:
v3.2.4.rc4.official
v4.x vulkan

OS/device including version:
Win10 64x

Issue description:
The editor seems to have a real struggle with very large Godot projects (50+ gb) ones per day on first startup.
I just clocked, the editor appears completely frozen/crashed for 7+ minutes on startup.

There is no feedback in both editor or console for users what is going on after the class_names and autoload scripts are successfully loaded.

There seems to be an issue with the file scanthread implementation. It seems the file scanthread has the desire to not only read all scriptfiles for changes but also open every single file each new day for no practical reason. As long as the scanthread is busy it blocks the entire editor until finished instead of working behind the scene.

The result is that the project appears nearly empty in the editor for a long time (shock moment, where are my files?) and also the editor stays unresponsive for several minutes as if it had a crash.

How to solve?

• Give continues, visual feedback to user about project load progress.
• Make file scanthread behave like a real, subtle background thread and not a nuisance by blocking editor startup until finished.
• Stop scanthread from opening all unchanged and very large asset files each day for no apparent, practical reason.

Steps to reproduce:
Create large Godot projects

Minimal reproduction project:
project size issue

[Calinou]

Related to #39841.

Make file scanthread behave like a real, subtle background thread and not a nuisance by blocking editor startup until finished.

This most likely won't change until 4.0 (and even then, I'm not sure). The issue with performing importing in the background is that you can easily end up with an inconsistent state for resources, since the user will be able to open and edit scenes before everything is imported.

However, parallel resource importing is planned to speed up the import process.

[smix8]

I understand. My issue is not related to firsttime imports of resources or open scripts files. I close scriptfiles out of habit when I finish my workday to avoid any slowdowns. Even with 100+ scripts open on a busy day the restart of both pc and the entire Godot editor takes less than a minute cause everything is on a ssd.

It is always only the firsttime editor start each day that bugs me cause it takes ages. Godot wants to reopen all the dusted, heavy 3D assets in the project folder for whatever reason and I can't understand why it must do that. Isn't the point of all those extra created import meta files and timestamps to avoid this constant reloading of heavy assets that havn't changed?

[smix8]

After source scanning through a couple editor functions I think a general issue with the lack of feedback on so many ends is that most editor functions that display a progress (bar) run and update only with the process notification.

Many of the problematic editor functions that take very long on large projects lock the entire editor until finished (no short waitusec or await in the loop). In this case a second frame that could show updated progress is never drawn cause the process notification is never send until the loop is finished. Without any frame updates for a few seconds e.g. Windows thinks that Godot has crashed when a user clicks on the window.

[skaiware]

Hi @smix8
Similar issue here, actually worst, perhaps with the same root cause.
Our current project is about (at the moment) total 500MB of models/textures/sounds... and about 50 gdscripts.
It happens when opening our project for the first time
OR
trying to upgrade to 3.3.2 from 3.3.1, the editor tries to reimport everything, allocates up to 16GB of RSS, and then eventually crashes probably because of a OOM after a dozen of minutes.
This makes impossible to:

• upgrade to 3.3.2
• open the project from scratch on a 16GB RAM machine

The .import folder is about 3GB.
To verify if same root cause, have you kept an eye on the mem usage of the editor process when your issue happens ?
Kind

[akien-mga]

trying to upgrade to 3.3.2 from 3.3.1, the editor tries to reimport everything, allocates up to 16GB of RSS, and then eventually crashes probably because of a OOM after a dozen of minutes.

That's surprising, as there's only a handful of commits between 3.3.1 and 3.3.2 and they wouldn't justify reimporting assets, as they don't invalidate previous imports.

I just tried opening a fully imported 3.3.1 project in 3.3.2 and no reimport was caused.

[Zireael07]

Yeah, suprised as I have around 100 MB of assets in my racer game, and dozens of scripts (probably somewhere in the vicinity of 50), and no reimport from 3.3.1. to 3.3.2, nor crashes.

@skaiware: There were changes to FBX import recently, do you use any FBX assets?

[skaiware]

Hi @Zireael07, Tks, good catch, we have indeed in this project about 50 fbx files.

[smix8]

@skaiware
No memory issues, my rig has plenty and Godot never seems to use more than 2-3 gb at the same time while in the editor.

[Zireael07]

Bump. Bad UX on large projects still happens, both in 4.0 and 3.x (my racer project is enough to stall the loading noticeably, I dunno, 5-10 secs)

[smix8]

Issue is still valid and more and more severe with growing project size.

From limited profiling on Windows10 I learned that the EditorFileSystem functions are not the main issue.
It takes long on first startup because it does a full recheck of all imported files with a lot of inefficient single file open/read operations but it is still done after roughly 50 seconds which is ok for a very large project (200+gb) full of small single files.

Godot afterwards gets stuck for multiple minutes due to sleep executions and thread waits.

Most problems seem to come from
EditorExportPlatformJavaScript::_server_thread_poll
_IP_ResolverPrivate::_thread_function
ScriptServer::thread_exit

So it seems to be a lock/thread issue that blocks the Editor from finishing after firsttime startup while it "does nothing".

[Calinou]

It takes long on first startup because it does a full recheck of all imported files with a lot of inefficient single file open/read operations

Do you know how to make those file open/read operations more efficient? If so, could you detail it here?

Most problems seem to come from
EditorExportPlatformJavaScript::_server_thread_poll
_IP_ResolverPrivate::_thread_function
ScriptServer::thread_exit

So it seems to be a lock/thread issue that blocks the Editor from finishing after firsttime startup while it "does nothing".

See #59787. Pull requests to make those use less CPU (and sleep when not required) are welcome 🙂

[akien-mga]

Most problems seem to come from
EditorExportPlatformJavaScript::_server_thread_poll
_IP_ResolverPrivate::_thread_function
ScriptServer::thread_exit
So it seems to be a lock/thread issue that blocks the Editor from finishing after firsttime startup while it "does nothing".

See #59787. Pull requests to make those use less CPU (and sleep when not required) are welcome slightly_smiling_face

I don't think that export plugins is causing the perceive freeze here though. It's normal that you see those as active threads if you stop the execution at a given time and inspect all threads - but that doesn't mean that they're the ones blocking the main thread.
ScriptServer might be relevant though if there are lots of scripts which need to be parsed on startup (not sure exactly what the ScriptServer thread does exactly).

[TechnoPorg]

Was this issue improved at all by #57766?

[Zireael07]

That was only for the editor itself - I still see the stall caused by opening my main scene.

[reduz]

I think this may have been effectively fixed by #57766, @Zireael07 project startup time is unrelated so I suggest you open a separate issue if you have a specific problem with running the project.

[Zireael07]

I effectively solved the issue by changing the "main scene" that automatically opens to something less heavy :P

[Jordyfel]

This is still present in 4.2 beta3.

This has been happening with an asset I put on the asset lib, which can also serve as a good reproduction project.

Steps to reproduce

1. Create a new godot project
2. Download https://godotengine.org/asset-library/asset/1653 (~2400 files)
3. Reload the project; filesystem scan time is reasonable (~1s on my pc)
4. Restart PC
5. Load the project; filesystem scan time is unreasonable (>10s on my pc)
6. Reload again; time is reasonable again

[Hilderin]

I tested a new project with https://godotengine.org/asset-library/asset/1653

With this PR #92650 it fixes the problem that the editor is frozen on start.

I was able to reproduce the problem that in takes longer after a fresh reboot. The problem comes from the cache validation which opens all the .import files on startup.

I did a couple of tests and I saw the process MsMpEng.exe spikes in CPU usage when starting the project after a fresh reboot. I added the project folder in the Windows Defender exceptions, and now even after a fresh reboot, the project loads really fast. I don't think Godot can do anyway about it except not rescanning the cache on startup, which does not seems a good idea.

Maybe an option to skip the cache validation on startup could be added with an option to manually force a full rescan for large projects? What takes time is not really findings all files, it is to read the .import file and validate the cache.

[Calinou]

I did a couple of tests and I saw the process MsMpEng.exe spikes in CPU usage when starting the project after a fresh reboot. I added the project folder in the Windows Defender exceptions, and now even after a fresh reboot, the project loads really fast.

We should document this somewhere, probably on the Troubleshooting and Import process pages. Any other locations where we should document this?

[Zireael07]

I am on Linux so my issue is definitely unrelated to MsMpEng.exe which is a part of Windows Defender

[akien-mga]

Reopening as #92650 was reverted for 4.3.

[Hilderin]

@smix8
I found a Project Setting that I did not know about. Can you test disabling the Project Setting "Editor > Import > Reimport Missing Imported Files" to see if the issue where all the assets are reimporting on the first startup on the day is still present:

If the issue is still present it will confirms that the source of the problem is the last modified date.

[smix8]

@Hilderin
Since I can only test this once per day on the project in question what I did is add

[editor]

import/reimport_missing_imported_files=false

to the ProjectSettings before starting it. It made no difference as Godot would still load everything.

[Hilderin]

@smix8
Can you try to see if #95678 fixes the problem where all your assets are reload on the first startup of the day? If my hypothesis is correct, on the this startup of the day, it will be a bit longer because md5 check should be triggered. Even if the last modification dates are different, now that I added an md5 of the .import content, the md5 should be check and it should be the same, preventing a reimportation.

[smix8]

@Hilderin
I used the Windows build artifact of that PR but couldn't see any noticeable speed difference on the first start. Godot was still scanning slowly the entire project. The build didn't have debug symbols so not sure what it all did, but it couldn't be just the md5 check, because doing an md5 check for the entire project in slow gdscript takes less than 2 seconds while the scan that Godot did took multiple minutes.

[Saul2022]

. Godot was still scanning slowly the entire project.

Maybe it because you have a ton of gbs worth of files, i doubt any engine can handle that pretty well, though can you try duplicating your project, deleting the .godot folder of that project and then start that one , then on second load it must be way faster to scan, just being the scene slowing you down.

[Hilderin]

@smix8
The ultimate test would be to start the project with the artifact from the PR to be sure the md5 is correctly saved in the cache and the following moring, starting the project with the same artifact.
Also, can you send me your .godot/editor/filesystem_cache8 file. That will tell me more about your project without having to share the lot project. Your can send me the file in private in to godot forum if you prefer.

Note: I also notice that loading a project for the first time in a day is a bit slower (not as slow as you are experiencing), I always tought it was the disk cache but maybe there's something else going on.