Improve asset importing and file renaming speeds for projects with a lot of files

[MightyPrinny]

Describe the project you are working on:
We have a huge community made game maker project with assets from over 50 people + a lot of assets already in the base game, I wanted to see if we could move our engine to Godot for future games.

Describe the problem or limitation you are having in your project:
I added a lot of assets to a new project to see how Godot would handle it, I took the assets from our game maker project and since game maker stores every animation frame individually I ended up with thousands of files, importing everything took longer than loading the project in game maker(that takes very long too).

Godot just couldn't handle that much, now adding new assets and renaming files takes several minutes, between audio and image files I had 52960 files in total counting the .import files.

Describe how this feature / enhancement will help you overcome this problem or limitation:
I could move the project over to Godot for future games if this wasn't an issue, I know that if we were to do that we would have much less files since sprites would be in sprite sheets, but I didn't include any of the the background files from our project and the project isn't finished, there's a lot to be done, so I'm not sure if it the engine could handle a project of that size even with sprite sheets.

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:
I don't think it's necessary since it's about optimization.

Describe implementation detail for your proposal (in code), if possible:
Sorry, I can't help much with this.

If this enhancement will not be used often, can it be worked around with a few lines of script?:
Definitely not.

Is there a reason why this should be core and not an add-on in the asset library?:
It's an optimization.

[Calinou]

I wonder if hiding the editor progress dialog would speed up importing, as the current implementation is known to slow down tasks significantly when it has to follow a large amount of steps. See also godotengine/godot#30850.

If this is the case, we can provide an editor setting to hide the progress dialog. This wouldn't be ideal, but it'd be better than nothing. Profiling the editor progress dialog would be welcome too 🙂

[MightyPrinny]

I wasn't talking about that(not completely that might help with improving the time it takes to rename a file), there are no progress bars when the editor scans the file system, other programs can detect file changes faster than Godot, my problem was mainly that when you add new assets to a project with a lot of files it takes a while before Godot realizes there's a new file and imports it.

[AaronRecord]

I wonder if it'd make sense to store all the import settings in one .import file in the project root that would be loaded when the project loads. This could allow resources with the same import settings to not have their import settings be duplicated. An idea for how it could be implemented is storing it as a dictionary where the resource path is the key and the value is a reference to its import settings (multiple resources could point to the same import settings). The disadvantage of this it might be harder to export a group of assets with their export settings, to share/publish them etc., maybe there should be a way to do this (like .unitypackage) (or maybe there already is a way to do this with .pck files that I'm not familiar with).

[Calinou]

Resource importing is now multithreaded in master (since 4.0alpha1 at least). I'm not sure if much can be done to improve import speeds in 3.x, other than using more aggressive compiler optimizations.

[Saul2022]

It might be good to test this stuff now that there wrre multiple improvements on the past years along with some upcoming prs like this godotengine/godot#95678 (comment) or godotengine/godot#93064 (comment)