Scons performance for rebuilds is slow

[RevoluPowered]

Issue description:
Rebuilding a single file on high-end systems takes half of a full rebuild time.

Steps to reproduce:
time scons -j16
make change to some cpp file
time scons -j16

I want to do some research into this, so gonna use this issue and document it.

[RevoluPowered]

jfons: it's computing a bunch of MD5 checksums that it shouldn't really need
11:13 PM for some reason the timestamp check is not doing its job

[RevoluPowered]

Normal MD5 timestamp

time scons -j16
[Initial build] scons: done building targets.
scons -j16  2283.42s user 180.40s system 1030% cpu 3:58.98 total

[RevoluPowered]

using Decider('timestamp-match')
scons -j16 2345.50s user 184.19s system 1095% cpu 3:50.99 total

[marstaik]

scons -j1 --profile=sconsprofiling.log target=release_debug with zero changes
sconsprofiling.log

[RevoluPowered]

@RaTcHeT302 we're profiling and fixing the problems, we already got OSX build times down to 14s for single file. So please try to keep on topic for the actual fixing part, we aren't focusing on 'scons is crappy' we want to work out what the problem is and fix it.

Findings from research

I measured build times and am consistently getting everything under 30s for single file rebuilds (which is not bad):

• you must disable windows defender (shaved 1m) off build times for full rebuild and got scons -j16 to be under 30s on windows.
• we need to possibly swap to /DEBUG:fastlink (10s vs 15s)
• linker commands with MSVC appear to be running out of order? (TBC, reproduction steps: delete exe and files in the bin/ dir and build with verbose=yes, look at the commands for linking they're weirdly ordered)
• the actual commands to the compiler are slow not the scons invokation
• the only slow part of scons is generating and not caching the -D defines from all the profiling files. (proven by loading scons profiling information from python and checking in RunSnakeRun)
• we should be able to shave a lot of time off builds if we cache the defines.

Right now manually linking with the direct commands is yeilding these results, so our link times are 10 or 15 seconds without /DEBUG:fastlink

Changing MD5 decider to timestamp-match etc only shaves off 50ms

[RevoluPowered]

Here is an example of the out of order linking flags
https://gist.github.com/RevoluPowered/dfc7c914eee09c94824b14c851519a58

[RevoluPowered]

Seems only assimp is the problem, i am testing linkage times without modules/assimp and thirdparty/assimp for now, might be a bug.

[Calinou]

@RaTcHeT302 Per the Godot Engine Code of Conduct, please stop derailing issues with unrelated conversations. This is your last warning.

Feedback is always welcome but keep your criticism constructive. We encourage you to open discussions, proposals, issues, and bug reports. Use the community platforms to discuss improvements, not to vent out frustration. Similarly, when other users offer you feedback please accept it gracefully.

(emphasis mine)

[RevoluPowered]

Here are some results from more tests with antivirus off in group policy.

# /INCREMENTAL /DEBUG:fastlink
C:\Users\Gordon\Projects\godot>timecmd.bat link /nologo /DEBUG:fastlink /INCREMENTAL /SUBSYSTEM:WINDOWS winmm.lib dsound.lib kernel32.lib ole32.lib oleaut32.lib user32.lib gdi32.lib IPHLPAPI.lib Shlwapi.lib wsock32.lib Ws2_32.lib shell32.lib advapi32.lib dinput8.lib dxguid.lib imm32.lib bcrypt.lib Avrt.lib dwmapi.lib cfgmgr32.lib opengl32.lib /STACK:8388608 /OUT:bin\godot.windows.tools.64.exe main\main.windows.tools.64.lib modules\modules.windows.tools.64.lib modules\module_xatlas_unwrap.windows.tools.64.lib modules\module_websocket.windows.tools.64.lib modules\module_webrtc.windows.tools.64.lib modules\module_webp.windows.tools.64.lib modules\module_webm.windows.tools.64.lib modules\module_vorbis.windows.tools.64.lib modules\module_visual_script.windows.tools.64.lib modules\module_vhacd.windows.tools.64.lib modules\module_upnp.windows.tools.64.lib modules\module_tinyexr.windows.tools.64.lib modules\module_theora.windows.tools.64.lib modules\module_tga.windows.tools.64.lib modules\module_svg.windows.tools.64.lib modules\module_stb_vorbis.windows.tools.64.lib modules\module_squish.windows.tools.64.lib modules\module_regex.windows.tools.64.lib modules\module_pvr.windows.tools.64.lib modules\module_opus.windows.tools.64.lib modules\module_opensimplex.windows.tools.64.lib modules\module_ogg.windows.tools.64.lib modules\module_mobile_vr.windows.tools.64.lib modules\module_mbedtls.windows.tools.64.lib modules\module_lightmapper_rd.windows.tools.64.lib modules\module_jsonrpc.windows.tools.64.lib modules\module_jpg.windows.tools.64.lib modules\module_hdr.windows.tools.64.lib modules\module_gridmap.windows.tools.64.lib modules\module_glslang.windows.tools.64.lib modules\module_gdscript.windows.tools.64.lib modules\module_gdnavigation.windows.tools.64.lib modules\module_gdnative.windows.tools.64.lib modules\module_freetype.windows.tools.64.lib modules\module_etc.windows.tools.64.lib modules\module_enet.windows.tools.64.lib modules\module_denoise.windows.tools.64.lib modules\module_dds.windows.tools.64.lib modules\module_cvtt.windows.tools.64.lib modules\module_csg.windows.tools.64.lib modules\module_camera.windows.tools.64.lib modules\module_bullet.windows.tools.64.lib modules\module_bmp.windows.tools.64.lib modules\module_basis_universal.windows.tools.64.lib modules\module_assimp.windows.tools.64.lib platform\platform.windows.tools.64.lib drivers\drivers.windows.tools.64.lib editor\editor.windows.tools.64.lib scene\scene.windows.tools.64.lib servers\servers.windows.tools.64.lib core\core.windows.tools.64.lib modules\freetype\freetype_builtin.windows.tools.64.lib platform\windows\godot_windows.windows.tools.64.obj platform\windows\crash_handler_windows.windows.tools.64.obj platform\windows\os_windows.windows.tools.64.obj platform\windows\display_server_windows.windows.tools.64.obj platform\windows\key_mapping_windows.windows.tools.64.obj platform\windows\joypad_windows.windows.tools.64.obj platform\windows\windows_terminal_logger.windows.tools.64.obj platform\windows\vulkan_context_win.windows.tools.64.obj platform\windows\context_gl_windows.windows.tools.64.obj platform\windows\godot_res.windows.tools.64.obj
   Creating library bin\godot.windows.tools.64.lib and object bin\godot.windows.tools.64.exp
command took 0:0:12.14 (12.14s total)
# /DEBUG (current default)
C:\Users\Gordon\Projects\godot>timecmd.bat link /nologo /DEBUG /SUBSYSTEM:WINDOWS winmm.lib dsound.lib kernel32.lib ole32.lib oleaut32.lib user32.lib gdi32.lib IPHLPAPI.lib Shlwapi.lib wsock32.lib Ws2_32.lib shell32.lib advapi32.lib dinput8.lib dxguid.lib imm32.lib bcrypt.lib Avrt.lib dwmapi.lib cfgmgr32.lib opengl32.lib /STACK:8388608 /OUT:bin\godot.windows.tools.64.exe main\main.windows.tools.64.lib modules\modules.windows.tools.64.lib modules\module_xatlas_unwrap.windows.tools.64.lib modules\module_websocket.windows.tools.64.lib modules\module_webrtc.windows.tools.64.lib modules\module_webp.windows.tools.64.lib modules\module_webm.windows.tools.64.lib modules\module_vorbis.windows.tools.64.lib modules\module_visual_script.windows.tools.64.lib modules\module_vhacd.windows.tools.64.lib modules\module_upnp.windows.tools.64.lib modules\module_tinyexr.windows.tools.64.lib modules\module_theora.windows.tools.64.lib modules\module_tga.windows.tools.64.lib modules\module_svg.windows.tools.64.lib modules\module_stb_vorbis.windows.tools.64.lib modules\module_squish.windows.tools.64.lib modules\module_regex.windows.tools.64.lib modules\module_pvr.windows.tools.64.lib modules\module_opus.windows.tools.64.lib modules\module_opensimplex.windows.tools.64.lib modules\module_ogg.windows.tools.64.lib modules\module_mobile_vr.windows.tools.64.lib modules\module_mbedtls.windows.tools.64.lib modules\module_lightmapper_rd.windows.tools.64.lib modules\module_jsonrpc.windows.tools.64.lib modules\module_jpg.windows.tools.64.lib modules\module_hdr.windows.tools.64.lib modules\module_gridmap.windows.tools.64.lib modules\module_glslang.windows.tools.64.lib modules\module_gdscript.windows.tools.64.lib modules\module_gdnavigation.windows.tools.64.lib modules\module_gdnative.windows.tools.64.lib modules\module_freetype.windows.tools.64.lib modules\module_etc.windows.tools.64.lib modules\module_enet.windows.tools.64.lib modules\module_denoise.windows.tools.64.lib modules\module_dds.windows.tools.64.lib modules\module_cvtt.windows.tools.64.lib modules\module_csg.windows.tools.64.lib modules\module_camera.windows.tools.64.lib modules\module_bullet.windows.tools.64.lib modules\module_bmp.windows.tools.64.lib modules\module_basis_universal.windows.tools.64.lib modules\module_assimp.windows.tools.64.lib platform\platform.windows.tools.64.lib drivers\drivers.windows.tools.64.lib editor\editor.windows.tools.64.lib scene\scene.windows.tools.64.lib servers\servers.windows.tools.64.lib core\core.windows.tools.64.lib modules\freetype\freetype_builtin.windows.tools.64.lib platform\windows\godot_windows.windows.tools.64.obj platform\windows\crash_handler_windows.windows.tools.64.obj platform\windows\os_windows.windows.tools.64.obj platform\windows\display_server_windows.windows.tools.64.obj platform\windows\key_mapping_windows.windows.tools.64.obj platform\windows\joypad_windows.windows.tools.64.obj platform\windows\windows_terminal_logger.windows.tools.64.obj platform\windows\vulkan_context_win.windows.tools.64.obj platform\windows\context_gl_windows.windows.tools.64.obj platform\windows\godot_res.windows.tools.64.obj
   Creating library bin\godot.windows.tools.64.lib and object bin\godot.windows.tools.64.exp
command took 0:0:17.39 (17.39s total)
# /DEBUG:fastlink (suggested default for debug)
:\Users\Gordon\Projects\godot>timecmd.bat link /nologo /DEBUG:fastlink /SUBSYSTEM:WINDOWS winmm.lib dsound.lib kernel32.lib ole32.lib oleaut32.lib user32.lib gdi32.lib IPHLPAPI.lib Shlwapi.lib wsock32.lib Ws2_32.lib shell32.lib advapi32.lib dinput8.lib dxguid.lib imm32.lib bcrypt.lib Avrt.lib dwmapi.lib cfgmgr32.lib opengl32.lib /STACK:8388608 /OUT:bin\godot.windows.tools.64.exe main\main.windows.tools.64.lib modules\modules.windows.tools.64.lib modules\module_xatlas_unwrap.windows.tools.64.lib modules\module_websocket.windows.tools.64.lib modules\module_webrtc.windows.tools.64.lib modules\module_webp.windows.tools.64.lib modules\module_webm.windows.tools.64.lib modules\module_vorbis.windows.tools.64.lib modules\module_visual_script.windows.tools.64.lib modules\module_vhacd.windows.tools.64.lib modules\module_upnp.windows.tools.64.lib modules\module_tinyexr.windows.tools.64.lib modules\module_theora.windows.tools.64.lib modules\module_tga.windows.tools.64.lib modules\module_svg.windows.tools.64.lib modules\module_stb_vorbis.windows.tools.64.lib modules\module_squish.windows.tools.64.lib modules\module_regex.windows.tools.64.lib modules\module_pvr.windows.tools.64.lib modules\module_opus.windows.tools.64.lib modules\module_opensimplex.windows.tools.64.lib modules\module_ogg.windows.tools.64.lib modules\module_mobile_vr.windows.tools.64.lib modules\module_mbedtls.windows.tools.64.lib modules\module_lightmapper_rd.windows.tools.64.lib modules\module_jsonrpc.windows.tools.64.lib modules\module_jpg.windows.tools.64.lib modules\module_hdr.windows.tools.64.lib modules\module_gridmap.windows.tools.64.lib modules\module_glslang.windows.tools.64.lib modules\module_gdscript.windows.tools.64.lib modules\module_gdnavigation.windows.tools.64.lib modules\module_gdnative.windows.tools.64.lib modules\module_freetype.windows.tools.64.lib modules\module_etc.windows.tools.64.lib modules\module_enet.windows.tools.64.lib modules\module_denoise.windows.tools.64.lib modules\module_dds.windows.tools.64.lib modules\module_cvtt.windows.tools.64.lib modules\module_csg.windows.tools.64.lib modules\module_camera.windows.tools.64.lib modules\module_bullet.windows.tools.64.lib modules\module_bmp.windows.tools.64.lib modules\module_basis_universal.windows.tools.64.lib modules\module_assimp.windows.tools.64.lib platform\platform.windows.tools.64.lib drivers\drivers.windows.tools.64.lib editor\editor.windows.tools.64.lib scene\scene.windows.tools.64.lib servers\servers.windows.tools.64.lib core\core.windows.tools.64.lib modules\freetype\freetype_builtin.windows.tools.64.lib platform\windows\godot_windows.windows.tools.64.obj platform\windows\crash_handler_windows.windows.tools.64.obj platform\windows\os_windows.windows.tools.64.obj platform\windows\display_server_windows.windows.tools.64.obj platform\windows\key_mapping_windows.windows.tools.64.obj platform\windows\joypad_windows.windows.tools.64.obj platform\windows\windows_terminal_logger.windows.tools.64.obj platform\windows\vulkan_context_win.windows.tools.64.obj platform\windows\context_gl_windows.windows.tools.64.obj platform\windows\godot_res.windows.tools.64.obj
   Creating library bin\godot.windows.tools.64.lib and object bin\godot.windows.tools.64.exp
command took 0:0:12.28 (12.28s total)

[RevoluPowered]

Testing in go_faster_mk2_testing

I have got single file rebuild time to: 13.092s with MSVC on Windows 10. 🎆

This CPU is a Ryzen 7 3700X for now it 'forces' /MT16 since I have 16 threads, later I will try and fix the warning in LINK.

$ time scons -j16
scons: Reading SConscript files ...
Automatically detected platform: windows
Configuring for Windows: target=debug, bits=default
Found MSVC version 14.2, arch amd64, bits=64
YASM is necessary for WebM SIMD optimizations.
WebM SIMD optimizations are disabled. Check if your CPU architecture, CPU bits or platform are supported!
Checking for C header file mntent.h... (cached) no
scons: done reading SConscript files.
scons: Building targets ...
Compiling ==> core\image.cpp
image.cpp
progress_finish(["progress_finish"], [])
Linking Static Library ==> core\core.windows.tools.64.lib
Linking Program        ==> bin\godot.windows.tools.64.exe
LINK : warning LNK4044: unrecognized option '/MT16'; ignored
scons: done building targets.

real    0m13.092s
user    0m0.000s
sys     0m0.015s

Entire build time is down from 3m to 2m 15s.

$ time scons -j16
scons: Reading SConscript files ...
Automatically detected platform: windows
Configuring for Windows: target=debug, bits=default
Found MSVC version 14.2, arch amd64, bits=64
YASM is necessary for WebM SIMD optimizations.
WebM SIMD optimizations are disabled. Check if your CPU architecture, CPU bits or platform are supported!
Checking for C header file mntent.h... no
scons: done reading SConscript files.
scons: Building targets ...
Generating method binders.
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\canvas.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\canvas_occlusion.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\scene_high_end.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\giprobe_sdf.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\gi.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\giprobe.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\giprobe_debug.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\sdfgi_debug.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\sdfgi_debug_probes.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\sdfgi_direct_light.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\sdfgi_integrate.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\sdfgi_preprocess.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\sky.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\volumetric_fog.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\bokeh_dof.glsl.gen.h"
Compiling ==> platform\windows\godot_windows.cpp
Compiling ==> platform\windows\crash_handler_windows.cpp
godot_windows.cpp
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\copy.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\copy_to_fb.glsl.gen.h"
crash_handler_windows.cpp
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\cubemap_downsampler.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\cubemap_filter.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\cubemap_roughness.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\cube_to_dp.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\luminance_reduce.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\resolve.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\roughness_limiter.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\screen_space_reflection.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\screen_space_reflection_filter.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\screen_space_reflection_scale.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\shadow_reduce.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\specular_merge.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\ssao.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\ssao_blur.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\ssao_minify.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\subsurface_scattering.glsl.gen.h"
Building RD_GLSL header: "servers\rendering\rasterizer_rd\shaders\tonemap.glsl.gen.h"
Compiling ==> platform\windows\key_mapping_windows.cpp
Compiling ==> platform\windows\joypad_windows.cpp
key_mapping_windows.cpp
Compiling ==> platform\windows\windows_terminal_logger.cpp
Compiling ==> platform\windows\vulkan_context_win.cpp
joypad_windows.cpp
....
pe1\type1.c
Compiling ==> thirdparty\freetype\src\type42\type42.c
type1.c
Compiling ==> thirdparty\freetype\src\winfonts\winfnt.c
type42.c
winfnt.c
Compiling ==> thirdparty\freetype\src\sfnt\sfnt.c
Building compilation database compile_commands.json
sfnt.c
progress_finish(["progress_finish"], [])
Linking Static Library ==> modules\freetype\freetype_builtin.windows.tools.64.lib
Linking Static Library ==> core\core.windows.tools.64.lib
Linking Program        ==> bin\godot.windows.tools.64.exe
LINK : warning LNK4044: unrecognized option '/MT16'; ignored
   Creating library bin\godot.windows.tools.64.lib and object bin\godot.windows.tools.64.exp
scons: done building targets.

real    2m16.477s
user    0m0.000s
sys     0m0.015s

[LinuxUserGD]

Does scons use ccache by default? Should make rebuilds faster too than just skipping already compiled files/targets I think.

[Calinou]

Does scons use ccache by default? Should make rebuilds faster too than just skipping already compiled files/targets I think.

No, but our SCons setup is configured to use ccache automatically as long as it's installed. You can run watch ccache -s while building Godot to confirm this.

[suncaller]

Adding another entry point in case it's useful. Incremental build with no change:

Windows 10, i5-6600k CPU, 3.50GHz, 16GB RAM

> scons target=debug -j16 --debug=time

[vcvarsall.bat] Environment initialized for: 'x64'
scons: Reading SConscript files ...
Automatically detected platform: windows
Configuring for Windows: target=debug, bits=default
Found MSVC compiler: amd64
Compiled program architecture will be a 64 bit executable (forcing bits=64).
...
scons: done reading SConscript files.
scons: Building targets ...
[ 99%] progress_finish(["progress_finish"], [])
[100%] Command execution time: progress_finish: 0.001998 seconds
[100%] Command execution time: .: 0.000000 seconds
scons: done building targets.
Total build time: 17.022759 seconds
Total SConscript file execution time: 6.119761 seconds
Total SCons execution time: 10.834999 seconds
Total command execution time: 0.067999 seconds

[Calinou]

Edit: This is now documented at godotengine/godot-docs#5593.

For those on Linux, Pyston is worth a try. It's a JIT-enabled implementation of Python compatible with SCons.

If anyone wants to try this, download a release from https://github.com/pyston/pyston/releases,
extract the portable .tar.gz and run ./pyston -m pip install scons to install SCons within Pyston. Then run the usr/bin/scons script that is located within the folder where you extracted Pyston while in the Godot root folder.

I've also tried Nuitka but it didn't provide any performance improvement after compiling SCons with it. PyPy isn't compatible with SCons and most likely won't be for a while (if ever).

To make calling SCons via Pyston easier, save this script as pyston-scons somewhere in your $PATH:

#!/bin/sh

# This assumes you installed Pyston to $HOME/.local/opt/pyston. Change if necessary.
"$HOME/.local/opt/pyston/usr/bin/scons" "$@"

Then make it executable by running chmod +x /path/to/pyston-scons.

Benchmark

• CPU: Intel Core i7-6700K @ 4.4 GHz
• SSD: Samsung 850 EVO (1 TB)
• OS: Fedora 34

The CPython version used for comparison is 3.9.0 compiled with pyenv/pyenv-installer with optimizations enabled. I'm performing a null build (no files changed between builds) and 1 warmup run before each comparison:

❯ hyperfine -w1 "scons -j10 use_llvm=yes use_lld=yes werror=no tests=no" "~/Downloads/pyston_2.3_portable-v2/usr/bin/scons -j10 use_llvm=yes use_lld=yes werror=no tests=no"
Benchmark #1: scons -j10 use_llvm=yes use_lld=yes werror=no tests=no
  Time (mean ± σ):      7.261 s ±  0.061 s    [User: 7.007 s, System: 0.286 s]
  Range (min … max):    7.193 s …  7.410 s    10 runs
 
Benchmark #2: ~/Downloads/pyston_2.3_portable-v2/usr/bin/scons -j10 use_llvm=yes use_lld=yes werror=no tests=no
  Time (mean ± σ):      5.176 s ±  0.024 s    [User: 4.962 s, System: 0.250 s]
  Range (min … max):    5.132 s …  5.214 s    10 runs
 
Summary
  '~/Downloads/pyston_2.3_portable-v2/usr/bin/scons -j10 use_llvm=yes use_lld=yes werror=no tests=no' ran
    1.40 ± 0.01 times faster than 'scons -j10 use_llvm=yes use_lld=yes werror=no tests=no'

[dmoody256]

I have a scons branch that builds godot with ninja:
https://github.com/dmoody256/scons/tree/ninja_scons_daemon

(venv) Sep.09 04:16 ubuntu[godot]: nproc 
16
(venv) Sep.09 04:16 ubuntu[godot]: python ../scons/scripts/scons.py platform=linuxbsd
scons: Reading SConscript files ...
Enabling ALSA
Enabling PulseAudio
Enabling D-Bus
Enabling udev support
Note: Building a debug binary (which will run slowly). Use `target=release_debug` to build an optimized release binary.
Checking for C header file mntent.h... yes
scons: done reading SConscript files.
scons: Building targets ...
[Initial build] Generating: build.ninja
[Initial build] scons: done building targets.
[Time elapsed: 00:00:05.777]
(venv) Sep.09 04:17 ubuntu[godot]: time ninja 2>&1 > /dev/null

real	5m24.044s
user	64m57.609s
sys	4m58.330s
(venv) Sep.09 04:25 ubuntu[godot]: touch core/core_bind.cpp 
(venv) Sep.09 04:25 ubuntu[godot]: time ninja core/core_bind.linuxbsd.tools.64.o 
[2/2] Compiling core/core_bind.linuxbsd.tools.64.o

real	0m5.172s
user	0m5.294s
sys	0m0.266s
(venv) Sep.09 04:25 ubuntu[godot]: git diff
diff --git a/SConstruct b/SConstruct
index 8feb9e61bb..a51f76dbba 100644
--- a/SConstruct
+++ b/SConstruct
@@ -69,6 +69,11 @@ elif platform_arg == "javascript":
 # want to have to pull in manually.
 # Then we prepend PATH to make it take precedence, while preserving SCons' own entries.
 env_base = Environment(tools=custom_tools)
+
+env_base.SetOption('experimental', ['ninja'])
+env_base['NINJA_DISABLE_AUTO_RUN'] = True
+env_base.Tool('ninja')
+
 env_base.PrependENVPath("PATH", os.getenv("PATH"))
 env_base.PrependENVPath("PKG_CONFIG_PATH", os.getenv("PKG_CONFIG_PATH"))
 if "TERM" in os.environ:  # Used for colored output.

The tool from my scons branch has been localized as an isolated tool for use in godot in #52793

[fire]

@akien-mga Changes made to the build system made scons faster?

[dmoody256]

just wanted to drop a note that ninja tool work I had been working on and testing integration with godot build previously has been merged into SCons master and is ready to be released in SCons 4.4.

[dmoody256]

I just tested godot master with scons master and ninja build worked. However there is still an issue with source code being generated during SConscript reading phase. Source code should be generated during the building phase forming proper scons nodes. This will cause unnecessary rebuilds when using ninja because ninja is only looking mtime so the files will be regenerated because they happen during sconscript reading, and ninja will detect the mtime change. More details outlined in this comment: #52793 (comment)

[akien-mga]

We've done a number of improvements recently which significantly improved the rebuild times, notably:

• SCons: Add fast_unsafe option for faster rebuilds #57806
• SCons: Make compilation database generation optional #57820
• SCons: Default num_jobs to max CPUs minus 1 if not specified #63087
• Add support for single compilation unit builds #77170
• SCons: Disable misbehaving MSVC incremental linking #80482
• SCons: Disable C++ exception handling #80612

For me on a Linux laptop, with the following command: scons -j7 p=linuxbsd dev_build=yes dev_mode=yes linker=mold scu_build=all, I get these metrics:

"Null" rebuild (no change): 4.8s
Minor incremental rebuild (adding empty line in main.cpp): 12.4s

I'm using Pyston to speed up the SCons parsing a bit, and mold to speed up linking. That brings the SCons overhead to around 5s as shown above. The rest is Godot's normal build + link time.

We'll keep looking for ways to improve things further, but I think this issue can be considered resolved.