Implemented switch from SCons to Meson to improve Godot's build performance #45093
Conversation
marstaik
requested review from
akien-mga,
AndreaCatania,
BastiaanOlij,
bruvzg,
hpvb,
JFonS,
neikeq,
reduz,
vnen and
a team
as code owners
|
Reserved. |
|
Is this feature https://docs.godotengine.org/en/stable/development/cpp/custom_modules_in_cpp.html already supported? How it would work? |
It almost works exactly the same. There is The docs need to be updated, but there should be tons of examples on how to do it with the existing modules. |
|
So it's still possible specify modules that are outside the godot directory using |
Ah, custom directory isn't done yet, but its pretty trivial. I will work on it. @fire keeps mentioning this to me. |
|
Awesome, thanks! |
|
Would this be doable to: #42875 ? |
The module_db can be extended for additional functions, so it is possible. The |
Chaosus
changed the title
|
I suppose the @reduz will be heavily against this change. Just a note from https://docs.godotengine.org/en/latest/development/compiling/introduction_to_the_buildsystem.html
|
This was discussed on IRC, both @reduz and I are OK with the idea to use Meson, if and only if the conditions that you quoted are fulfilled. Meson ticks some of the boxes (can also run Python scripts), and the rest have to be ensured in this PR and/or in a feature branch if we decide to merge into a branch while contributors keep working on it. If the end result proves to be not only faster, but also has the same or better capabilities to support all our platforms (including cross-compilation for many of them), and with a clean buildsystem setup that is easy to extend and maintain, then yes, this switch could be done. But it needs to give the same or better output, we don't want to have regressions due to a buildsystem change (one exception being WebM support as the build setup for it was unmaintainable and we plan to move video decoding to first-party plugins). |
|
Note: I haven't reviewed any code yet. I did some testing on Linux, here are some initial results and comments, in no specific order. System specs: Configuration stepThe configuration step looks cool and is very fast: (Takes 2 sec.) I haven't looked into the code yet to see how the dependency relationships are tracked, though I noticed that reconfiguring a finalized build with Build tests
Those were not thorough tests as I did not specify the exact same configuration (I don't know how to get the exact same config with Meson yet) and I was using Firefox with a number of open tabs in parallel (and occasionally a Godot 3.2 instance), so some CPU and RAM taken. CPU perf limited at 3.1 GHz for both (80% capacity) as otherwise my CPU overheats and tends to throttles. Then for null builds, Meson+Ninja obviously wins: SCons takes ~9 seconds, Ninja takes ~0 seconds. For small incremental/leaf builds (e.g. adding an empty line to Build flagsTo properly assess this buildsystem change, we need to ensure that we compare things which are comparable, and as such we need to make sure to replicate the build flags used by SCons for the different Currently, SCons defaults to It's important that we start from a setup where the build flags are the same, so that we can be sure that the resulting binaries will be the same / very similar. Our current build flags (to be clear I mean stuff like this: https://github.com/godotengine/godot/blob/master/platform/linuxbsd/detect.py#L89-L111) are used in production and are thus heavily tested. While it might be relevant to review build flags and change some eventually, this should not be done as part of this conversion work, but in future follow-up PRs. They need to be addressed independently of the buildsystem technology, as they're not linked to it (they're linked to compilers, and we're not changing those). For example, the discrepancy in current build flags might explain why a clean build is much slower with Meson than SCons Another gain from ensuring that current SCons and Meson setups use the same flags and config is that it makes reviewing the PR and learning how Meson works easier for existing contributors, as we can compare how existing SCons code was converted to Meson for a given component or platform. We should definitely use this opportunity to re-evaluate some of our build flags and options as well as fix bugs, but this should ideally be done in follow-up PRs to avoid having to review both tooling and workflow changes on one hand, and build configuration and binary size/performance on the other hand. Build namingThe current SCons buildsystem gives each object a suffix that loosely matches the build configuration, which more or less allows to have several build configs concurrently in the same Git clone, and the possibility to do incremental builds for each of them with little conflicts (it's not perfect as some files are still shared, but it works somewhat OK). It's convenient when you build e.g. Mono and non-Mono versions and use both, or The Meson setup seems not to reproduce this, and generates a binary named This will create That's not necessarily a big issue, but that's something worth noting. I also planned to implement something like godotengine/godot-proposals#1416 at some point to have generated files even better identified, and I'm not sure how/whether it should interface with the proposed Meson setup for now. Build accuracyWhile SCons takes a while to parse its config, a big plus of using it is that (unless we mess up in our With separate configure (Meson) and build (Ninja), you need to remember to manually reconfigure your build folder if anything changed that requires it - and there's a number of things which do:
For a fast moving target like Godot's I'm not sure what's the proper way to handle this in the Meson world, but to get back the safety of SCons we'd probably have to tell users to always reconfigure before running Ninja (maybe Meson can be configured so that the Ninja build files are actually configured to force a reconfigure). The configure step is fast (less than 2 s for me on Linux) so that's not a big deal. Not sure how that plays with e.g. VS as a backend though. Cross-compilationBeing able to cross-compile from one platform to another is a core feature of Godot's buildsystem and something we will not compromise on. For example, the official Godot buildsystem compiles everything from Linux: Linux binaries on Ubuntu 14.04, all other binaries on Fedora 32 (Android with the Linux SDK/NDK, macOS and iOS with OSXCross, Windows with MinGW-w64/GCC, JavaScript with Emscripten for Linux, UWP with Visual Studio via WINE).
I'm a bit concerned about this as we lose some of the SCons buildsystem's flexibility around the
That's it for some initial thoughts, will add more as I play more with it and review the code. |
|
agree with @akien-mga . We should be able to manually specify our own flags for debug, release_debug, etc (actually my preference would be -Og for debug and -Os for release). Not sure what upstream defaults are, but we should be able to explicitly define what we would like to see. I imagine it should be be easy to match these 1:1 to what's currently defined in scons. |
In Godot most platforms uses EDIT: You can see the flag used in |
Sure ya, I just mean |
Same thing on Arch Linux with meson 5.10.4 and ninja 1.10.2. EDIT: Looks like setting |
Build tests & Build flagsBy default I have set the current base configuration of meson to However, In terms of build flags, the only flags that are not mirrored exactly are the debug/optimization flags. I am going to argue here that debug level and optimization flags should not be specified be hardcoded into any build files. The meson devs have also advised against using The idea of What is the purpose behind having set optimization and debug flags? The only viable reason I can think of is to ensure package releases for different platforms are optimal and consistent, and use well-defined options. I would argue that using a
You can then quickly configure a build for ci, release, testing, etc, simply by calling I have to say that I spend a lot of time ensuring that the windows and linux platform flags that matter, aside from debug and optimization are precise. Please compare If you still decide against it, there are ways to do it by disabling optimization and debug flags, and setting them all manually, but then you take flexibility away from developers. Also, we can also always add an option outside of Build NamingThe build naming can be easily solved in multiple ways, by combining platform name, tools enabled, bits on the cpu, etc.
etc, etc. Either way not a hard issue to solve :) Build Accuracy
Meson and ninja perfectly track adding new files to the meson.build and will reconfigure themselves when you simply compile. The doc classes confuse me, as you already have to bootstrap godot to generate new document files. At that point, you should also be reconfiguring anyways. I get that they are picked up by scons because it fully configures itself every time, but these xml files are not produced by the buildsystem, and they cannot be tracked yet. A simple solution to that is to make doctool touch a .md5 file, or a timestamp file, and simply have meson listen to changes in that file to reconfigure itself.
This is completely incorrect. When you pull changes and the sources referenced by the meson build file changes, it automatically reconfigures itself. Cross-compilationThe platform flag has not dissappeared, it is still there and does exactly what you would like. I think one of the concepts that people may be missing is that a native build file and cross file allow you to override/specify/add additional parameters to a build. The primary purpose of This makes it so you dont have to ensure your CXX compiler and CC, and OBJC, and OBJC++ paths are correct before you run meson or cmake. For example on windows, I have a personal native build file that looks like this: All I am doing here is making it really easy for myself to specifically use msvc's Also, I specify the platform manually, but its not needed if I load that file on windows anyways. NotesI think it may be best to set up a meeting and communicate how these things can be done effectively and cleanly. If you consider that windows, linux, linux-cross, and macos can be implemented so easily, with much cleaner |
Yeah, I think we should have a custom
To my knowledge tools depends on extra checks and info that would make release builds slower and more memory-hungry. |
I get the same error with |
Build notesWe are only mapping the compiler flags that should have nothing to do with cpp-defines necessary for code building. WindowsOptimization/Debug table
Notes
LinuxBSDOptimization/Debug table
Notes
ConclusionAs you can see from the tables, all of the flags have direct equivalents controllable by users in meson, without needing to add them in manually - the other platforms are also the same way. Optimization and Debug can already be controlled by the builtin flags, and allows users to customize optimization level even more, either by using the builtin options (which meson synchronizes to behave the same across different platforms (MSVC, gcc, Intel, llvm-clang, etc)) or specifying them manually/through a build file if they want to save a build state for themselves. Meson already does everything scons does better, more consistently, and with less thinking/overhead on our end. |
Finished all modules on except for mono and webm. Windows compilation with MSVC (LLVM-mingw kind of) Linux compilation with GCC Linux cross compile to windows with gcc-mingw-64
|
Taking a quick glance at this and I do like how it turned out. I have zero experience with Meson but it didn't seem hard to understand what's going own in the build files. Still, there are some things I'm still in doubt
Which seems to add a compiler flag directly. Won't this be a problem when using a different compiler (like MSVC?).
If those are things you already planned to do later, that's great. I'm just mentioning since I didn't see any comment about those. I still think it's a great step up. Thanks for putting effort into this change. |
|
I've worked a bit on the javascript platform, and got a build (no ZIP bundling yet) at this branch: With the following option: Many issues to fix:
|
| @@ -108,6 +108,7 @@ logs/ | |||
| [Rr]elease/ | |||
| x64/ | |||
| build/ | |||
| build*/ | |||
There was a problem hiding this comment.
This adds modules/mono/build_scripts/ to the ignore list. I'm going to rename it to modules/mono/meson_scripts, but adding gitignore entries like is to ask for trouble IMO.
There was a problem hiding this comment.
If you start this with a slash, Git will only ignore it in the top source directory. So /build* would match a top level build-linux but not modules/mono/build_scripts.
|
The mono module port to Meson is mostly done: https://github.com/neikeq/godot/commits/mono-meson Since empty builds are very slow with MSBuild I had to make Meson track the input files of the C# projects to only run MSBuild when there are changes, otherwise it would kill all the speed benefits. I also had to drop wildcards in C# projects as Meson doesn't support that. Overall it works pretty well in the end. A few issues that need to be addressed:
|
This is one of those things that we have considered adding every now and then, but since there has not been a strong need for it it has not been implemented (limited resources and all that). |
|
Is superseded by #51153. |
|
Superseded by #51153. Thanks for the contribution nonetheless, it's been a very helpful base so far 🙂 |
Proposal: godotengine/godot-proposals#1797
Meson build for Godot
This is the initial implementation of the meson build system for godot.
Features
Intro
How is meson different from scons? Scons is a Turing-complete build system built around python. It both controls configuration and building.
Meson, on the other hand, is a buildfile/makefile/ninja/vcproj generator.
It assembles the appropriate files needed to run ninja, Visual Studio, make, and delegates the responsibility of building to system appropriate tooling.
Changes
The port to meson was an opportunity to not only move to a newer more supported build system, but also served as an opportunity to really clean up the build.
Thirdparty libraries are now built into their own static libraries, instead of being directly injected into godot's build files.
Dependencies are now tracked appropriately.
Build system cleanup has occured, centralizing/cleaning up as many
#definesas possible.A ModuleDb file has been created, which holds all enabled modules, dependency information, and additional information.
This can also lead to even more flexibility in the future.
This has set up the future for potential dynamic linking for development speed. Once the main libraries such as core, scene, editor are isolated, we can begin
dllspec'ing code and move towards dynamic linking.Build Instructions
Head over and download meson on your system. You will also need python3.
Generally, from the project root, you can initiate a build structure by typing:
meson ./build, substitutingbuildwith your desired folder. Then, from your build folder, executemeson compile.You can change build types by specifying
-Dbuildtype=[plain,debug,debugoptimized,release]. The current default build isdebugoptimized.All of the available options can be found in
meson_options.txt.I recommend reading through https://mesonbuild.com/SimpleStart.html for more information.
Here is a quick example block:
Windows
For now, you will need to run meson from an
Developer Command Prompt(MSVC) to be able to use the msvc compilers.There are plans to build in msvc detection for meson.
Due to a bug in meson adding
/ZIas a default option for debug, launching the editor/binary on windows debug is excruciatingly slow. I have worked around this for now by making theplainbuildtype intodebug, so just useplainfor now. This should be fixed in meson master soon.Example:
meson ./build -Dbuildtype=plainLinux to Windows Cross Compile
There is a cross-file specified in
cross/calledlinux-mingw-win64. Assuming you have mingw64 installed, you simply have to specify the cross file in your meson configuration step.Example:
meson ./build --cross-file cross/linux-mingw-wing64 -Dbuildtype=debugThings left to do...
monoandwebmsetup