build: add a meson build configuration #1910
Conversation
|
To be clear: the build here does not work. In Linux at least I get a completed build but some functions are missing. Probably not much is needed to get to a basic working state though. |
|
Maybe as a first step this can be brought up to parity with and replace CMake. I'd rather not have three parallel build systems. To replace our autotools setup, we'd need to support all its capabilities and someone would have to commit to maintaining the build system in the medium term. |
If we change build systems at all, I think they should be replaced. In other words, Meson would replace both CMake and Autotools. To answer your question, yes, under the right circumstances.
Apart from this example, why would it be more nice? I have gotten a lot of people seemingly being afraid of Autotools, but I have never understood why. So why?
So the most important things here: What was the difference, and what C flags where you using in each case?
I mean, we can establish run-time detection of CPU's. It is not hard. For x86, we can simply utilize the |
|
I could maintain a Meson build system if a PR was made that such both Autotools and CMake could be removed. Hence, the functionalities of Meson should include:
|
All these things will be doable somehow. I will check what looks like a good way though... I'll see if I can actually get a working build first though :)
From python-flint's perspective there are two primary advantages that would come from Flint using either meson or cmake rather than autotools. The same considerations would apply to many other downstream projects as well though:
Projects that use autotools don't work properly as subprojects because they are opaque from the outside and cannot keep track automatically of when it is necessary to rerun With meson a rebuild can keep track of when there are changes to the meson.build files or anything else or when built files should be removed, when dependencies have been updated to a new version etc. This is like if Not all projects can use e.g. meson as their primary build system because build system tooling creates a bootstrapping problem. For example GMP and MPFR are dependencies of gcc itself. Likewise Python is a dependency of meson so it would be awkward for CPython to use meson as its own build system. A solution to that from python-flint's perspective is that we can add external wrapdb files for GMP and MPFR (other people want this as well mesonbuild/wrapdb#97). In the case of Flint there is no bootstrapping issue though and the ideal solution for python-flint would be if Flint used either meson or cmake. If Flint used meson then it could also track GMP and MPFR as subprojects using the wrap file mechanism. In fact I think if Flint did that then it would not be necessary for python-flint or other downstream projects to do it explicitly because GMP and MPFR could just become recursive subprojects from python-flint's perspective (I haven't tested this). |
Currently I get: I guess some symbols are not being exported correctly. |
|
Fair points. Let me know if you need help interpreting |
Actually this does give a working build I think at least on Linux. I was just linking python-flint against the wrong libflint.so. The fix is that you have to actually install libflint.so after building it! diff --git a/meson.build b/meson.build
index ae7cd95a1..1169a801b 100644
--- a/meson.build
+++ b/meson.build
@@ -28,6 +28,7 @@ install_headers(headers_all, subdir: 'flint')
libflint = library('flint', c_files_all,
dependencies: flint_deps,
include_directories: include_directories('src'),
+ install: true
)
pkg_mod = import('pkgconfig') |
I'm not explicitly setting any flags so just the default ones are being used. This is how the meson build compiles the C files by default: This is how make compiles them The main difference for build time is After fixing it to use For a no-op build i.e. running Probably basic build speed like this is not a significant factor here. What is more important is that it is a more reproducible build each time and that the reproducibility does not come at the cost of slower builds. You would get a big gain in speed in situations where you would otherwise need to use Also you can configure multiple build directories differently like: Now you can have incremental rebuilds in each differently configured build directory without them clobbering each other: With autotools after |
|
Yeah, I really like the rebuilding times you presented. Could you just present the exact building times you get from Meson vs Make for building the whole library (not including the configuration process)? You can go with CFLAGS="-O0" for simplicity. I just want to make sure that building with Meson for one-time builds are not 30% slower or something. |
|
And I just want to make it clear that I will not merge something that does not superset-ish what Autotools currently does. I know this is somewhat of a big task to take on, just so that you are aware of my requirements. Make sure that you exactly know what the Autotools stuff is doing. And, once more, I am here to help you with the interpretation of the Autoconf stuff. |
|
These are timings from a fresh git clone using this PR as it stands. The timings are on a relatively slow Linux machine with 4 cores and build using Ubuntu's stock gcc in both cases. Both builds give a usable This is the timing for a full rebuild in a fresh clone with $ git clone https://github.com/oscarbenjamin/flint.git
$ cd flint/
$ git checkout pr_meson
$ ./bootstrap.sh
$ ./configure
$ time make -j5 > make-output.txt
...
real 5m20.621s
user 18m48.493s
sys 2m8.120sThis is the full rebuild timing with meson+ninja: $ git clone https://github.com/oscarbenjamin/flint.git
$ cd flint
$ git checkout pr_meson
$ ./bootstrap.sh
$ ./configure
$ meson setup build
$ time meson compile -C build
...
real 5m43.528s
user 17m48.237s
sys 2m24.565sIn this run meson is 7% slower than make but I've run these both a few times today and seen a lot of variability. Many times meson seemed faster but it all takes a bit too long for me to just run it many times and average. For the quoted timings I was careful not to use the computer while the build ran although the screen did go to sleep briefly during the meson build so I woke it up. These are warm cache timings for a no-op build like running $ time meson compile -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/oscar/.pyenv/shims/ninja -C /home/oscar/current/active/flint/tmp/flint/build
ninja: Entering directory `/home/oscar/current/active/flint/tmp/flint/build'
ninja: no work to do.
real 0m1.329s
user 0m1.035s
sys 0m0.355s
$ time make
make: Nothing to be done for 'all'.
real 0m6.010s
user 0m5.764s
sys 0m0.242sSo in this case These are timings after $ touch src/gr_mpoly.h
$ time make
CC gr/mpoly.c
CC gr_mpoly/add.c
CC gr_mpoly/combine_like_terms.c
CC gr_mpoly/equal.c
CC gr_mpoly/fit_bits.c
CC gr_mpoly/fit_length.c
CC gr_mpoly/fit_length_fit_bits.c
CC gr_mpoly/fit_length_reset_bits.c
CC gr_mpoly/gen.c
CC gr_mpoly/get_coeff_scalar_fmpz.c
CC gr_mpoly/get_coeff_scalar_ui.c
CC gr_mpoly/init.c
CC gr_mpoly/inlines.c
CC gr_mpoly/is_canonical.c
CC gr_mpoly/mul.c
CC gr_mpoly/mul_johnson.c
CC gr_mpoly/mul_monomial.c
CC gr_mpoly/mul_scalar.c
CC gr_mpoly/neg.c
CC gr_mpoly/push_term.c
CC gr_mpoly/randtest_bits.c
CC gr_mpoly/randtest_bound.c
CC gr_mpoly/set.c
CC gr_mpoly/set_coeff_scalar_fmpz.c
CC gr_mpoly/set_coeff_scalar_ui.c
CC gr_mpoly/set_scalar.c
CC gr_mpoly/sort_terms.c
CC gr_mpoly/sub.c
CC gr_mpoly/write.c
Building libflint.so.20.0.0
real 0m10.885s
user 0m9.677s
sys 0m1.055s
$ touch src/gr_mpoly.h
$ time meson compile -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/oscar/.pyenv/shims/ninja -C /home/oscar/current/active/flint/tmp/flint/build
ninja: Entering directory `/home/oscar/current/active/flint/tmp/flint/build'
[30/30] Linking target libflint.so
real 0m5.194s
user 0m8.241s
sys 0m1.769sIt looks like they both built 30 targets and meson did it in 5 seconds vs 10 seconds for make. These are timings for $ touch src/fmpz/fmpz.c
$ time make
CC fmpz/fmpz.c
Building libflint.so.20.0.0
real 0m6.411s
user 0m5.942s
sys 0m0.467s
$ touch src/fmpz/fmpz.c
$ time meson compile -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/oscar/.pyenv/shims/ninja -C /home/oscar/current/active/flint/tmp/flint/build
ninja: Entering directory `/home/oscar/current/active/flint/tmp/flint/build'
[2/2] Linking target libflint.so
real 0m3.550s
user 0m2.434s
sys 0m1.020sBoth built 2 targets (fmpz.o and libflint.so) and it took 3 seconds for meson vs 6 seconds for make. |
|
These are timings for a full build in a fresh clone but on a faster Mac laptop that has a 12 core M3 CPU. On this computer I have a few local changes meaning that the meson build can get to the end but it fails with a link error, due to some assembly preprocessor defines not being correct. I think that the timings are approximately comparable any way though because linking is only the last step. This is the timing with make: $ git clone https://github.com/oscarbenjamin/flint.git
$ cd flint/
$ git checkout pr_meson
$ ./bootstrap.sh
$ ./configure
$ time make -j13 > make-output.txt
...
make -j13 > make-output.txt 213.18s user 102.57s system 841% cpu 37.527 totalThis is the timing with meson: $ git clone https://github.com/oscarbenjamin/flint.git
$ cd flint/
$ git checkout pr_meson
$ git apply ../mac-fixes.diff # some local fixes...
$ ./bootstrap.sh
$ ./configure
$ meson setup build
$ time meson compile -C build
...
meson compile -C build 189.25s user 86.31s system 710% cpu 38.780 totalIn this case the meson build is about 11% faster than make but the final link step did not complete: The problem is something to with this preprocessor define: flint/src/mpn_extras/mul_basecase.c Lines 275 to 291 in 940102e The diff applied to make that get as far as it did was: diff --git a/meson.build b/meson.build
index 6a73b0dbe..31734a767 100644
--- a/meson.build
+++ b/meson.build
@@ -1,7 +1,10 @@
project('FLINT', 'c',
version: '3.1.0',
license: 'LGPL3+',
- default_options: ['buildtype=release'],
+ default_options: [
+ 'buildtype=release',
+ 'c_std=c11',
+ ],
)
cc = meson.get_compiler('c')
@@ -12,6 +15,14 @@ mpfr_dep = dependency('mpfr', version: '>= 4.1.0')
flint_deps = [gmp_dep, mpfr_dep, m_dep]
+add_project_arguments(
+ '-march=armv8-a',
+ '-DBUILDING_FLINT',
+ '-DFLINT_NOSTDIO',
+ '-DFLINT_NOSTDARG',
+ '-Werror=implicit-function-declaration',
+ language: 'c')
+
subdir('src')
headers_all = []
diff --git a/src/meson.build b/src/meson.build
index 66c0451ab..2a9d0d4fd 100644
--- a/src/meson.build
+++ b/src/meson.build
@@ -1947,15 +1947,15 @@ c_files_all_subdir = [
'fft/mul_truncate_sqrt2.c',
'fft/negmod_2expp1.c',
'fft/normmod_2expp1.c',
- #'fft_small/default_ctx.c',
- #'fft_small/fmpz_poly_mul.c',
- #'fft_small/mpn_helpers.c',
- #'fft_small/mpn_mul.c',
- #'fft_small/mulmod_statisfies_bounds.c',
- #'fft_small/nmod_poly_mul.c',
- #'fft_small/sd_fft.c',
- #'fft_small/sd_fft_ctx.c',
- #'fft_small/sd_ifft.c',
+ 'fft_small/default_ctx.c',
+ 'fft_small/fmpz_poly_mul.c',
+ 'fft_small/mpn_helpers.c',
+ 'fft_small/mpn_mul.c',
+ 'fft_small/mulmod_statisfies_bounds.c',
+ 'fft_small/nmod_poly_mul.c',
+ 'fft_small/sd_fft.c',
+ 'fft_small/sd_fft_ctx.c',
+ 'fft_small/sd_ifft.c',
'fft/split_bits.c',
'fmpq/add.c',
'fmpq/addmul.c',
@@ -5275,7 +5275,6 @@ c_files_all_subdir = [
'qsieve/poly.c',
'qsieve/primes_init.c',
'qsieve/square_root.c',
- 'test/main.c',
'thread_pool/clear.c',
'thread_pool/distribute_work.c',
'thread_pool/find_work.c', |
|
Timings are looking very good. I think we can afford a small drop in a one-time build while we gain a lot in rebuilds! Thank you for looking into this. |
What would be great is if you could just summarise from a high level what it is that configure is doing to modify the sources and also how that affects the generated Makefile. Right now I am (inadvisedly?) trying to make the meson build work after configure has run so that I don't need to replicate its effect on the sources yet. However it seems that configure has encoded some information into the Makefile in relation to fft_small that I don't immediately understand to be able to replicate in meson. |
|
Regarding the wrong assembly linking: Is this with Autotools or is this with Meson? |
With meson. It works fine with autotools. |
|
Oh, I see. If it is with Meson then it should be expected (unless you have gotten that far already). |
I definitely haven't :) |
|
We (= me = Albin) use M4 to make writing assembly a little bit easier. One such things is making the assembler write the correct symbol. MacOS typically prefix their symbols with an underscore, which we test during the configuration. |
|
So if you write |
|
There are a couple of other things that is needed to be able to write assembly, but I will try to write a list of what the whole Autotools process is doing. |
|
Note that I will herein only describe the process for Autotools. Any local File structure
|
Things to consider when replacing
|
|
I don't think I missed anything, but of course I left some (more or less obvious) things unsaid. Let me know if there is something that needs to be explained in more detail. |
In meson it is not possible to simply set different compiler flags on a per file basis but this kind of case where a few files use different flags can be handled by building them into an intermediate static library that is then linked into the end product:
This is all doable. Some kinds of assembly have explicit support in meson but in general you can use custom targets to run any commands that need to be run if needed.
With meson this is taken care of automatically by the build configuration. Something like: It doesn't use environment variables etc except when configuring.
Adding source files automatically is disallowed in meson. There are workarounds but the intention is that files are listed explicitly and this is what enables it to be fast and reproducible. Dependencies among .c and .h files are determined automatically by ninja but the files themselves must be listed explicitly for this to work. We can easily make a Python script that could update this for all but one or two flint modules (I imagine fft_small or mpn_extras would be exceptions).
I won't show the full output but: $ meson configure build-dir
Core properties:
Source dir /Users/enojb/work/dev/flint/tmp/flint
Build dir /Users/enojb/work/dev/flint/tmp/flint/build
Main project options:
Core options Current Value Possible Values Description
-------------- ------------- --------------- -----------
auto_features auto [enabled, disabled, auto] Override value of all 'auto' features
backend ninja [ninja, vs, vs2010, vs2012, vs2013, Backend to use
vs2015, vs2017, vs2019, vs2022, xcode,
none]
buildtype release [plain, debug, debugoptimized, release, Build type to use
minsize, custom]
...Those are built-in options but Flint-specific custom options would get listed there as well.
By default it looks like what I showed above: $ meson compile -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/oscar/.pyenv/shims/ninja -C /home/oscar/current/active/flint/tmp/flint/build
ninja: Entering directory `/home/oscar/current/active/flint/tmp/flint/build'
[30/30] Linking target libflint.soWhen you actually run that it shows a running counter like
It shows at the final link step. For a full build it's something like
This was a meson bug that was fixed specifically because someone was using meson to build Flint:
These things will all be possible somehow. It might be that some things that are handled in the Makefile would need to be a separate Python script or something.
The simple way to do this is just $ meson setup new-build-dir
$ meson configure new-build-dir -Dfoo=bar
$ meson compile -C new-build-dirNote that meson does everything out-of-tree. It only creates/modifies files etc inside the build directory and leaves the source tree itself untouched.
This is just |
I don't like this at all. Seems like a very basic thing that high performance libraries want to utilize. Hopefully we can do it gracefully.
Yeah, don't be too worried about this. I'm down for only doing it manually.
Haha, that's funny! Nice that is was being fixed.
Alright. I just want to make it clear that I would like to avoid users having to call something else than Although I have my worries, I think this all sounds good. I'm looking forward to see the progress for this. |
|
And thank you for looking into this! |
I think all of these things are handled natively by meson's test command:
Testing won't work with the meson.build here yet though because I haven't added any configuration to build the tests. |
|
There is a meson bug about test executables being built by default: A workaround is to have a configure option for building tests: It should be possible to make it so that |
detection/meson.build
Outdated
| cfg_data_internal.set_quoted('LT_OBJDIR', '.libs/', | ||
| description: 'Define to the sub-directory where libtool stores uninstalled libraries.') | ||
| cfg_data_internal.set_quoted('PACKAGE_BUGREPORT', | ||
| 'https://github.com/flintlib/flint/issues/', | ||
| description: 'Define to the address where bug reports for this package should be sent.') | ||
| cfg_data_internal.set_quoted('PACKAGE_NAME', 'FLINT', | ||
| description: 'Define to the full name of this package.') | ||
| cfg_data_internal.set_quoted('PACKAGE_STRING', 'FLINT ' + FLINT_VERSION_FULL, | ||
| description: 'Define to the full name and version of this package.') | ||
| cfg_data_internal.set_quoted('PACKAGE_TARNAME', 'flint', | ||
| description: 'Define to the one symbol short name of this package.') | ||
| cfg_data_internal.set_quoted('PACKAGE_URL', 'https://flintlib.org/', | ||
| description: 'Define to the home page for this package.') | ||
| cfg_data_internal.set_quoted('PACKAGE_VERSION', FLINT_VERSION_FULL, | ||
| description: 'Define to the version of this package.') | ||
|
|
||
| cfg_data_internal.set('STDC_HEADERS', 1, | ||
| description: '''Define to 1 if all of the C89 standard headers exist (not just the ones | ||
| required in a freestanding environment). This macro is provided for | ||
| backward compatibility; new code need not use it.''') |
There was a problem hiding this comment.
You do not need to define these, just to let you know.
There was a problem hiding this comment.
In the initial attempt to get to a free standing build that does not depend on the bootstrap.sh and the autotools ./configure I tried to reproduce config.h and flint-config.h as exactly as possible so that I could diff them to see what defines were missing/different. I got that to a good enough state though so what is needed now is just ensuring that the logic for choosing the actual defined values is correct and. The unneeded defines can be removed in later cleanup.
Also it seems that configure generates a bunch of HAVE_FOO_H defines that are mostly unused in Flint i.e. these are the only ones actually used:
$ git grep 'HAVE_.*_H'
acinclude.m4:dnl FLINT_HAVE_FFT_SMALL_ARM_H
acinclude.m4:AC_DEFUN([FLINT_HAVE_FFT_SMALL_ARM_H],
acinclude.m4:dnl FLINT_HAVE_FFT_SMALL_X86_H
acinclude.m4:AC_DEFUN([FLINT_HAVE_FFT_SMALL_X86_H],
acinclude.m4:AC_REQUIRE([FLINT_HAVE_FFT_SMALL_ARM_H])
acinclude.m4:AC_REQUIRE([FLINT_HAVE_FFT_SMALL_X86_H])
doc/source/history.rst: * Change ``#ifdef FLINT_HAVE_FFT_SMALL`` to ``#if FLINT_HAVE_FFT_SMALL`` (AA).
src/flint.h.in:#if defined(FLINT_HAVE_VA_LIST) && defined(FLINT_HAVE_FILE)
src/fmpz_lll/is_reduced_d.c:#if HAVE_FENV_H
src/fmpz_lll/is_reduced_d.c:#if HAVE_FENV_H
src/fmpz_lll/is_reduced_d_with_removal.c:#if HAVE_FENV_H
src/fmpz_lll/is_reduced_d_with_removal.c:#if HAVE_FENV_H
src/mpn_extras.h:# define FLINT_HAVE_MUL_FUNC(n, m) FLINT_HAVE_MUL_N_FUNC(n)
src/test/t-io.c:#if HAVE_UNISTD_H && FLINT_COVERAGE
src/thread_pool/init.c:#if FLINT_USES_PTHREAD && defined(HAVE_PTHREAD_NP_H)
src/thread_pool/restore_affinity.c:#if FLINT_USES_PTHREAD && defined(HAVE_PTHREAD_NP_H)
src/thread_pool/set_affinity.c:#if FLINT_USES_PTHREAD && defined(HAVE_PTHREAD_NP_H)Do the others like HAVE_MALLOC_H etc still need to be included in config.h or flint-config.h?
Is it that the configure is just supposed to fail if e.g. malloc.h is not available?
There was a problem hiding this comment.
We require the standard headers stdio.h, stdlib.h, stdint.h, string.h, stdarg.h, math.h, float.h and errno.h. We check for this during the configuration in order to be able to debug easier.
We assume that malloc.h exists for Windows and MinGW, but check if it exists in the configuration as well.
Other things are dependent on other things. fenv.h is used if user has it; same goes with pthread_np.h.
There was a problem hiding this comment.
You do not need to generate HAVE_FOO_H for everything, that is. However, I want it to be present in any logs generated, so that it is easier to debug.
|
I am intending to continue with this but just busy with other things at work for the time being. The current state of this is that it can build from a fresh checkout without using any of the autotools stuff. I have added a ./meson_bootstrap.py
./configure
make(This overwrites configure/Makefile so you would need to run The The things that work are: What does not work currently or is not implemented:
I don't think any of these things is particularly difficult if just translating or reusing the existing code. I'm not sure if there are other significant checks or options missing. It just takes a bit of time to translate and test each of the configurations and check what the existing configure logic is doing. |
|
Btw, this will require users to have Python installed. I know that there was a crossover period between Python 2 and Python 3, and there is incompatibilities between them. Does every modern OS distribute Python 3 instead of Python 2 these days? |
|
I would say so. I do not know a single system that doesn't come with Python 3. |
I mean, Ubuntu 20.04 is about 6 years old software-wise, so I don't think we should worry about that as you say. |
|
Does Flint have any policy about support for being able to build Flint on older Linux distros? I would pay more attention to the minimum Python 3 version rather than availability of Python 2. Current minimum Python version for meson is Python 3.7: To have Python 3.6 as the minimum Python version we need meson 0.61.5 as the minimum meson version. To support Python 3.6 as the oldest Python version we can set that as the oldest meson version: diff --git a/_meson_build/meson.build b/_meson_build/meson.build
index 96a32fffc..a6d60fb58 100644
--- a/_meson_build/meson.build
+++ b/_meson_build/meson.build
@@ -1,6 +1,7 @@
project('FLINT', 'c',
version: '3.2.0-dev',
license: 'LGPL3+',
+ meson_version: '>= 0.61.5',
default_options: [
'buildtype=release',
'c_std=c11',Then if you run meson you get warnings about using meson features that are not supported in 0.61.5: $ ./configure
--------------------------------------------------------------------------------
$ meson setup build.meson
--------------------------------------------------------------------------------
The Meson build system
Version: 1.4.0
Source dir: /home/oscar/current/active/flint
Build dir: /home/oscar/current/active/flint/build.meson
Build type: native build
WARNING: Project targets '>= 0.61.5' but uses feature introduced in '1.1': meson.options file. Use meson_options.txt instead
Project name: FLINT
Project version: 3.2.0-dev
C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C linker for the host machine: cc ld.bfd 2.38
Host machine cpu family: x86_64
Host machine cpu: x86_64
Library m found: YES
Found pkg-config: YES (/usr/bin/pkg-config) 0.29.2
Run-time dependency gmp found: YES 6.2.1
Run-time dependency mpfr found: YES 4.1.0
Checking for function "atan2" with dependency -lm: YES
Checking if "mpz_init links" with dependency gmp: links: YES
Checking if "mpn_mul_basecase links" with dependency gmp: links: YES
Checking if "mpn_gcd_11 links" with dependency gmp: links: YES
Checking if "mpn_div_q links" with dependency gmp: links: YES
Checking if "mpn_add_n_sub_n links" with dependency gmp: links: YES
Checking if "mpn_add_nc links" with dependency gmp: links: YES
Checking if "mpn_addlsh1_n links" with dependency gmp: links: YES
Checking if "mpn_addlsh1_n_ip1 links" with dependency gmp: links: NO
Checking if "mpn_addmul_2 links" with dependency gmp: links: YES
Checking if "mpn_modexact_1_odd links" with dependency gmp: links: YES
Checking if "mpn_rsh1add_n links" with dependency gmp: links: YES
Checking if "mpn_rsh1sub_n links" with dependency gmp: links: YES
Checking if "mpn_sub_nc links" with dependency gmp: links: YES
Has header "immintrin.h" : YES
Has header "arm_neon.h" : NO
detection/meson.build:197: WARNING: Project targets '>= 0.61.5' but uses feature introduced in '1.1.0': feature_option.enable_auto_if().
Message: FFT_SMALL: enabled
detection/meson.build:234: WARNING: Project targets '>= 0.61.5' but uses feature introduced in '1.1.0': feature_option.enable_auto_if().
Message: ASSEMBLY: enabled
Header "stdlib.h" has symbol "aligned_alloc" : YES
Has header "alloca.h" : YES
Has header "arm_neon.h" : NO (cached)
Has header "dlfcn.h" : YES
Has header "errno.h" : YES
Has header "fenv.h" : YES
Has header "float.h" : YES
Has header "immintrin.h" : YES (cached)
Has header "inttypes.h" : YES
Has header "malloc.h" : YES
Has header "math.h" : YES
Has header "pthread_np.h" : NO
Has header "stdarg.h" : YES
Has header "stdint.h" : YES
Has header "stdio.h" : YES
Has header "stdlib.h" : YES
Has header "string.h" : YES
Has header "strings.h" : YES
Has header "sys/param.h" : YES
Has header "sys/stat.h" : YES
Has header "sys/types.h" : YES
Has header "unistd.h" : YES
Has header "windows.h" : NO
Configuring config.m4 using configuration
Program m4 found: YES (/usr/bin/m4)
Configuring flint.h using configuration
Configuring flint-config.h using configuration
Configuring config.h using configuration
include/flint/meson.build:45: WARNING: Project targets '>= 0.61.5' but uses feature introduced in '0.64.0': fs.copyfile.
Build targets in project: 180
WARNING: Project specifies a minimum meson_version '>= 0.61.5' but uses features which were added in newer versions:
* 0.64.0: {'fs.copyfile'}
* 1.1: {'meson.options file'}
* 1.1.0: {'feature_option.enable_auto_if()'}
NOTICE: Future-deprecated features used:
* 0.64.0: {'copy arg in configure_file'}
Found ninja-1.10.1 at /home/oscar/.pyenv/shims/ninjaSome of these meson features can easily be avoided but some might be more awkward. For now the question then is what minimum Python version to target and whether specific versions of meson are being shipped in old distros. It is easy to install meson with pip if needed but if distros already have meson/ninja then it would be useful to be able to support the versions they supply. Ubuntu 19.04 is the oldest Ubuntu version to ship with Python 3.7 as the system Python. It looks like it ships with an older version of meson (0.45.1): |
These are the warnings if putting 0.45.1 as the minimum meson version: I don't think it would be hard to avoid using all of those features but it would complicate things a little. I haven't actually tested whether everything would work though. |
No, but there are two things that has to be outweighed: backwards-compatibility and maintainability. We want to be as backwards-compatible as possible, but we also want the process of maintaining to be as easy as possible. Hence, we have to weigh the two different cases and optimize it accordingly. For me, if there is something that makes it much more maintainable which would require a version that is two or less years old, I would vote for using it. I base this off of the latest Ubuntu LTS which at most is two years old. Less notable, it is also important IMO to be modern to make contributions less of a hassle for newcomers. If we rely on 20 year software, newcomers may have a harder time understanding the source code and, in the process, may avoid to contribute. Of course, this could be said for Autotools as well.
That seems fine as 3.7 was released 6 years ago. |
|
What is exactly the problem with the cmake build, expect lack of documentation? cmake nowadays plays nicely with Python, and projects like pyzmq/0mq are switching to cmake after trying meson. |
I don't have any problem with cmake except that I don't personally know it. I think that having Flint use either meson or cmake would solve the original problems that motivated writing this. I think that the meson build here already implements more of the features of the existing autotools build than the cmake build does. The main thing missing both from here and also from the cmake build is running config.guess to get the CPU and using that to set compiler flags etc. Otherwise I think the meson build here is just missing a few features like An alternative to having this be the standard build system for Flint is that it can be added to the meson patch database. Then it would still be possible for downstream projects to use meson as the build system for Flint if there is a need to do so. |
|
maybe @minrk can comment on the choice of cmake for pyzmq |
The problem is that we (more specifically myself) do not want to maintain multiple different building systems for a single project. We have CMake, but that is only for Windows, and I will not maintain it for other systems. For instance, CMake does not check some things it would need to check since it assumes a Windows build, and it cannot build the assembly code. So the solution is to replace all building systems in FLINT with a single building system. |
|
I think either CMake via scikit-build-core or meson-py is a fine choice, you definitely don't need both, and both are pretty equivalent capability-wise. CMake is much older and more mature, so there's both a lot more baggage and a lot more precedence/documentation, which I think makes it easier to find out how to do things with CMake. I initially tried to use meson-py, but struggled to implement the pyzmq logic of "maybe bundle libzmq." libzmq having its own cmake build system made it easier to integrate it into pyzmq's own as well (meson can technically try to do this, too). Once I was able to figure out how to do what I wanted in CMake, I think the same solution could have worked in meson, but it was definitely easier for me to iterate and figure it out in CMake. I had no experience with meson and very little experience with CMake when I did this. One problem I did encounter was that the configure stage of meson-py was incapable of informing the project that it is building a source distribution, which means it goes through the whole configure stage for building a wheel just to build an sdist, and then does it again to build the wheel from the sdist when using standard repo->sdist->wheel workflow. In my case, that actually meant sdist builds actually failed because unused dependencies were missing and I couldn't find a way to ignore them. scikit-build-core has much simpler, more Pythonic, and more controllable sdist behavior. Meson is a lot more restrictive and I got a fair amount of purity-beats-practicality feeling from the project when looking up how to do things, which I personally found off-putting. Meson is a much nicer, more modern system and language, especially if your build and dependencies are straightforward. |
Note that meson-python and meson are not the same thing. Flint is not a Python library and it does not need a Python PEP 517 build backend like meson-python or scikit-build-core. Rather it needs a general purpose build system for non-Python projects like meson or cmake. It is easy to get confused between these because meson-python obviously uses meson and likewise scikit-build-core uses CMake but they are not the same thing. On the other hand python-flint is a Python library that wraps Flint and uses meson-python. In that situation it is much nicer if Flint can be built with either meson or CMake rather than autotools so that it can be incorporated as a subproject in python-flint's meson build. I can't speak for CMake or scikit-build-core but if considering the quality of documentation or maturity it is important to distinguish meson from meson-python again. The meson project's first release was in 2016 whereas meson-python's first release was 2021. Lots of non-Python projects use meson and lots of projects that don't use meson can still be built with meson. Anyone using meson-python (rather than meson) to build Python projects is an early adopter at this stage though.
This is not relevant to Flint or meson (rather than meson-python and python-flint). I can see the same when building an sdist for python-flint but I don't consider it to be a significant problem. The relevant command for Flint would be |
|
Thanks, I don't actually know what flint is, so I jotted down my experience from pyzmq. I think the real deciding factor should be what do maintainers like the look of, and, if you need another reason, follow what neighboring projects in your community seem to be using (e.g. if python-flint uses meson-python, using meson makes perfect sense, especially if there is contributor overlap). I don't have a technical argument in favor of one or the other. I will say the scope and maturity of docs/examples/Q&A point was about cmake vs meson and not scikit-build-core vs meson-python, both of which are similarly young projects. |
This is incomplete but I am putting it here as a talking point:
Would there be any interest in flint using meson as a build system or as an alternative experimental build system?
For downstream usecases it would be nicer if Flint used something like meson rather than autotools et al. For python-flint for example if flint used either cmake or meson then it could be incorporated as a meson subproject after flintlib/python-flint#129 is merged.
From my limited testing it seems faster to build Flint with meson+ninja rather than configure+make. Currently though the meson configuration here does not build on all platforms and it is probably compiling things differently so any timing comparison is not meaningful at this stage.
Another potential advantage of using meson could be using its (experimental) SIMD module which facilitates using runtime detection for things like AVX:
https://mesonbuild.com/Simd-module.html
Runtime detection of CPU capabilities is what would be needed for python-flint to leverage AVX because it is not possible for Python wheel tags to be more fine-grained than e.g.
x86_64.The meson configuration code shown here is incomplete because it does not handle options and things and also it is deliberately written in the simplest possible way.
To test building this with meson first install meson and ninja (e.g.
pip install meson ninja) and then:You can then install with
Arguments like
--prefixshould be passed tomeson setup.A usable implementation of building with meson would need to work without the bootstrap/configure steps. I cannot personally turn the changes in this PR into a complete replacement of configure etc but I could turn it into a usable experimental alternative.
Another possibility is that I could submit the same files to meson's wrapdb database so that other projects can build flint using meson without the meson build files being part of the flint source tree:
https://mesonbuild.com/Wrapdb-projects.html