Find new build process for Fortran libraries

[mdpiper]

The babelizer uses numpy.distutils to build Fortran libraries. In Python 3.10, numpy.distutils has been deprecated in favor of setuptools, and it will be removed entirely in Python 3.12. While setuptools has incorporated much of numpy.distutils, it's not a complete replacement. Significantly, it will not include Fortran build support.

We need to find a new way to build Fortran libraries through the babelizer.

[mdpiper]

Some recommendations from NumPy on migrating to something new. I'm just going to copy/paste this verbatim.

Migration advice

It is not necessary to migrate immediately - the release date for Python 3.12 is October 2023. It may be beneficial to wait with migrating until there are examples from other projects to follow (see below).

There are several build systems which are good options to migrate to. Assuming you have compiled code in your package (if not, we recommend using Flit) and you want to be using a well-designed, modern and reliable build system, we recommend:

• Meson
• CMake (or scikit-build as an interface to CMake)

If you have modest needs (only simple Cython/C extensions, and perhaps nested setup.py files) and have been happy with numpy.distutils so far, you can also consider switching to setuptools. Note that most functionality of numpy.disutils is unlikely to be ported to setuptools.

[mdpiper]

A blog post on Meson, Fortran, and Cython.

[mcflugen]

Thanks for looking into this @mdpiper! I started down this road a couple months ago trying to get mesas to build on conda-forge. Although, in the end, I got it to build, I found all the options to be rather overwhelming. Even though the use-case is different than what we're trying to do with the babelizer, maybe there is something you can use in the mesas feedstock. I ended up using cmake because I was trying to get it to build with minimal changes to the existing code base but I'm not sure that is necessary for us.

Godspeed.

[mdpiper]

A note for posterity on behavior related to this issue.

Building a babelized Fortran project was failing yesterday. I think the issue was setuptools silently overwriting parts of numpy.distutils (see pypa/setuptools#3532 and pypa/setuptools#3505). Here's the key from the traceback:

File "/private/var/folders/_y/rmt8prks4rgd6wwjxmbssk800000gn/T/pip-build-env-llql5toh/overlay/lib/python3.10/site-packages/numpy/distutils/mingw32ccompiler.py", line 28, in <module>
  from distutils.msvccompiler import get_build_version as get_build_msvc_version
ModuleNotFoundError: No module named 'distutils.msvccompiler'

I fixed the issue by rolling back to setuputils<60.0. However, the problem seems to have resolved on its own overnight. Perhaps a library was updated? I've saved logs from failed and successful builds, so I'll compare them.

I don't think this merits a separate issue, since this issue (i.e., #77) will supersede it. However, I wanted to note it in case I see it again.

[eli-schwartz]

I think the issue was setuptools silently overwriting parts of numpy.distutils

Technically, what happened is that setuptools intentionally overwrites all of distutils, and numpy.distutils modifies some of distutils, and it's all libraries modifying each other all the way down.

Perhaps a library was updated?

setuptools broke API. Setuptools then reverted the breakage.

...

In general, the complexity of this interconnected stack is the reason numpy has deprecated numpy.distutils, and recommends that people pin setuptools < 60.0 in the short term while exploring other options.

[mdpiper]

@eli-schwartz Thank you very much for clearly explaining this. I appreciate your help!

[mdpiper]

🚨 I created an example of building a model--written in Fortran and wrapped in Python with the babelizer--with meson-python. See https://github.com/mdpiper/pymt_heatf. I think it can be used as a prototype for moving the babelizer from numpy.distutils to Meson for Fortran models. (And possibly for C/C++ models, as well!)