BLD: build with numpy 2.0.0rc1 (or newer) on Python >= 3.9

[neutrinoceros]

For more context:

• oldest-supported-numpy is superseded by NumPy itself since version 1.25.0
• NumPy 2 just reached ABI stability with the release of version 2.0.0rc1 a couple days ago, and forward compatibility with it requires building against this newest version instead of oldest ones.

It is recommended by NumPy developers to plan releases of packages with Python extensions a some point between 2.0.0rc1 and 2.0.0 (final).

Apologies if I missed something obvious: I didn't have time to test locally so I'm relying on CI to reveal any potential incompatibilities. I'll have more time later today to properly dig into it if needed.

[CLAassistant]

All committers have signed the CLA.

[neutrinoceros]

I was able to validate locally:

• building (on a M2 mac)
• running test/run_all.py

[neutrinoceros]

I see a couple failures in CI which look unrelated to the patch (failing to build wheels for mpi4py), and which seem to have run in a small time window between releasing and yanking of setuptools 69.3.0 and 69.4.0, so it would be my primary suspect. Re-running now would either confirm or allow to reject this suspicion.

[jswhit]

all tests passing - is this ready to merge?

[neutrinoceros]

Absolutely!

[ocefpaf]

@neutrinoceros did you change your local build environment to get dev numpy 2.0? All the CIs here ran with numpy<2 BTW, so the tests did pass, but none really used np2. How did you satisfy cftime with np2? Did you also build that locally? There are no wheels published for that one yet.

[neutrinoceros]

I ran running test/run_all.py with numpy 2.0.0rc1 installed locally and got 0 errors, and I assumed that numpy 2.0.0 final would be available soon enough that adding a special job in CI for it was probably unnecessary.

[ocefpaf]

That's odd b/c you should at least see errors related to cftime not being built against np2, b/c that one isn't available. Anyway, this PR is not bad but I'd like a full CI testing with numpy 2.0. I'm working on it...

[neutrinoceros]

Thanks for pointing this out. I went back and checked my testing env and I was actually using numpy 1.26. Forcing the upgrade, I get import-time errors for cftime indeed.

[ocefpaf]

NP. I'm building cftime now in conda-forge/cftime-feedstock#53. I'll send a PR here to add testing against np2 later today when those artifacts are available in the channel.

[ocefpaf]

See #1319

[wpbonelli]

Hi, is there a timeline for a release supporting np2? I help maintain a project which uses netcdf4-python and we are looking to plan a release of our own. Thanks!

[ocefpaf]

@wpbonelli the current code passed all NP2 without the need for any modifications. In a way, it already supports NP2. I do not recommend to publish stable code with unstable dependency but, people are doing that all the time, still you don't need to wait on this project if you want to test. You can either build your own wheel with NP2 or use conda-forge's experimental channel in a similar way I did here.

[neutrinoceros]

Releasing binaries built against numpy 2.0.0rc1 is the official recommendation and numpy maintainers are actually waiting for as many dependees to do that before they make the final release.

[wpbonelli]

Thanks @ocefpaf @neutrinoceros, I have done some testing as you say. We hope to release a np2-compatible version shortly as the numpy maintainers recommend doing and this is our last dependency which hasn't made its own release yet. Ideally we want to pin our dependencies so installs will keep working smoothly with pre- and post-2.0 numpy. I just hoped to get an idea for this project's timeline so we can plan around it.

[ocefpaf]

Releasing binaries built against numpy 2.0.0rc1 is the official recommendation and numpy maintainers are actually waiting for as many dependees to do that before they make the final release.

@neutrinoceros just b/c it is the official recommendation it doesn't mean everybody agrees on it and/or it is a good one. On the packaging side it is a nightmare to use an unstable release to build stable Software. Numpy devs don't do that and don't really care about that part. They do want to ensure others take the time to test their unstable Software without a promise that it is stable, otherwise they would release a stable one.

I am part of a packaging community that is in touch with numpy devs on this topic. We are not complaining just for the sake of disagreement and we do value numpy's team efforts. With that said, we need to set better standards for the scientific community than relying on unstable releases for a base package.

[neutrinoceros]

The ABI is actually guaranteed stable at this point, but I'm not an expert in packaging so I assume that I'm just not getting your point.

[ocefpaf]

Many big projects, like Python itself, mention a guaranteed ABI compat and yet, sometimes something passes and breaks everything, this happens more than we would like to admit.

Regardless of stable or not, there is the build infrastructure, maintainers time to update CIs to RCs, than stable, etc... It is not a good practice. Folks are using it b/c:

1. PyPI mixes unstable with stable in a very confusing way, sometimes even when folks do not explicitly passes the option to install dev versions;
2. The unstable side is safe from online criticism b/c their breakage is expected. Those building stable packages from the unstable one aren't.

[neutrinoceros]

Thanks for your insightful reply !