GH-40376: [Python] Update for NumPy 2.0 ABI change in PyArray_Descr->elsize

[jorisvandenbossche]

Rationale for this change

NumPy 2.0 is changing some ABI, see the issue description and numpy/numpy#25946 for more details.

The changes here should make our code compatible both with current numpy 1.x and future numpy 2.x

• GitHub Issue: [Python] NumPy 2 ABI changes (elsize) #40376

[github-actions]

⚠️ GitHub issue #40376 has been automatically assigned in GitHub to PR creator.

[jorisvandenbossche]

@seberg we are also using the c_metadata field, which was removed as well.

arrow/python/pyarrow/src/arrow/python/arrow_to_pandas.cc

Lines 538 to 543 in bdd04c0

	void SetDatetimeUnit(NPY_DATETIMEUNIT unit) {
	PyAcquireGIL lock;
	auto date_dtype = reinterpret_cast<PyArray_DatetimeDTypeMetaData*>(
	PyArray_DESCR(reinterpret_cast<PyArrayObject*>(block_arr_.obj()))->c_metadata);
	date_dtype->meta.base = unit;
	}

[jorisvandenbossche]

Should use PyDataType_C_METADATA for that? (with a similar backport for building with 1.x as for elsize)

[seberg]

Should use PyDataType_C_METADATA for that? (with a similar backport for building with 1.x as for elsize)

Yes, exactly, thanks!

[jorisvandenbossche]

And the same for fields ;)

[jorisvandenbossche]

@github-actions crossbow submit -g python

[github-actions]

Revision: 27f15ab

Submitted crossbow builds: ursacomputing/crossbow @ actions-d5247ffd0e

Task	Status
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.5.0
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-python
test-debian-11-python-3-amd64
test-debian-11-python-3-i386
test-fedora-39-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3

[seberg]

Thanks, the changes look good to me from the NumPy side. Maybe we should add a way to get datetime metadata more directly some time (and if fields/names would be real functions we would have more freedom to modify it, but that isn't a downstream code change now at least :)).

[kou]

@github-actions crossbow submit -g wheel

[kou]

It seems that wheel-manylinux failures and wheel-windows failures are related to this.

[jorisvandenbossche]

Yes, those are related, but it's because it is not yet picking up the latest nightly numpy wheel to build against (the changes in this PR require numpy changes of a few days ago).
How do you ensure the docker image doesn't use a cached layer?

[jorisvandenbossche]

@github-actions crossbow submit wheel-windows-cp312-amd64

[github-actions]

Revision: 74caa26

Submitted crossbow builds: ursacomputing/crossbow @ actions-e37c72b24e

Task	Status
wheel-windows-cp312-amd64

[jorisvandenbossche]

OK, my last commit adding some extra prints to our cmake set up at least confirms that in this case it is using an older numpy:

-- Found NumPy version: 2.0.0.dev0+git20240203.5c34c87
-- NumPy include dir: C:/Python312/Lib/site-packages/numpy/_core/include

(from beginning of February, which matches with the time when those images were last updated #39622)

Will then try again to force rebuilding them here in this PR (it's not clear to me how that can be done generally)

[jorisvandenbossche]

@github-actions crossbow submit wheel-windows-cp312-amd64

[github-actions]

Revision: 49df2c5

Submitted crossbow builds: ursacomputing/crossbow @ actions-d87170315f

Task	Status
wheel-windows-cp312-amd64

[jorisvandenbossche]

@github-actions crossbow submit wheel-windows-cp312-amd64

[github-actions]

Revision: 0d97bf8

Submitted crossbow builds: ursacomputing/crossbow @ actions-7909cb5614

Task	Status
wheel-windows-cp312-amd64

[jorisvandenbossche]

OK, my last commit did succeed in ensuring we weren't accidentally using the cached images, and now it was building with against numpy 2.0.0b1 as shown by cmake, and now the windows build did succeed! (well, still red but now that's the known test failure on windows, which will be fixed with rebasing)

So the actual code changes are all good here. Now I only need to figure out how to ensure we rebuild the cached images properly (instead of just not using the cache as I did in the last commit, which I don't want to merge to main)

[jorisvandenbossche]

@github-actions crossbow submit wheel-windows-cp312-amd64

[github-actions]

Revision: c3e18d1

Submitted crossbow builds: ursacomputing/crossbow @ actions-5ccdad511a

Task	Status
wheel-windows-cp312-amd64

[jorisvandenbossche]

@github-actions crossbow submit test-conda-python-3.10-pandas-nightly test-conda-python-3.11-pandas-upstream_devel

[github-actions]

Revision: c3e18d1

Submitted crossbow builds: ursacomputing/crossbow @ actions-5525cfe7b2

Task	Status
test-conda-python-3.10-pandas-nightly
test-conda-python-3.11-pandas-upstream_devel

[kou]

+1

[jorisvandenbossche]

Remaining failures are now actual (and known) test failures (the pandas nightly is being fixed by #40429)

[seberg]

Thanks for working on this, I wish rebuilding from scratch was less tedious.

[jorisvandenbossche]

The tediousness was mostly because of our own building infrastructure, not because of numpy ;) Thanks for the help!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit a421314.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.