Lockfile updates

[trexfeathers]

🚀 Pull Request

Description

• All lockfiles updated (from [iris.ci] environment lockfiles auto-update #4967).
• Changes to a test based on changes in Cartopy v0.21.0:
  • The specific change is Cartopy adopting a new default Transverse Mercator projection - changed in SciTools/cartopy@fcb784d, which was planned since SciTools/cartopy@8860a81. Read those for more context.
  • Our test was written to alert us to any changes in behaviour, rather than to enforce a 'correct' answer. @pp-mo, @bjlittle and I have reviewed the changed behaviour and accepted it by changing the expected test result.
  • The test 'area' has been also moved West to better demonstrate the intended masking behaviour, as the original area was showing very few masks since the Cartopy change.
• NetCDF4 !=1.6.1 pin to avoid segfaults.

TODO:

• What's New
• Minimum Cartopy pin (>=0.21)
• Diagnose intermittent GHA worker crashes

---

Consult Iris pull request check list

[trexfeathers]

The intermittent HDF5 errors and GHA worker crashes appear associated with the same tests - those that read NetCDF files. I'm certain they are dependency related as I have simultaneously re-run another GHA run with our existing lockfiles (run 3060581803) and those all passed.

My guess is either the netCDF4 or NumPy changes.

I can't reproduce locally.

[trexfeathers]

NetCDF4 confirmed, it seems. See also:

• Very frequent segfaults with the new netCDF4=1.6.1 Unidata/netcdf4-python#1192
• New netCDF4=1.6.1 (most probably) causing a fairly large number of Segmentation Faults conda-forge/netcdf4-feedstock#141
• TST: CI instability (Windows + h5py) yt-project/yt#4128

[bjlittle]

@trexfeathers Brilliant detective work 🕵️‍♂️

A couple of super minor format changes suggested (to align with existing formatting), otherwise let's get this banked 👍

[tinyendian]

Hi @trexfeathers and @bjlittle, I had come across the test failures in test_rotate_winds.py that were addressed in this pull request while working on issue #4972.

If I am not mistaken, the code in iris.analysis.cartography._transform_distance_vectors_tolerance_mask essentially estimates the matrix norm ||A|| of a rotation matrix A that represents the actions of function _transform_distance_vectors, with

||A|| = inf/sup{ ||Ax|| for all ||x|| = 1}

where ||x|| is the magnitude of vector x, to find and mask cases with ||A|| < 0.999 or ||A|| > 1.001 where vector magnitudes are stretched or shrunken beyond a threshold of 0.1%.

At the moment, only two vector directions are tested, x=(1,0) and x=(0,1), which doesn't seem to capture the matrix norm sufficiently to identify all cases that fall outside the threshold.

The situation improves when one adds additional directions using something like

    ones = np.ones(x.shape)
    mask = np.logical_and(ones,ones)
    for angle in np.linspace(0, 0.5*np.pi, num=4):
        u_t, v_t = _transform_distance_vectors(np.cos(angle)*ones, np.sin(angle)*ones, ds, dx2, dy2)
        mag_sq = u_t**2 + v_t**2
        mask = np.logical_and(
            mask,
            np.isclose(mag_sq, ones, atol=2e-3),
        )
    mask = np.logical_not(mask)

For the failing unit test, unmasked vectors are now very close to the 0.1% threshold. The downside is that it is evidently a bit more expensive to compute, and the number and range of angles are free parameters - 4 angles between 0 and 90 degrees work in this case, but it's not immediately clear that this will suffice in others.

Not sure how critical it is for rotated wind vector magnitudes to stay close to the stretching threshold, but just in case you are interested in looking at that.

[trexfeathers]

Thanks very much @tinyendian, it's good to have this information here; this is a topic that the team struggle to understand fully. We're happy with the situation as it is currently, but if we ever need to come back we can use the wisdom you've provided here 🙂