BUG: link check fails to detect sysroot on PPC if build_platform sysroot >2.12

[h-vetinari]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

Many maintainers in conda-forge try to pay attention to the output of the link check, and fix eventual issues. Some feedstocks even turn any failures for overlinking/overdepending to errors; for example, matplotlib is such a feedstock.

With the long-coming switch from cos6 -> cos7 (and thus bumping the glibc/sysroot version from 2.12 to 2.17), we got rid of the track-feature used to weigh down newer sysroots (that we neede vis-à-vis the 2.12 baseline) in conda-forge/linux-sysroot-feedstock#65.

This had the curious effect that conda-forge/matplotlib-feedstock#390 first ran with the following build environment:

    sysroot_linux-64:                          2.12-he073ed8_17           conda-forge
    sysroot_linux-ppc64le:                     2.17-h23d7e6c_14           conda-forge

leading to a clean link-check

   INFO (matplotlib-base,lib/python3.9/site-packages/matplotlib/_image.cpython-39-powerpc64le-linux-gnu.so): Needed DSO lib/libstdc++.so.6 found in conda-forge/linux-ppc64le::libstdcxx-ng==14.1.0=h262982c_0
   INFO (matplotlib-base,lib/python3.9/site-packages/matplotlib/_image.cpython-39-powerpc64le-linux-gnu.so): Needed DSO powerpc64le-conda-linux-gnu/sysroot/lib64/libm.so.6 found in CDT/compiler package conda-forge/noarch::sysroot_linux-ppc64le==2.17=h23d7e6c_14
   INFO (matplotlib-base,lib/python3.9/site-packages/matplotlib/_image.cpython-39-powerpc64le-linux-gnu.so): Needed DSO lib/libgcc_s.so.1 found in conda-forge/linux-ppc64le::libgcc-ng==14.1.0=h31e42bb_0
   INFO (matplotlib-base,lib/python3.9/site-packages/matplotlib/_image.cpython-39-powerpc64le-linux-gnu.so): Needed DSO powerpc64le-conda-linux-gnu/sysroot/lib64/libc.so.6 found in CDT/compiler package conda-forge/noarch::sysroot_linux-ppc64le==2.17=h23d7e6c_14
   INFO (matplotlib-base,lib/python3.9/site-packages/matplotlib/_image.cpython-39-powerpc64le-linux-gnu.so): Needed DSO powerpc64le-conda-linux-gnu/sysroot/lib64/ld64.so.2 found in CDT/compiler package conda-forge/noarch::sysroot_linux-ppc64le==2.17=h23d7e6c_14
                                                                                                                                                                  ^^^^^^^^^
                                                                                                                                                                  see here

while the CI run on main (after conda-forge/linux-sysroot-feedstock#65) pulled in the following

    sysroot_linux-64:                          2.17-h4a8ded7_16           conda-forge
    sysroot_linux-ppc64le:                     2.17-h23d7e6c_16           conda-forge

and produced

   INFO (matplotlib-base,lib/python3.9/site-packages/matplotlib/_image.cpython-39-powerpc64le-linux-gnu.so): Needed DSO lib/libstdc++.so.6 found in conda-forge/linux-ppc64le::libstdcxx-ng==14.1.0=h262982c_0
   INFO (matplotlib-base,lib/python3.9/site-packages/matplotlib/_image.cpython-39-powerpc64le-linux-gnu.so): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/libm.so.6 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h4a8ded7_16
   INFO (matplotlib-base,lib/python3.9/site-packages/matplotlib/_image.cpython-39-powerpc64le-linux-gnu.so): Needed DSO lib/libgcc_s.so.1 found in conda-forge/linux-ppc64le::libgcc-ng==14.1.0=h31e42bb_0
   INFO (matplotlib-base,lib/python3.9/site-packages/matplotlib/_image.cpython-39-powerpc64le-linux-gnu.so): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/libc.so.6 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h4a8ded7_16
  ERROR (matplotlib-base,lib/python3.9/site-packages/matplotlib/_image.cpython-39-powerpc64le-linux-gnu.so): $RPATH/ld64.so.2 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?

To sum up my understanding:

• This only happens with ppc64le, not with aarch64 (which is cross-compiled in exactly the same way)
• This doesn't affect the entirety of the sysroot contents, only ld64.so.2 it seems
• However, the existence/location of ld64.so.2 in sysroot_linux-ppc64le hasn't changed from _14 to _16
• Moreover it doesn't trigger if the sysroot of the build environment is 2.12...?!

Xref: conda-forge/linux-sysroot-feedstock#67

Conda Info

conda-forge CI

Conda Config

conda-forge CI

Conda list

conda-forge CI

Additional Context

No response

[h-vetinari]

In terms of work-arounds (short of removing error_overlinking: true and living with the warnings):

overlinking_ignore_patterns: specific ❌

      overlinking_ignore_patterns:  # [ppc64le]
        - $RPATH/ld64.so.2          # [ppc64le]

overlinking_ignore_patterns: wildcard ✅

      overlinking_ignore_patterns:  # [ppc64le]
        - "**"                      # [ppc64le]

missing_dso_whitelist: specific ✅

      missing_dso_whitelist:    # [ppc64le]
        - $RPATH/ld64.so.2      # [ppc64le]

So it looks like there's some inconsistencies hiding in overlinking_ignore_patterns as well (at least w.r.t. the less obscure missing_dso_whitelist).

[isuruf]

It's not only ld64.so.2 that's misidentified. There's Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/libc.so.6 which is the wrong sysroot to look at.

Looking at it more

Failing builds have

INFO: sysroot: '$BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/' files: '['usr/share/zoneinfo', 'usr/share/locale/zh_TW/LC_MESSAGES/libc.mo', 'usr/share/locale/zh_CN/LC_MESSAGES/libc.mo', 'usr/share/locale/vi/LC_MESSAGES/libc.mo']'
INFO: sysroot: '$BUILD_PREFIX/powerpc64le-conda-linux-gnu/sysroot/' files: '['usr/share/zoneinfo', 'usr/share/locale/zh_TW/LC_MESSAGES/libc.mo', 'usr/share/locale/zh_CN/LC_MESSAGES/libc.mo', 'usr/share/locale/vi/LC_MESSAGES/libc.mo']'

while previously passing builds had

INFO: sysroot: '$BUILD_PREFIXpowerpc64le-conda-linux-gnu/sysroot/' files: '['usr/share/zoneinfo/zone1970.tab', 'usr/share/zoneinfo/zone.tab', 'usr/share/zoneinfo/tzdata.zi', 'usr/share/zoneinfo/right/Zulu']'
INFO: sysroot: '$BUILD_PREFIX/x86_64-conda-linux-gnu/sysroot/' files: '['usr/share/zoneinfo/zone.tab', 'usr/share/zoneinfo/tzdata.zi', 'usr/share/zoneinfo/right/Zulu', 'usr/share/zoneinfo/right/WET']'

i.e. order matters.

This reminded me of a core call where someone (@mbargull ? @beckermr ?) mentioned that the number of files in the sysroot is taken in for ordering. See

conda-build/conda_build/post.py

Lines 1364 to 1365 in 0c136f5

	# Sort the sysroots by the number of files in them so things can assume that
	# the first sysroot is more important than others.

[h-vetinari]

I tested #5404 in matplotlib, and the sysroot is found correctly with it:

   INFO (matplotlib-base,lib/python3.9/site-packages/matplotlib/_image.cpython-39-powerpc64le-linux-gnu.so): Needed DSO lib/libstdc++.so.6 found in conda-forge/linux-ppc64le::libstdcxx-ng==14.1.0=h262982c_0
   INFO (matplotlib-base,lib/python3.9/site-packages/matplotlib/_image.cpython-39-powerpc64le-linux-gnu.so): Needed DSO powerpc64le-conda-linux-gnu/sysroot/lib64/libm.so.6 found in CDT/compiler package conda-forge/noarch::sysroot_linux-ppc64le==2.17=h23d7e6c_16
   INFO (matplotlib-base,lib/python3.9/site-packages/matplotlib/_image.cpython-39-powerpc64le-linux-gnu.so): Needed DSO lib/libgcc_s.so.1 found in conda-forge/linux-ppc64le::libgcc-ng==14.1.0=h31e42bb_0
   INFO (matplotlib-base,lib/python3.9/site-packages/matplotlib/_image.cpython-39-powerpc64le-linux-gnu.so): Needed DSO powerpc64le-conda-linux-gnu/sysroot/lib64/libc.so.6 found in CDT/compiler package conda-forge/noarch::sysroot_linux-ppc64le==2.17=h23d7e6c_16
   INFO (matplotlib-base,lib/python3.9/site-packages/matplotlib/_image.cpython-39-powerpc64le-linux-gnu.so): Needed DSO powerpc64le-conda-linux-gnu/sysroot/lib64/ld64.so.2 found in CDT/compiler package conda-forge/noarch::sysroot_linux-ppc64le==2.17=h23d7e6c_16

Thanks a lot Isuru! 🙏

Given the importance of the sysroot¹, I think this would be worth considering to still squeeze into 24.7 (.0 or .1), even though the hour is late. 😅
CC @beeankha @kenodegard

Footnotes

1. this gets misdetected in basically every recipe, even though it only fails if error_overlinking: true ↩