pip doesn't handle PEP600 compatible manylinux wheels when --platform is specified.

[Clockwork-Muse]

Description

When pip is asked to install or download a manylinux wheel and the --platform option is specified, it fails to find either tag aliases, or to download an "earlier" manylinux tag, if there was no exact match.

The manylinux project defines the following legacy aliases in PEP 600:

• manylinux1_x86_64 is now an alias for manylinux_2_5_x86_64
• manylinux1_i686 is now an alias for manylinux_2_5_i686
• manylinux2010_x86_64 is now an alias for manylinux_2_12_x86_64
• manylinux2010_i686 is now an alias for manylinux_2_12_i686
• manylinux2014_x86_64 is now an alias for manylinux_2_17_x86_64
• manylinux2014_i686 is now an alias for manylinux_2_17_i686
• manylinux2014_aarch64 is now an alias for manylinux_2_17_aarch64
• manylinux2014_armv7l is now an alias for manylinux_2_17_armv7l
• manylinux2014_ppc64 is now an alias for manylinux_2_17_ppc64
• manylinux2014_ppc64le is now an alias for manylinux_2_17_ppc64le
• manylinux2014_s390x is now an alias for manylinux_2_17_s390x

... these tags should be equivalent to each other.

In addition, manylinux tags are supposed to be forward compatible - if a particular platform is specified, it should find the "latest" tag, if no exact match was found.

Expected behavior

pip downloads a compatible manylinux wheel.

pip version

21.3.1

Python version

3.8.10

OS

Ubuntu 20.04

How to Reproduce

$ python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux2010_x86_64 numpy==1.19.0
$ python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux2014_x86_64 numpy==1.19.0
$ python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux_2_12_x86_64 numpy==1.19.0
$ python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux_2_17_x86_64 numpy==1.19.0
$ python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux_2_24_x86_64 numpy==1.19.0
$ python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux_2_17_x86_64 numpy==1.22.0
$ python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux_2_24_x86_64 numpy==1.22.0

(The problem affects any wheel on pypi, numpy was just chosen as a well known package)

The first five commands should find the same manylinux2010 wheel, but only the first two commands do so.
The last two commands should find the same manylinux_2_17 wheel, but only the first command does so.

Output

Collecting numpy==1.19.0
  Downloading numpy-1.19.0-cp38-cp38-manylinux2010_x86_64.whl (14.6 MB)
     |████████████████████████████████| 14.6 MB 4.5 MB/s 
  Saved /tmp/d/numpy-1.19.0-cp38-cp38-manylinux2010_x86_64.whl
Successfully downloaded numpy
Collecting numpy==1.19.0
  File was already downloaded /tmp/d/numpy-1.19.0-cp38-cp38-manylinux2010_x86_64.whl
Successfully downloaded numpy
ERROR: Could not find a version that satisfies the requirement numpy==1.19.0 (from versions: 1.20.3, 1.21.0rc1, 1.21.0rc2, 1.21.0, 1.21.1, 1.21.2, 1.21.3, 1.21.4, 1.21.5)
ERROR: No matching distribution found for numpy==1.19.0
ERROR: Could not find a version that satisfies the requirement numpy==1.19.0 (from versions: 1.22.0rc1, 1.22.0rc2, 1.22.0rc3, 1.22.0)
ERROR: No matching distribution found for numpy==1.19.0
ERROR: Could not find a version that satisfies the requirement numpy==1.19.0 (from versions: none)
ERROR: No matching distribution found for numpy==1.19.0
Collecting numpy==1.22.0
  Downloading numpy-1.22.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
     |████████████████████████████████| 16.8 MB 166 kB/s 
  Saved /tmp/d/numpy-1.22.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Successfully downloaded numpy
ERROR: Could not find a version that satisfies the requirement numpy==1.22.0 (from versions: none)
ERROR: No matching distribution found for numpy==1.22.0

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[Clockwork-Muse]

Side note: the command is actually being run as part of a deployment tool, pulling from a requirements file, but tricks like downloading the correct packages and using --find-links or explicitly listing the local path in the requirements file still fail with the same error.

[tvalentyn]

I was looking at PEP-600 and also concur that current behavior does not match my understanding of PEP-600.
My usecase: I will be running a package on a Linux platform that uses glibc 2.31. I'd like to download a wheel that will be compatible for this platform, so I'd want to run

python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux_2_31_x86_64 numpy==1.22.0

and expect that the latest compatible wheel is fetched.

[tvalentyn]

cc: @pfmoore

[pradyunsg]

If somone wants to see this addressed, we'd be happy to accept a PR for this (subject to the regular code review process).

The relevant file for this would be https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/compatibility_tags.py, and I imagine that test improvements would be appreciated.

[vanschelven]

My usecase: I will be running a package on a Linux platform that uses glibc 2.31. I'd like to download a wheel that will be compatible for this platform, so I'd want to run [..] and expect that the latest compatible wheel is fetched.

A workaround is to specify all platforms that the target platform supports. Figuring out that list can be achieved using some variant of the below on the target-platform (and ignoring the message about not automatically parsing):

python -m pip debug --verbose | grep cp39-cp39 | awk -F- '{print $3}'

[ssbarnea]

Sorry to post on an old bug, but I find myself in the impossible attempt to download wheels for a specific linux platform but different architecture and apparently this bug prevents me from composing a download command that would work.

python -m pip debug --verbose | grep many|grep py310
  py310-none-manylinux_2_38_aarch64
  py310-none-manylinux_2_37_aarch64
  ...
  py310-none-manylinux_2_17_aarch64
  py310-none-manylinux2014_aarch64

For latest version of pyyaml the only command that will succeed is pip download -vv -d dist/linux/x86_64 --platform=manylinux_2_17_x86_64 --only-binary=:all: --python-version=310 pyyaml==6.0.1

Unless I fully mention the manylinux_2_17_x86_64, I will not be able to download "whatever blend of manylinux for x86_64 is available.

The grep command above makes me believe that pip is trying ~20 names until it finds that one to be acceptable. Still, I see no way I can build the same fallback list when attempting to perform to call download.

To give some insights, I am trying to download full list of dependencies for two different platforms (arm and intel) while avoiding to run under qemu as that makes the process 10x slower, this being the preparatory task for building a dual-arch container image.

Basically I have the impression that --platform argument is so restrictive that it makes itself unpractical to use as it requires a perfect match.

[tim-stephenson]

#12466 It feels tangentially related (but a separate issue with pip download --platform)