architecture-specific binaries are preferred on macOS when universal2 should always be selected

[glyph]

Description

If pip sees both x86_64 and universal2 wheels on an x86_64 machine, it will prefer to install x86_64 wheels, creating a partially-broken virtual environment that cannot be used to create universal2 (i.e. "broadly usable") redistributables.

Expected behavior

I would expect that pip would prefer to install universal2 artifacts when available, so that when building an app with e.g. py2app in a virtual environment created with pip, I'd be able to create a universal2 application even if I don't have an Apple Silicon mac handy.

This is particularly an issue because package maintainers who want to be friendly to older versions of pip that don't support universal2 properly are creating architecture-specific binaries, and the existence of those creates a worse and more error-prone experience for newer, more up-to-date pip users.

pip version

22.3

Python version

3.11.0

OS

macOS

How to Reproduce

arch -arch x86_64 ./asdf/bin/python3 -m pip install cryptography

Output

Downloading http://127.0.0.1:3141/root/pypi/%2Bf/3fc/26e22840b7732/cryptography-38.0.1-cp36-abi3-macosx_10_10_x86_64.whl (2.8 MB)

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[albertosottile]

This is actually exacerbated by the impossibility of manually installing universal2 wheels when platform-specific wheels are also available (see #5453).

[uranusjr]

I would expect that pip would prefer to install universal2 artifacts when available

If we do that, how can the x86_64 and arm64 wheels ever be selected?

[glyph]

If we do that, how can the x86_64 and arm64 wheels ever be selected?

A command line option that would select the more specific native architecture? I'd suggest --prefer-single-architecture, which could perhaps even translate to other platforms if arm64 windows or android-x86 ever gets big enough.

Normally I'd try to avoid suggesting nerdy new options, but wanting architecture-specific binaries seems like a pretty nerdy optimization thing to want; while I can't see zero use-cases for it, I am pretty comfortable saying that wanting this is niche.

To be clear, this is not to say that x86_64 should not be selected if no other options are available; this is just about what to do by default if pip can see both.

[uranusjr]

If we add a new option, we can also (somehow) use that for pip to select a universal wheel, and this becomes a choice between different defaults. I’m not arguing for either to be correct, although selecting the more narrow choice by default seems to make sense to me, because those wheels exist and why not—if the optimisation is considered too nerdy, I’d want to argue PyPI should not accept those architecture-specific wheels at all. Not that I want to make such a proposal, and I suspect “pertty nerdy” may not be how many people would characterise this. Perhaps this needs a discussion in a wider scope. Or we could skip that altogether, let pip keep the current default behaviour, and simply introduce a way (new flag or otherwise) for pip to select universal wheels.

[rgommers]

Imho the proposal in the issue description makes a really poor trade-off. Building installable applications with py2app is a very niche use case, compared to the vast majority of users who just want a binary that runs on their system. Significantly increasing download size and disk space for all those users does not seem reasonable.

I'll also note that universal2 is a weird thing to begin with, and is at best a decent stopgap during Apple's architecture transition. We also don't bundle 32/64-bit binaries together, or x86-64/aarch64 for Linux. By now we're 2 years into the transition, we have M1 CI and arm64 wheels for most projects. So it's time to phase universal2 out completely, rather than start preferring it. py2app & co should do their own homework and learn how to fuse thin wheels.

If we add a new option, we can also (somehow) use that for pip to select a universal wheel,

That would be a useful option for those few packagers who indeed need universal2 wheels.

[pradyunsg]

x-ref pypa/packaging#381

[justvanrossum]

I'm sad to see this issue is still unresolved. What is needed is a mechanism that says "prefer universal2 over x86 if both are available". The --platform option doesn't really cut it, as it seems to require an exact tag, as well has having the restriction of requiring --no-deps.

[glyph]

it's time to phase universal2 out completely, rather than start preferring it.

You're at least 4 years too early with this take.

Apple will be continuing regular support for Intel hardware until 2028, and "vintage" support for 2 years after that. While I am generally in favor of aggressively encouraging users to upgrade to more recent OSes and Pythons and generally keep their software house in order with security updates, EOL hardware is a different ball of wax, as a decision to de-support it prematurely is a potential motivation for the production of e-waste.

py2app & co should do their own homework and learn how to fuse thin wheels

I agree that this should be a higher priority for py2app than for pip and I've written my own automation to do so: https://github.com/glyph/encrust/

But much as this is not going to be a supported and encouraged workflow, people do copy virtualenvs between machines, as sharing a working Python installation is an incredibly annoying and difficult task. Creating environments which are broken by default (and not in a nice, legible way like an error about it having been copied from a different machine, but instead throwing obscure linker errors) is pretty hostile to beginners struggling with this difficulty (especially when copying environments around like this works a lot of the time as a practical matter on other OSes).

[danielbair]

Same here for dortania/OpenCore-Legacy-Patcher#1099 and jawah/charset_normalizer#277.
The solution I found is from #5453 (comment).

pip3 uninstall charset-normalizer
pip3 download --only-binary=:all: --platform=macosx_10_9_universal2 --platform=macosx_11_0_universal2 --no-cache-dir charset-normalizer
pip3 install --upgrade --only-binary=:all: --no-cache-dir --no-index --find-links ./ charset-normalizer

[glyph]

Thanks for this workaround @danielbair !

[rgommers]

You're at least 4 years too early with this take.

Obviously I did not mean "remove support for x86-64". 4 years is much too short for that probably; I recently dealt with a bug report about a regression on PowerPC macOS. People like their old hardware:)

x86-64 is and continues to be supported the standard way: by uploading regular x86_64 wheels.

people do copy virtualenvs between machines

If it were free to support such copying, sure why not. But it's far from free, and such copying will also have other failure modes. So by itself it isn't too convincing. We also wouldn't consider it reasonable to start combining 32/64-bit Windows wheels. And people confusing themselves with 32-bit Python on 64-bit Windows is probably a lot more common an issue than copying Python envs between two macOS machines with different architectures.