Various package-index filtering flags do not affect the environment markers

[pradyunsg]

This is a common pattern that I'm seeing in pip's issue tracker; so filing an issue to (a) trigger a discussion of how to improve documentation / output / errors to better deal with the mismatch in user expectations vs behaviour here and (b) consolidate those issues.

Basically, the fundamental problem here is that pip has various options that affect the wheels that pip will consider when triaging things (--platform <platform>, --python-version <python_version>, --implementation <implementation>, --abi <abi>) but those do not affect the marker evaluation environment for dependencies.

This results in a subtle failure mode: pip install [options] package will use a wheel for package that has a (hypothetical) Python version 9.22 but when evaluating environment markers (https://packaging.python.org/en/latest/specifications/dependency-specifiers/#environment-markers), that would still use the environment for the current Python interpreter.

It is not possible to compute the environment markers based on the values passed in by the user via the CLI.

[pradyunsg]

#10050 (comment) has the relevant piece from that issue.

[pradyunsg]

I think the right move here is doing what pex has done, and taking a complete environment specification including environment markers; and not inferring anything from the interpreter when that's done. It'll be a slightly disruptive change, like any other change, but we'll be in a better spot in terms of this.

Thoughts on this?

[pfmoore]

Agreed, this sounds better. In fact, I'd like to go even further and completely isolate environment discovery - not just marker values, but sysconfig locations, and anything else that we introspect from the Python interpreter. That would allow us to manage any sort of environment, even "artificial" ones constructed from information supplied independently of the running interpreter.

I'm thinking of this in support of --python, or --target/"install scheme" types of option.

I'm happy to limit the change to markers as an incremental step, if that's easier to achieve, but I'd like to have our long-term goal be to have full isolation of "interpreter introspection".

[pradyunsg]

I really don't want to allow the user to specify arbitrary paths with schemes (this was extensively discussed during installer's API/CLI design discussions) because I'm wary of what hacky layouts people might expect to be "supported" if we allow that.

I am OK with exposing basically everything else, since misuse/abuse of them seems less likely/problematic to me.

[pfmoore]

I wasn't suggesting that paths get exposed directly as user options. Simply that internally, we collect everything that is introspected from the interpreter in one place. Then, flags for things like --platform can modify that internal structure. There won't be flags to alter the paths, but existing means of configuring paths, like --target or --user, can be modified to simply change the "interpreter config" structure, rather than having effects across the code base. It's not a big deal, though.

I absolutely agree, we don't want user options to individually specify paths. That way lies chaos...

[pradyunsg]

Oh, that wasn't clear. 😅

We're in complete agreement here. :)

[sbidoul]

I'm copying @uranusjr's UX proposal from #9981 (comment) here:

pip install --marker="os_name='nt'" --marker="platform_release='...'"

I assume that whenever one --marker is provided, none of the other markers usually provided by the platform will be made available to the resolver.
I suppose it would also make sense to rely only on explicit markers whenever any of --platform, --implementation or --python-version is used.
Installation or build would fail if any specifier uses a marker that is not explicitly provided by the user.

[ichard26]

If it's feasible (I haven't checked), it may be worth it to emit a warning when a marker is encountered when a package index filtering option is given.

[jsirois]

It is not possible to compute the environment markers based on the values passed in by the user via the CLI.

@pradyunsg although it's true you can't compute them all, you can compute the ones that seem to be used the large majority of the time in the wild.

Pex has had (has) its own legacy --platform tag, and it's a string with it's own format that includes the 4 components of Pip's --platform, --python-version, --implementation, --abi. This (abbreviated) --platform idea was misguided and in early 2022 I added --complete-platform which is a json blob containing the full environment marker dict and full compatibility tag list for the foreign platform to fully characterize it for resolution purposes. That said, despite --platform being misguided, Pex tries to never break backward compatibility; so I came up with this to continue to support the abbreviated --platform, but better than Pip does natively with the 4 options:
https://github.com/pex-tool/pex/blob/27c2db2bf26039bef41323c964bc4e0317a7b4f5/pex/pep_508.py#L18-L130

That derives as many environment markers as it can from the 4 platform values and leaves the rest None. Since most environment markers in the wild are python_version, this tends to work. When it does not there is other code that handles the None environment value accesses and provides a useful error message; something like:

Failed to resolve for abbreviated platform cp311-cp311-macosx_10_9_x86_64.
Resolve requires evaluation of unknown environment marker: 'platform_release' does not exist in evaluation environment.

Here the user passed --platform macosx_10.9_x86_64-cp-311-cp311 to Pex, which translates into --platform macosx_10_9_x86_64 --python-version 311 --implementation cp --abi cp311 for Pip.

All this is done using minimal runtime patches to off the shelf Pips from 20.3.4 to 24.1.1.

To underscore though, pex --platform ... / pip download --platform ... --python-version ... --implementation ... --abi ... were not well thought out and something like --complete-platform is the way to go from experience. Many users have shipped AWS and Google cloud lambda function zips from their Macs using --complete-platform for a few years now. If Pip can deprecate and drop those 4 options eventually, that seems sane.