Pre-release not selected when remotely available pre-releases for version specifiers is the only versions that satisfies the version specifier

[notatallshaw]

Description

Currently if Pip selects something like "pandas>2.1.4" it gets the error Could not find a version that satisfies the requirement pandas>2.1.4, but there is a version that would match this 2.2.0rc0, and as I read the documentation:

accept remotely available pre-releases for version specifiers where there is no final or post release that satisfies the version specifier

As it stands there are no final releases that match "2.1.4", but "2.2.0rc1" does match ">2.1.4".

Expected behavior

"2.2.0rc1" is installed

pip version

23.3.2

Python version

3.11

OS

Linux

How to Reproduce

1. Be on a date before Pandas 2.2.0 is installed or use pypi-timemachine to set the date to 2024-01-09
2. Run python -m pip install --dry-run --no-deps --ignore-install "pandas>2.1.4"

Output

ERROR: Could not find a version that satisfies the requirement pandas>2.1.4 (from versions: 0.1, 0.2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0, 0.19.1, 0.19.2, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.21.1, 0.22.0, 0.23.0, 0.23.1, 0.23.2, 0.23.3, 0.23.4, 0.24.0, 0.24.1, 0.24.2, 0.25.0, 0.25.1, 0.25.2, 0.25.3, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.4.0rc0, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.5.0rc0, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 2.0.0rc0, 2.0.0rc1, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.1.0rc0, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.2.0rc0)
ERROR: No matching distribution found for pandas>2.1.4

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

Btw, this behavior is not related to the fact this uses the > operator instead of the >= operator

Testing using pypi-timemachine and testing pandas>=2.1.5 I get the same error:

py -m pip install --dry-run "pandas>=2.1.5" --index-url http://localhost:9999
Looking in indexes: http://localhost:9999
ERROR: Could not find a version that satisfies the requirement pandas>=2.1.5 (from versions: 0.1, 0.2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0, 0.19.1, 0.19.2, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.21.1, 0.22.0, 0.23.0, 0.23.1, 0.23.2, 0.23.3, 0.23.4, 0.24.0, 0.24.1, 0.24.2, 0.25.0, 0.25.1, 0.25.2, 0.25.3, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.4.0rc0, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.5.0rc0, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 2.0.0rc0, 2.0.0rc1, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.1.0rc0, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.2.0rc0)
ERROR: No matching distribution found for pandas>=2.1.5

[webknjaz]

Yes, this was mentioned in some other issues in the past with the agreement that pip should follow what PEP 440 says. There is only one corner case that works — if a project on the PyPI has only pre-releases. Then, it'll select a pre-release.

Oh, and another one is that if you explicitly mention a pre-release version in the dependency specifier, like pip install 'pandas > 2.1.4.dev0', that would allow pre-releases specifically for said project (pandas in this example).

Here's some earlier discussions:

• Follow PEP 440 and install non-dev versions if --pre is not given #7579 (comment)
• pip install is downloading pre-release version without the --pre argument #5503 (comment)

I suggest following up there so there's less duplicates on the tracker.

[webknjaz]

Duplicate of #9121

[notatallshaw]

While the spirit of the two issues you linked is about the way pip fails to follow the specification on prereleaes, neither of them focus on the part of the specification I am quoting here.

Therefore closing this and marking it duplicate is essentially killing discussion on this part of the specification. Which is up to the pip maintainers, I'm not going to fight that, but I am pointing it out because this is a very specific clear case of pip failing the prerelease specification where as those issues are broad and require interpretation of different parts of the spec.

[webknjaz]

@notatallshaw and the third one? We can reopen, if you still think it's not the same. It's just that I've been meaning to look into the issue you're talking about since the fall and I felt like they are talking about the same things if you read deeper into the conversations. Especially, the #9121 one.

[notatallshaw]

I didn't read that one initially sorry, but reading it now it is also not the same issue, because it's specifically talking about when prerelease markers are included in the version specifier provided by the user (or at all the examples given by the user focus on that).

But the line of the spec that I am quoting does not require prerelease markers:

accept remotely available pre-releases for version specifiers where there is no final or post release that satisfies the version specifier

As I read this (and I don't think there's much room for interpretation):

• The user does not need to provide a prerelease marker in the version specifier
• The user does not need to provide a prerelease flag
• The package can have final releases, but they must not be possibilities within the version specifier provided

FYI, I believe Poetry implements this spec, correctly.

After having lots of long discussions on prereleases recently both here on the pip issue tracker and on the Python discussion forum I know that I am unlikely to try and fix this myself, so feel free to mark this as a duplicate of that issue if you really think it's the same.

Personally I would be happier if this was marked as a duplicate of some issue that is more meticulously going through the spec and putting a plan together with how pip could address it, as discussed here: #12471 (comment). But feel free to take what action you think it best in terms of moving towards a solution.

[webknjaz]

@notatallshaw I'm still convinced that all those linked places are talking about the same thing, maybe not calling it out as explicitly and not in as much detail as you do, maybe talking about different aspects of it. But still the same.

Nevertheless, I do agree that the case you're describing needs to be adjusted to adhere to what the standard suggests. I stumbled upon it when I was implementing dependency resolution in another ecosystem (ansible-galaxy CLI) and I was comparing notes with what PEP 440 has and what pip does.

[pfmoore]

Personally, I would like to see a unified issue that covers what pip should be doing, both in terms of spec compliance and in terms of UI, for pre-releases. The work needs to be separated into two parts - where do we want to get to, and how do we get there.

I agree with @webknjaz that having multiple issues all around the general idea of "pre-release handling" is not ideal, but I also think that @notatallshaw has a point that we need to be sure that we capture all of the various points and picking one (or two) existing issues as the "master" one(s) risks failing to do that.

As @notatallshaw pointed out, this is basically the conclusion that #12471 came to, and I still think it's the right approach. In the meantime, I'm going to reopen this issue for now, as I don't think there's any great pressure to close it until we have the "unified" issue to link to. @webknjaz - are you interested in pulling together the various issues into a new, unified one? Or @notatallshaw - I know you're pretty burned out on this whole discussion, but would you be willing to help out with that work?

[notatallshaw]

With regards to prerelease, and the packaging spec in general, I've become convinced the more valuable place to put effort is into having a common test suite for compliance, that any Python package installer could use.

I would therefore be more happy to engage on seeing if we could, without much overhead, add packse to pip's test suite. And then look at adding specific scenarios like this to packse, and being able to mark them as XFAIL for pip for now.

I would be interested if you (pip maintainers) also think that would be valuable and perhaps fit into a "big picture" way to approach this for pip.

[pfmoore]

As I said in #12526 I like that idea. One issue with it specifically in regard to pre-release handling is that the spec allows for the user to explicitly opt in to pre-releases, so there's not necessarily one correct behaviour. We may need to clarify the standard on exactly what pre-release behaviour should be - and packse might be a good starting point to drive that discussion.

I still think we need to have a separate discussion on at least the UI for pre-release handling, though. Unless the standard changes to remove the idea of the user asking for pre-releases, we need some UI, and a blanket "allow pre-releases" option may not be enough - it all depends on how much changes if we fix things to be standards-compliant.