Consider discarding < 4 upper-bound on Requires-Python

[charliermarsh]

See: #4021 (comment)

[charliermarsh]

It's awkward but perhaps a pragmatic decision to treat >=3.7 as compatible with >=3.7,<4.

[zanieb]

Thanks for opening a tracking issue.

Here's some relevant discussion:

• please stop pinning to major versions by default / forcing SemVer, especially for Python itself python-poetry/poetry#3747
• fix: Remove '<4' from python_requires in packaging guide pypa/packaging.python.org#850
• The current project's Python requirement (>=3.11,<4.0) is not compatible with numpy requires Python <3.13,>=3.9 python-poetry/poetry#8616

Probably more out there too.

[zanieb]

It's awkward but perhaps a pragmatic decision to treat >=3.7 as compatible with >=3.7,<4.

Seems pretty reasonable until Python 4 is like... a real thing that might happen. If they switch to CalVer <4 is going to be a problem anyway :)

[charliermarsh]

Haha. But if Python 4 is a thing, then users could set <4. I was more thinking we'd throw this out on declared metadata.

[hmc-cs-mdrissi]

A broader thing you could do is ignore all upper bounds for Requires-Python. https://discuss.python.org/t/requires-python-upper-limits/12663 has much longer discussion but while standards say require python works for lower bounds, for upper bounds it's less clear.

edit: The main issue is how will backtracking behave and older versions often do not have requires-python upper bound set even though it's unlikely older version actually does work better.

[charliermarsh]

Henry's comment there is interesting:

It really should have two settings, one for Requires-Python, and one for solving.

In Poetry, the published Requires-Python is coupled to the Requires-Python used for solving.

[zanieb]

cc @henryiii :)

[charliermarsh]

That suggests we shouldn't use Requires-Python but a separate setting for this hah.

[henryiii]

Requires-Python can't be used for upper bounds, it doesn't work properly. The field currently is used to back-solve, and backsolving is the wrong thing to do for an upper bounds. Libraries that have added an upper bound here (numba, numpy, etc) have had to remove it as it provides a worse experience to have the solver just permanently install a really old and broken package instead of the latest broken package. :)

https://iscinumpy.dev/post/bound-version-constraints/ talks about it some, and I've proposed some solutions in https://discuss.python.org/t/requires-python-upper-limits/12663 - though it got a bit stuck because packaging.tags should have a way to compute unions on SpecifierSets to help libraries detect an upper bound reliably, and I don't think we fully settled on a preferred solution.

Only a small handful of libraries actually have a true, known upper bound; libraries like numba and numpy - but those also have binaries, so --only-binary works instead. Most other libraries, including every single library in the world that sets <4, is just wildly sticking something there.

After the next Python is released, old releases may actually have a computable upper bound, but you can't go back and modify old metadata.

[charliermarsh]

What would you suggest we do? We can compute unions on specifier sets.

[henryiii]

For now, ignoring <4 when making a lock file is safe. IMO ignoring upper bounds when making a lock file is safe, as you can't guarantee the upper solution is in fact compatible regardless (for example, https://pypi.org/project/rembg/#data says <3.13 but it's because a dependency states that, and poetry would force them to do that. But when the dependency updates, that fixes the package depending on it, a package should state what they support!), but it's not as surprising to respect it as it is to respect <4.

If you want to have a lock file range, you can default to requires-python, but make sure there's a way to specify it without affecting the public metadata, please!

[dimbleby]

this has come up a few times in poetry, I think python-poetry/poetry#6884 contains the most useful discussion. fwiw I continue to hold the view that I expressed there: that it would be better to get the rules changed - presumably by an updated PEP - than to have solvers guess that folk meant something different than what they wrote.

Incidentally re

for example, https://pypi.org/project/rembg/#data says <3.13 but it's because a dependency states that, and poetry forces them to too.

While I think it's true that poetry's choice to default requires-python as capped has proved somewhat infectious, this particular project is a very curious example to choose to make that point - as rembg does not use poetry.

[henryiii]

Poetry would force this though, and it's just the most recent thing I'd seen with the cap.

Currently, no one knowns what Python 4 will be like, other than it will be "less disruptive" than Python 3, and that it will come after 3.33. So I think it's perfectly fine to ignore "<4" as it only purely incorrect for anyone to use it now. Certainly don't make it infectious.

This Python limit is special, though, unless you can also solve for the Python version, which Poetry, PDM, etc. can't do. If you can't ever fix a solve by using this, what is the point of potentially breaking working solves?

And in general, if any solver starts going back to older versions of a package with an upper cap (which they all do), they not going to arrive a better solve with an older package just because they find one without a cap.

I feel most solvers go for mathematical correctness here when practically backsolving is only useful for lower bounds, and is at best expensive (for perfect metadata) and at worst wrong (for un-editable metadata).

Do keep in mind, though, I work on a total of 0 solvers. :)

[dimbleby]

Poetry would force this though, and it's just the most recent thing I'd seen with the cap.

sure, it's not that you're wrong about poetry - but you are wrong to assume that everything is poetry's fault!

I agree that most solvers go for correctness, I agree that it's plausible that if the folk writing the spec had understood the consequences of that then they might have written it differently. But perhaps we disagree about the right resolution to that:

• you're willing to just ignore the part of the rules that you don't like
• I say: if the spec is wrong then first get the spec fixed
  • if everyone agrees that the spec is wrong then that should be easy, if not then perhaps it shouldn't be done...

Maybe it would be interesting to have a solver take the other path and see what happens.

[charliermarsh]

This Python limit is special, though, unless you can also solve for the Python version, which Poetry, PDM, etc. can't do.

What does this mean @henryiii?

[henryiii]

assume that everything is poetry's fault!

Sorry, didn't mean everything is Poetry's fault! I'll settle for "almost everything". :P

Poetry and PDM solve for a range - that's why this is infectious. Say you have one dependency: numpy. But this is during the few weeks where they had >=3.9,<3.13. Now Poetry will force you to set your Requires-Python to >=3.9,<3.13 (or narrower), since it can't find a lockfile that will be declared to support 3.13. However, as a python library, that's wrong, since your library just depends on numpy, and when numpy updates for 3.13 (or drops this silly cap, which they did), then your library now supports Python 3.13. It's only your lockfile that doesn't. So yes, you need to update your lockfile, but you shouldn't have to update your metadata and make a new release of your library, because you should't have been forced to modify your library metadata in the first place.

if the spec is wrong then first get the spec fixed

What would you suggest fixing? I have three options, I don't think we ever picked one though. And IMO it's still pretty squarely on the solvers to fix; an official statement that the upper bound is ignored or disallowed would help, of course. If I don't wan't to put a limit on my usage of Python, a solver should not force me to because my dependency does. I'm fine if it forces me to acknowledge that with tool.something, but not my project metadata.

What does this mean

Just that numpy<2 is not the same as requires-python<3.11. You can select numpy<2, but you can't pick the Python you are running on (in the mentioned tools). So a limit only serves to either cause backsolving on packages to look for a higher or missing upper cap (which is wrong for an upper limit) or to cause the solve to error, which turns a non-zero chance of failure into a guaranteed failure. The NumPy cap might cause a working solve by selecting an older NumPy. If a tool does solve for Python, like conda does, then it is much more like a normal package cap, and could cause a working solve by lowering Python rather than backsolving for older libraries without caps.

Conda lockfiles have a pinned version of Python. Poetry, PDM, etc. have a range of Pythons.

(TL;DR: the solution here may be affected by uv's Python installation plans ;) )

[zanieb]

Regarding updating Requires-Python to be narrower than your dependents see #4071 which proposes only narrowing the lockfile value and #4021 where we require users to narrow the pyproject.toml value. I find narrowing just the lockfile relatively compelling.

I'm also quite intrigued by the idea of "solving" for Python versions but haven't fully grasped what that would look like in practice.

[dimbleby]

What would you suggest fixing?

depends what exactly you think is broken! eg I think I have seen you assert that requires-python was never intended / should not be used for upper bounds at all (apologies if this is misrepresenting you, or you now think something different). In that case, one might propose something like: introduce min-python, deprecate requires-python.

If I don't wan't to put a limit on my usage of Python, a solver should not force me to because my dependency does

While related, I think this is a separate question from whether or not upper bounds and consequent backsolving were a good idea. eg per zanieb the published range and the locked range do not have to be identical, this is something that could - at least in principle - be addressed today without changes to specs.

[henryiii]

Yes, I have said both, though the former is not correct, the designers assumed it could be used for anything, it just turned out that it doesn't "just work" when you design solvers. Adding a cap (remember when it was designed nothing had it, so it was always "added") just causes backsolving to before the cap existed.

requires-python originally supported dropping Python 3.x versions while keeping 2.7, which is why it needed to be a specifier set. Changing it to a new field would be a monumental task compared to just disallowing an upper cap. And if there was a situation like that in the future, it could also be determintal to handicap ourselves now. I have three suggestions in the linked Python discussion, but I don't think the churn of replacing the whole thing is acceptable.

[henryiii]

But anyway, I am curious to see what's chosen here, and eventually will try to restart the Python discussions.

[charliermarsh]

I find narrowing just the lockfile relatively compelling.

I think there's something to this but it doesn't quite work as-is. We include your own project in the lockfile, so that sets an upper limit on the Requires-Python that we could infer -- like, it can't get any wider than your own project's requirement.

But anyway, I am curious to see what's chosen here, and eventually will try to restart the Python discussions.

My guess right now is that we'll ignore upper bounds on declared Requires-Python from dependencies.

[hmc-cs-mdrissi]

I'm also quite intrigued by the idea of "solving" for Python versions but haven't fully grasped what that would look like in practice.

This I take to mean treat python as another dependency and install it. uv pip install python==3.11 for example. And that user could specify,

python>=3.10
numpy>=1.24
...

and then treat requires-python same as adding a dependency constraint on python itself and be able to install/upgrade/downgrade python like any other library. That's what makes conda special. Conda can install python packages, but it can also install non-python packages and even python interpreter itself.

[charliermarsh]

👍 Yeah, "solving" for Python would mean you end up solving for a specific Python version (as opposed to solving for a range, I guess).