Take installed packages into account when upgrading another package

[nlhkabu]

In #7744 we have decided to warn users when pip creates a dependency conflict when upgrading another package.

This issue is a proposal to change this behavior. This proposal is supported by user research, a summary of which will shortly be published on pip's docs.

What's the problem this feature will solve?
Currently, when a user asks pip to upgrade a package, it does this without considering what other packages are already installed. This can mean that pip can cause a dependency conflict.

This scenario (used in our user research) summarises the problem well:

Imagine you have package tea and coffee with the following dependencies:

tea 1.0.0 - depends on water<1.12
tea 2.0.0 - depends on water>=1.12
coffee 1.0.0 - depends on water<1.12
coffee 2.0.0 - depends on water>=1.12

You have the following packages installed:

tea 1.0.0
coffee 1.0.0
water 1.11.0

You ask pip to upgrade tea.  What should pip do?

If pip upgrades tea to 2.0.0, water needs to be upgraded as well, creating a conflict with coffee...

Describe the solution you'd like
Pip should:

• Detect whether or not a conflict may occur, based on the packages that are already installed
• If a conflict is detected, pip should not install anything, and show an error message to the user

This reflects the most popular response to our research (71.7% of respondents) who said that pip should "Install nothing. Show an error explaining that the upgrade would cause incompatibilities."

Additional context
Example error message (happy to iterate on this as needed!)

ERROR: pip cannot upgrade tea, as this would introduce a dependency conflict:
  you have water 1.11.0 installed
  you have coffee 1.0.0 installed, requires water<1.12
  tea 2.0.0 requires water>=1.12

[pradyunsg]

Awesome. Looks like a supermajority of the folks we've asked in the survey about "should pip take installed packages into account" responded that we should. We'd likely want to do this at a later date, sometime in 2021.

IF @pypa/pip-committers (specifically, @pfmoore or @uranusjr) want to try to make this happen sooner, I'm not opposed. ;)

[junpuf]

@nlhkabu

It looks like I didn't catch the bus when the user survey was on going. But I also agree with "Install nothing. Show an error explaining that the upgrade would cause incompatibilities.". But I do have one question regarding this.

Will we be just getting a warning that has ERROR text in it? Or are we getting a non-zero exit code as well?

[uranusjr]

I’d expect the command to exit with non-zero since the message says cannot.

[miguelbalmeida]

Just adding my +1 here in favor of the same option as the 71.7% in your survey. I prefer that pip does not install anything and exits (with non-zero exit code) if proceeding would create an inconsistent environment.

@nlhkabu , +1 on your suggested error message also.

[brianou7]

Like @miguelbalmeida, I also support the users proposal: "Install nothing. Show an error explaining that the upgrade would cause incompatibilities."

@pypa/pip-committers please don't let forget this proposal. Thanks!

[pfmoore]

A PR implementing this would, of course, be welcome. But be aware, it's not a simple change (that's why nothing has been done on it yet!)

[pradyunsg]

I don't think this would be something that we could jump to a PR for directly! That said, we very much would appreciate having someone come and champion this. :)

This is a multi-faceted change, with a bunch of interesting design considerations to make in addition to trying to actually implement the logical change here. Work on this would very much be welcome here, and please do chat with us if you're interested in driving this!

[notatallshaw]

FYI, a simple workaround to get OPs behavior is something like this:

pip freeze --quiet --exclude-editable > current-env.txt
pip install <upgrade commands> --constraint current-env.txt

[leorochael]

FYI, a simple workaround to get OPs behavior is something like this:

pip freeze --quiet --exclude-editable > current-env.txt
pip install <upgrade commands> --constraint current-env.txt

Or, in a single line (if your shell permits, the line below works on bash):

pip install <upgrade commands> --constraint <( pip freeze --quiet --exclude-editable )

Though it must be said that this workaround prevents installation of valid package combinations. For instance, it prevents the installation of pydantic==1.10.13 after having installed fastapi==0.104.0.

That's because this workarounds tells pip to assume the currently installed versions as immutable, rather than checking the actual constraints required by the currently installed packages.

[notatallshaw]

Though it must be said that this workaround prevents installation of valid package combinations. For instance, it prevents the installation of pydantic==1.10.13 after having installed fastapi==0.104.0.

That's because this workarounds tells pip to assume the currently installed versions as immutable, rather than checking the actual constraints required by the currently installed packages.

Yes, this takes into account what is installed, not what you asked to install. For the latter you need a higher level package manager than Pip, there are fortunately a growing number of them.

Side note: I actually roll my own as I find all of them more complex for basic stuff than I would like. It's a simple script that rebuilds a vritual evironment when you change a requirements file.