Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Poetry removes it's own dependencies with install --no-dev when it manages the environment it is installed in #3957

Closed
3 tasks done
gbdlin opened this issue Apr 16, 2021 · 12 comments
Labels
kind/bug Something isn't working as expected

Comments

@gbdlin
Copy link

gbdlin commented Apr 16, 2021

Issue

When you specify a package in the dev-dependencies that is either a dependency of the poetry itself or it depends on one of the poetry dependencies, and you've installed poetry inside an environment that poetry is supposed to manage, poetry will uninstall this mentioned dependency after invoking poetry install --no-dev, making itself unusable in the future.

I'm aware that it is advised to install poetry outside of the environment it should manage, but this is very often seen inside docker images (such example provided in the gist linked above) to save on the resources.

Example output presenting mentioned issue (without -vvv ands):

$ pip install poetry
Requirement already satisfied: pip in /usr/local/lib/python3.9/site-packages (21.0.1)
Collecting poetry
  Downloading poetry-1.1.6-py2.py3-none-any.whl (172 kB)
[ ... 70 irrelevant output lines truncated ... ]
Collecting appdirs<2,>=1.4.3
  Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Installing collected packages: [ ... ] appdirs, [ ... ] poetry
Successfully installed [ ... ] appdirs-1.4.4 [ ... ] poetry-1.1.6 [ ... ]

$ poetry config virtualenvs.create false && poetry install --no-dev

Skipping virtualenv creation, as specified in config file.
Installing dependencies from lock file

Package operations: 0 installs, 0 updates, 1 removal

  • Removing appdirs (1.4.4)
Removing intermediate container af920367a74d

As you can see, appdirs, which is required by the poetry, was uninstalled by poetry. Full log, with the -vvv and the attempt to run poetry again, available here

@gbdlin gbdlin added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Apr 16, 2021
@abn
Copy link
Member

abn commented Apr 17, 2021

This is expected, since you are asking poetry to manage the system site, the same that you have installed poetry into. I suspect here the case is because appdirs is in your project's dependency tree, and you have told poetry to make it so that none of the project dev dependencies are installed. Well, poetry does so, dutifully.

Honestly, creating a virtual environment for the project does not save any meaningful amount of resources and it causes more problems than it solves.

And with upcoming release of poetry you can even create virtual envs without any packages (incl. pip, setuptools).

Alternatively, you can also just do pip install /path/to/project. The caveat here is that you loose any locked versions. But hey, you avoid adding poetry or creating venvs. ;)

@gbdlin
Copy link
Author

gbdlin commented Apr 17, 2021

If this is expected, it's for sure not documented properly. Both in the documentation and when just using poetry install --help, the --no-dev option is described as Do not install the development dependencies., not Get rid of the development dependencies. There is also a 2nd command line options --remove-untracked, from which I would expect to remove some dependencies.

But even if the --no-dev's expected behavior is to remove all dev dependencies, poetry should do one of those:

  • do an exception for its own dependencies if it's being run from the same virtualenv,
  • don't remove anything implicitly when it's being run from the same virtualenv,
  • provide additional command line option that allows skipping the removal of dev dependencies.

@AlexandreDecan
Copy link

AlexandreDecan commented Apr 22, 2021

I ran into a similar issue on Travis where poetry removes virtualenv even if --no-dev is not used (actually, it is "removed" when I execute poetry update --lock).

The issue only arises with Python 3.6 and PyPy (see https://travis-ci.com/github/AlexandreDecan/portion/builds/223762003) but I think it's a consequence of pre-commit relying on virtualenv but I don't know why pre-commit is skipped for Python 3.6.12 since my dependency on it declares pre-commit = {version = "^2.12.1", python = "^3.6.1"} (hence, ^3.6.1 should be satisfied the same way than 3.7, or 3.8, or...). Anyway, that's a different issue ;-)

I agree with @gbdlin that ideally, something should be done about this (esp. since I also did not expect poetry to remove packages unless --remove--untracked is provided).

@lainisourgod
Copy link

@gbdlin I agree that poetry's behaviour is invalid but for now you can avoid the problem by installing poetry in isolated environment: curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python -

@blueyed
Copy link
Contributor

blueyed commented May 18, 2021

Duplicate of #3139? (at least that one is older, and covers removal of virtualenv also)

@gbdlin
Copy link
Author

gbdlin commented May 19, 2021

@blueyed

Duplicate of #3139? (at least that one is older, and covers removal of virtualenv also)

This issue is mentioned by @sontek in #3139 (comment)_ but the response below advises to create a new issue, so here I am.

@sontek maybe raise a new big with a bit more information, and maybe an example with logs we could try to reproduce with.

As a note, it is not recommended that use the same environment poetry is installed in as your project's virtual environment.

Originally posted by @abn in #3139 (comment)

@MasterNayru
Copy link
Contributor

I am just hitting this issue myself now.

It seems like the behaviour that I am after is pretty much for poetry to be treated like a runtime dependency of the project, but NOT added as an actual dependency of any packages built and pushed to a PyPi repo. Is there any clean way to get that behaviour?

I get that the curl url | python install method puts poetry in its own venv and side-steps this issue, but given that poetry is distributed through pip, it seems crazy to me to not support the case where poetry and the project being managed are in the same environment. Is there any reliable way for poetry to establish that it isn't isolated in its own environment and put in some form of exception to stop uninstalling its own dependencies?

@pierresouchay
Copy link

pierresouchay commented Jun 7, 2021

Yes, we have been hitting this issue as well (we call it the "suicide" behaviour)

Could simply the behavior of --no-dev be changed not not uninstall dev-dependencies? (just ensure that main dependencies are installed)

@mwgamble
Copy link

I think a better approach would be to have a poetry clean command or something along those lines, that uninstalls packages that aren't required . This is preferable IMO, rather than tying it to poetry install.

@johncronan
Copy link

johncronan commented Mar 24, 2022

Could simply the behavior of --no-dev be changed not not uninstall dev-dependencies? (just ensure that main dependencies are installed)

I feel like this is the right answer. --no-dev should mean perform operations without regard to dev packages, instead of remove dev packages from the list of what needs to be in the environment.

(I only found this by accident because of something related, but I too use poetry in a Docker image with virtualenvs.create false.)

edited.. Wait, is it impossible? Maybe I'm trying to make a distinction without a difference.

@finswimmer
Copy link
Member

This has change with poetry 1.2.0a2: Poetry wan't remove installed packages unless the --sync option is given.

However, if a dependency will be removed due to a version change e.g. on poetry update the package will be removed of course from the venv.

fin swimmer

Copy link

github-actions bot commented Mar 1, 2024

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 1, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Something isn't working as expected
Projects
None yet
Development

No branches or pull requests