Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip-compile fails on packages using poetry #721

Closed
eliasbrange opened this issue Jan 24, 2019 · 20 comments
Closed

pip-compile fails on packages using poetry #721

eliasbrange opened this issue Jan 24, 2019 · 20 comments
Labels
help wanted Request help from the community

Comments

@eliasbrange
Copy link

pip-compile fails if a package uses poetry which is PEP 517 compliant (https://poetry.eustace.io/docs/pyproject/#poetry-and-pep-517).

Simply doing pip install pendulum does not require e.g. poetry to be installed. However, installing it made pip-compile work which I'm not sure we as consumers of the package in question should need to do.

Environment Versions
  1. Alpine 3.8
  2. Python version: 3.7.2
  3. pip version: 19.0.1
  4. pip-tools version: 3.3.1
Steps to replicate
# DOES NOT WORK

FROM python:3.7.2-alpine3.8

RUN pip3 install --upgrade pip==19.0.1 setuptools pip-tools>==3.3.1

RUN echo "pendulum" > req.txt

RUN pip-compile --generate-hashes req.txt


# WORKS

FROM python:3.7.2-alpine3.8

RUN pip3 install --upgrade pip==19.0.1 setuptools pip-tools>==3.3.1

RUN echo "pendulum" > req.txt

RUN pip3 install poetry

RUN pip-compile --generate-hashes req.txt
@vphilippon
Copy link
Member

Hi @eliasbrange, thanks for the report.

I'm unfamiliar with PEP517 (/todo Read that thing!). I would naively wish for pip to handle the mechanic.

One thing to note, is that this specific install seem to work with pip==18.1.

Also, there's an issue on the subject in the pendulum project, about issues with pip>=19.0.1:
python-pendulum/pendulum#335

Additionally, there seem to be some issues related to pip 19.0.1 and poetry compatibility (I don't have quite the time to dig into these at the moment):
https://github.com/sdispater/poetry/issues

So there's the possibility that this is one of the upstreams issue, or that there's something more to do in pip-tools now to initialise some PEP517 mechanic (hunch from the pip._vendor.pep517.wrappers.BackendUnavailable exception raised, as seen when using --verbose)

Any help to untangle this would be greatly appreciated!

@vphilippon vphilippon added the help wanted Request help from the community label Jan 24, 2019
@eliasbrange
Copy link
Author

Trying to investigate a bit further. Noticed that pip-compile succeeds with pytzdata which also uses poetry as a build system: https://github.com/sdispater/pytzdata/blob/master/pyproject.toml

@atugushev
Copy link
Member

It raises here:
https://github.com/pypa/pip/blob/master/src/pip/_vendor/pep517/_in_process.py#L36

with an error:

No module named 'poetry'

For some reason it wants poetry. Keep digging.

@eliasbrange
Copy link
Author

Which is described in pendulums pyproject.toml as per PEP517/PEP518.

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

@atugushev
Copy link
Member

atugushev commented Jan 25, 2019

Right!

It doesn't go here:
https://github.com/pypa/pip/blob/master/src/pip/_internal/operations/prepare.py#L119

because should_isolate is False (because build_isolation is False). So it cannot build requires. I've hardcoded should_isolate=True and pip-compile went fine.

@atugushev
Copy link
Member

The main problem is here

'build_isolation': False

@eliasbrange
Copy link
Author

So that should probably be an optional flag to let consumers choose between installing required packages themselves, or doing isolated builds (which will take forever if there are lots of packages using different build systems).

@atugushev
Copy link
Member

atugushev commented Jan 25, 2019

let's investigate how pip download pendulum decides to build it in isolation mode. Ah it seems pip download builds in isolation mode by default.

@atugushev
Copy link
Member

atugushev commented Jan 25, 2019

So that should probably be an optional flag to let consumers choose between installing required packages themselves, or doing isolated builds (which will take forever if there are lots of packages using different build systems).

I guess so. Build isolation could be set on by default and an option to disable it:

  --no-build-isolation        Disable isolation when building a modern source

@atugushev
Copy link
Member

atugushev commented Jan 25, 2019

Gentle ping to @vphilippon.

What do you think about this? Since pip builds sources in isolation by default, perhaps we should consider to do it as well. But it could be a "breaking change".

@vphilippon
Copy link
Member

I went and read https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support to catch up a bit on the subject of build isolation and what it implies.

It seems pip enables build isolation since its first inclusion (pypa/pip@c08d4cc)
So it seems reasonable to enable it by default in pip-tools too.

As for the "breaking change" nature of this change, I don't mind as it's meant to adapt to the "new reality and expectation" of the python packaging ecosystem.

The Grand Question: Do you think that:

A) It's actually a pip-tools bug we're only catching now, that having isolation disabled most likely caused more pain to users since pip 10.0.0 release, and that enabling it will cause little surprise/impact to users? If so, include in our next feature/bugfix.

B) Enabling isolation by default may have multiple impacts that will require careful attention for a substantial part of the userbase, or could bring odd behavior in pip-tools itself, and thus should be announced strongly? If so, time for a 4.0.0 (which is ok, it's just numbers! Let's take advantage of Semantic Versionning to raise user awareness on this, as we can).

@atugushev @eliasbrange Voice your opinion, you likely have more info on the latest feature of pip than I do right now.
Also pingning @techalchemy and @pradyunsg for input on the impact/risk, if you have time 😄 .

@atugushev
Copy link
Member

atugushev commented Jan 27, 2019

A) I don't think it is a bug, "no isolation" might be chosen for a special reason. Would be appreciated to get a comment from @techalchemy here.

B) I totally agree. Users has to be warned if default behavior suddenly changes.

There is a soft way though. We could introduce --no-build-isolation and --build-isolation flags, where --build-isolation is default.

@vphilippon vphilippon pinned this issue Jan 29, 2019
@pradyunsg
Copy link
Contributor

Seems related to pypa/pip#6164.

I'm inclined to think poetry needs to update its backend since it's not compatible with pip's PEP 517 implementation as things stand.

@pradyunsg
Copy link
Contributor

As for rollout of isolation to end users, @techalchemy can probably explain in better terms. :)

@jkp
Copy link

jkp commented Mar 7, 2019

Just ran into this with a package of mine that uses pyproject.toml to declare some build system requirements. Forcing build isolation as mentioned above fixed the issue.

Understand that a breaking change is problematic - is there a plan to at least add an optional flag to allow users to specify this behaviour if they need it?

@atugushev
Copy link
Member

atugushev commented Mar 7, 2019

Hello @jkp,

I've just prepared a PR #758 for the optional flag you've mentioned. I hope it'll be merged soon.

@atugushev
Copy link
Member

atugushev commented Mar 7, 2019

@vphilippon i think it's time to provide an optional flag to allow enabling/disabling build isolation. "No build isolation" still be the default behavior, but we could change it in a future releases (as a breaking change). WDYT?

@techalchemy
Copy link
Contributor

I am super late responding to this, but I'm not really sure what toggling this does for folks. It's pretty desirable to turn on for installation itself, but can cause issues with resolution (though which issues specifically I couldn't tell you at this point).

We have had a lot of problems making pip-tools play nicely with pip during the resolution of editable and local packages or remote files (which pip-tools does not resolve by default). I recently implemented a complete workaround for this in requirementslib which now fully leverages pep517 with a fallback to setuptools and is extremely effective at resolving these dependencies without issue.

In the specific case of this issue, I am fairly sure you will need build isolation turned on if you want pep517 builders to work for resolution, but that's new territory. Like I explain above I am currently ignoring this question entirely and just invoking pep517 myself. The issues I've run into mainly revolve around pip's temporary directory cleanup during the isolated build process, so by handling it myself I am able to achieve build isolation while bypassing the weird issues. Hope this helps.

@auvipy auvipy unpinned this issue May 3, 2019
@RobbieClarken
Copy link

I was getting the following error when running pip-compile on a project that installed a package from a submodule with a pyproject.toml file (the pyproject.toml was only being used to configure black, not as a replacement for setup.py).

pip._internal.exceptions.InstallationError: Command "/Users/clarkr/.pyenv/versions/3.6.3/bin/python3.6 /Users/clarkr/.pyenv/versions/3.6.3/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/gc/lwr8t4zs14g6y7t401x8n315xkgl13/T/tmpr6_ewj7j" failed with error code 1

Updating requirementslib fixed the issue:

python3 -m pip install -U requirementslib

@energizah
Copy link

I'm inclined to think poetry needs to update its backend since it's not compatible with pip's PEP 517 implementation as things stand.

@pradyunsg Would you mind elaborating on this or possibly filing an issue on Poetry, so we understand what needs to be fixed?

gsong added a commit to Techtonica/techtonica.org that referenced this issue Jan 18, 2020
Add poetry due to issue with poetry, see jazzband/pip-tools#721
gsong added a commit to Techtonica/techtonica.org that referenced this issue Jan 19, 2020
Add poetry due to issue with poetry, see jazzband/pip-tools#721
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Request help from the community
Projects
None yet
Development

No branches or pull requests

8 participants