Poetry build using wrong virtualenv when installed with pipx?

[alexifm]

System Info

• OS version and name: macos 10.15.6
• Poetry version: 1.1.0
• Relevant pyproject.toml file contents:

[tool.poetry]
name = "infima"
version = "0.0.1"
readme = "README.md"
packages = []
build = "build.py"

...


[build-system]
requires = ["poetry_core>=1.0.0", "setuptools", "Cython>=0.29", "numpy>=1.18"]
build-backend = "poetry.core.masonry.api"

Setup

Installed 1.1.0 with pipx:

$ pipx list
venvs are in /Users/alexifm/.local/pipx/venvs
apps are exposed on your $PATH at /Users/alexifm/.local/bin
   package poetry 1.1.0, Python 3.8.5
    - poetry

Have a clean virtualenv with pyenv and Python 3.8.5:

$ pyenv virtualenv 3.8.5 test
$ pyenv shell test

What happens

Building a wheel throws an error saying numpy isn't available

$ poetry build -vvv -f wheel
Using virtualenv: /Users/alexifm/.pyenv/versions/3.7.7/envs/test
Building infima (0.31.4)
  - Building wheel

...

A setup.py file already exists. Using it.
Traceback (most recent call last):
  File "/Users/alexifm/Projects/pyinfima/setup.py", line 109, in <module>
    from build import *
  File "/Users/alexifm/Projects/pyinfima/build.py", line 4, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'
  Stack trace:

...

  1  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py:149 in _run_build_command
      147│ 
      148│     def _run_build_command(self, setup):
    → 149│         subprocess.check_call(
      150│             [
      151│                 self.executable.as_posix(),

  CalledProcessError

  Command '['/Users/alexifm/.local/pipx/venvs/poetry/bin/python', '/Users/alexifm/Projects/pyinfima/setup.py', 'build', '-b', '/Users/alexifm/Projects/pyinfima/build']' returned non-zero exit status 1.

  at /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py:364 in check_call
       360│     if retcode:
       361│         cmd = kwargs.get("args")
       362│         if cmd is None:
       363│             cmd = popenargs[0]
    →  364│         raise CalledProcessError(retcode, cmd)
       365│     return 0
       366│ 
       367│ 
       368│ def check_output(*popenargs, timeout=None, **kwargs):

The logs say that poetry has detected the virtualenv and will use it:

Using virtualenv: /Users/alexifm/.pyenv/versions/3.7.7/envs/test

But the called process uses the isolated pipx environment, which of course doesn't have numpy

/Users/alexifm/.local/pipx/venvs/poetry/bin/python

What's expected to happen

• I would expect that poetry would use the proper venv /Users/alexifm/.pyenv/versions/3.7.7/envs/test
• I also expected that the build system requirements would be installed.

Maybe I'm using Poetry wrong and it shouldn't be used in a clean environment but run as part of a development process in a worn-in environment where poetry install has been run.

That said, I still don't see why the pipx environment is being used.

[abn]

Related issue is #2789

The python executable detected seems to be wrong as you have indicated. It should use the project venv.

[abn]

@alexifm can you try the fix at #3056 please?

[alexifm]

Looks good. It's using the correct venv:

CalledProcessError

  Command '['/Users/alexifm/.pyenv/versions/3.8.5/envs/infima_test/bin/python', '/Users/alexifm/Projects/pyinfima/setup.py', 'build', '-b', '/Users/alexifm/Projects/pyinfima/build']' returned non-zero exit status 1.

I take it that I poetry build should be run after poetry install? We have switched to using pip wheel . in CI pipelines instead which works for us and seems like it's the correct thing to do. (no errors after running poetry install)

[abn]

Yes, we do not handle build-system requirements automatically at the moment, and that is intentional. #2789 is open to discuss this. Build scripts will be better supported going forward, right now it is till a hack. The first step will be to, ensure that the PEP 517 requirements get respected (python-poetry/poetry-core#58), second we need to get the build happening in isolation, maybe using the pep517 library (#2983) etc.

We have switched to using pip wheel . in CI pipelines instead which works for us and seems like it's the correct thing to do. (no errors after running poetry install)

For projects with build scripts that do not require poetry to publish the wheel this is fine and probably easier since you do not want to install dev dependencies just for a wheel build.

But this situation will get better in the next minor release.

[alexifm]

Thanks for that reply. I really appreciate it. Makes a lot more sense now. I barely understood the build systems before and was rather unsure about what exactly we had done to get Cython working with Poetry for our project. But I'm now understanding 517 a lot more and how Poetry does things. Sounds like a lot of good things in store as the project evolves.

(And because of not totally understanding it all, I got a little irate trying to debug stuff yesterday and blamed it on Poetry in a call session with a colleague. Sorry about that. We really do appreciate the project and how much it's helped our package development.)

[abn]

And because of not totally understanding it all, I got a little irate trying to debug stuff yesterday and blamed it on Poetry in a call session with a colleague.

Hah. That happens. A good tool is meant to be invisible, when it is not people get mad/frustrated. In any case, happpy that you are finding Poetry benifitial. Keep the bug reports (and even contributions) coming in. If you have questions regarding the build system etc, the discord server might be a good place to start.

[Yevgnen]

I got a similar error when using Poetry (version 1.2.0a2)

Command '['/home/user/.direnv/python-3.8.11/bin/python', '/home/user/.local/pipx/venvs/poetry/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/user/.direnv/python-3.8.11', '--no-deps', '/home/user/.cache/pypoetry/artifacts/b9/cc/b6/9dd390523196e4d45696773322710a1d42e9a14e81a0e1f17257da84fe/torch_sparse-0.6.12.tar.gz']' returned non-zero exit status 1.

The /home/user/.direnv/python-3.8.11/bin/python part is correct while the /home/user/.local/pipx/venvs/poetry/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip part seems wrong.

[abn]

Why is it wrong? I suspect.if you did poetry run pip install /home/user/.cache/pypoetry/artifacts/b9/cc/b6/9dd390523196e4d45696773322710a1d42e9a14e81a0e1f17257da84fe/torch_sparse-0.6.12.tar.gz or pip install /home/user/.cache/pypoetry/artifacts/b9/cc/b6/9dd390523196e4d45696773322710a1d42e9a14e81a0e1f17257da84fe/torch_sparse-0.6.12.tar.gz it fail due to build requirements not being met for torch-sparse.

[Yevgnen]

@abn

Hi, sorry for missing details. The TOML is here.

The failed message indeed is ModuleNotFoundError for torch.

• Installing torch-scatter (2.0.9): Failed

 CalledProcessError

 Command '['/home/user/.direnv/python-3.8.11/bin/python', '/home/user/.local/pipx/venvs/poetry/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/user/.direnv/python-3.8.11', '--no-deps', '/home/user/.cache/pypoetry/artifacts/7d/13/fc/04227bfd9baf499dd26cac8cbc75ba3a329fbca25179f507d36c653252/torch_scatter-2.0.9.tar.gz']' returned non-zero exit status 1.

 at ~/.asdf/installs/python/3.8.11/lib/python3.8/subprocess.py:516 in run
      512│             # We don't call process.wait() as .__exit__ does that for us.
      513│             raise
      514│         retcode = process.poll()
      515│         if check and retcode:
   →  516│             raise CalledProcessError(retcode, process.args,
      517│                                      output=stdout, stderr=stderr)
      518│     return CompletedProcess(process.args, retcode, stdout, stderr)
      519│ 
      520│ 

The following error occurred when trying to handle this error:


 EnvCommandError

 Command ['/home/user/.direnv/python-3.8.11/bin/python', '/home/user/.local/pipx/venvs/poetry/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-21.3.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/user/.direnv/python-3.8.11', '--no-deps', '/home/user/.cache/pypoetry/artifacts/7d/13/fc/04227bfd9baf499dd26cac8cbc75ba3a329fbca25179f507d36c653252/torch_scatter-2.0.9.tar.gz'] errored with the following return code 1, and output: 
 Looking in indexes: https://pypi.doubanio.com/simple/
 Processing /home/user/.cache/pypoetry/artifacts/7d/13/fc/04227bfd9baf499dd26cac8cbc75ba3a329fbca25179f507d36c653252/torch_scatter-2.0.9.tar.gz
   Installing build dependencies: started
   Installing build dependencies: finished with status 'done'
   Getting requirements to build wheel: started
   Getting requirements to build wheel: finished with status 'error'
   ERROR: Command errored out with exit status 1:
    command: /home/user/.direnv/python-3.8.11/bin/python /tmp/tmpwavm3r0k get_requires_for_build_wheel /tmp/tmp97wvo18e
        cwd: /tmp/pip-req-build-7o__dv62
   Complete output (27 lines):
   Traceback (most recent call last):
     File "/tmp/tmpwavm3r0k", line 363, in <module>
       main()
       └ <function main at 0x7f1045c9c550>
     File "/tmp/tmpwavm3r0k", line 345, in main
       json_out['return_val'] = hook(**hook_input['kwargs'])
       │                        │      └ {'kwargs': {'config_settings': None}}
       │                        └ <function get_requires_for_build_wheel at 0x7f1045c9ae50>
       └ {'unsupported': False, 'return_val': None}
     File "/tmp/tmpwavm3r0k", line 130, in get_requires_for_build_wheel
       return hook(config_settings)
              │    └ None
              └ <bound method _BuildMetaBackend.get_requires_for_build_wheel of <setuptools.build_meta._BuildMetaLegacyBackend object at 0x7f104...
     File "/tmp/pip-build-env-xcqarj7c/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
       return self._get_build_requires(
     File "/tmp/pip-build-env-xcqarj7c/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 143, in _get_build_requires
       self.run_setup()
       └ <setuptools.build_meta._BuildMetaLegacyBackend object at 0x7f1042bd8280>
     File "/tmp/pip-build-env-xcqarj7c/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 267, in run_setup
       super(_BuildMetaLegacyBackend,
     File "/tmp/pip-build-env-xcqarj7c/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 158, in run_setup
       exec(compile(code, __file__, 'exec'), locals())
                    │     └ 'setup.py'
                    └ "import os\nimport sys\nimport glob\nimport os.path as osp\nfrom itertools import product\nfrom setuptools import setup, find_pa...
     File "setup.py", line 8, in <module>
       import torch
   ModuleNotFoundError: No module named 'torch'
   ----------------------------------------
 WARNING: Discarding file:///home/user/.cache/pypoetry/artifacts/7d/13/fc/04227bfd9baf499dd26cac8cbc75ba3a329fbca25179f507d36c653252/torch_scatter-2.0.9.tar.gz. Command errored out with exit status 1: /home/user/.direnv/python-3.8.11/bin/python /tmp/tmpwavm3r0k get_requires_for_build_wheel /tmp/tmp97wvo18e Check the logs for full command output.
 ERROR: Command errored out with exit status 1: /home/user/.direnv/python-3.8.11/bin/python /tmp/tmpwavm3r0k get_requires_for_build_wheel /tmp/tmp97wvo18e Check the logs for full command output.

I run a fresh install for this TOML and all the packages except for torch-* were installed successfully.

I run import torch in /home/user/.direnv/python-3.8.11/bin/python then and it was just OK. So I suspect torch-* were using a wrong virtual env when installing.

After the poetry install failure I tried pip install /home/user/.cache/pypoetry/artifacts/b9/cc/b6/9dd390523196e4d45696773322710a1d42e9a14e81a0e1f17257da84fe/torch_sparse-0.6.12.tar.gz as suggested, it didn't throw any error though.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.