Package dependency resolution issues in poetry >= 1.2 ( example with fasttext )

[zendesk-cirode]

• Poetry version: 1.2.0, 1.2.1

• Python version: 3.8.3

• OS version and name: macos 12.5

• pyproject.toml: https://gist.github.com/zendesk-cirode/7548ee5cd075cab84553c0a9da0c08ae

• I am on the latest stable Poetry version, installed using a recommended method.

• I have searched the issues of this repo and believe that this is not a duplicate.

• I have consulted the FAQ and blog for any relevant entries or release notes.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Using the above simplified pyproject.toml in poetry 1.1.15 works fine, however in 1.2.0 and above it results in the following

 poetry install -vvv
Creating virtualenv testing-11mUp4j6-py3.8 in /Users/chris.rode/Library/Caches/pypoetry/virtualenvs
Using virtualenv: /Users/chris.rode/Library/Caches/pypoetry/virtualenvs/testing-11mUp4j6-py3.8
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 3 installs, 0 updates, 0 removals

  • Installing numpy (1.23.3): Pending...
  • Installing numpy (1.23.3): Installing...
  • Installing numpy (1.23.3)
  • Installing pybind11 (2.10.0): Pending...
  • Installing pybind11 (2.10.0): Installing...
  • Installing pybind11 (2.10.0)
  • Installing fasttext (0.9.2): Pending...
  • Installing fasttext (0.9.2): Installing...
  • Installing fasttext (0.9.2): Failed

  Stack trace:

  2  ~/.asdf/installs/python/3.8.3/lib/python3.8/site-packages/poetry/utils/env.py:1472 in _run
      1470│                 )
      1471│             else:
    → 1472│                 output = subprocess.check_output(
      1473│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1474│                 )

  1  ~/.asdf/installs/python/3.8.3/lib/python3.8/subprocess.py:411 in check_output
       409│         kwargs['input'] = '' if kwargs.get('universal_newlines', False) else b''
       410│
    →  411│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
       412│                **kwargs).stdout
       413│

  CalledProcessError

  Command '['/Users/chris.rode/Library/Caches/pypoetry/virtualenvs/testing-11mUp4j6-py3.8/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/Users/chris.rode/Library/Caches/pypoetry/virtualenvs/testing-11mUp4j6-py3.8', '--no-deps', '/Users/chris.rode/Library/Caches/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz']' returned non-zero exit status 1.

  at ~/.asdf/installs/python/3.8.3/lib/python3.8/subprocess.py:512 in run
       508│             # We don't call process.wait() as .__exit__ does that for us.
       509│             raise
       510│         retcode = process.poll()
       511│         if check and retcode:
    →  512│             raise CalledProcessError(retcode, process.args,
       513│                                      output=stdout, stderr=stderr)
       514│     return CompletedProcess(process.args, retcode, stdout, stderr)
       515│
       516│

The following error occurred when trying to handle this error:


  Stack trace:

  3  ~/.asdf/installs/python/3.8.3/lib/python3.8/site-packages/poetry/utils/pip.py:49 in pip_install
       47│
       48│     try:
    →  49│         return environment.run_pip(*args)
       50│     except EnvCommandError as e:
       51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e

  2  ~/.asdf/installs/python/3.8.3/lib/python3.8/site-packages/poetry/utils/env.py:1435 in run_pip
      1433│         pip = self.get_pip_command()
      1434│         cmd = pip + list(args)
    → 1435│         return self._run(cmd, **kwargs)
      1436│
      1437│     def run_python_script(self, content: str, **kwargs: Any) -> int | str:

  1  ~/.asdf/installs/python/3.8.3/lib/python3.8/site-packages/poetry/utils/env.py:1712 in _run
      1710│     def _run(self, cmd: list[str], **kwargs: Any) -> int | str:
      1711│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
    → 1712│         return super()._run(cmd, **kwargs)
      1713│
      1714│     def get_temp_environ(

  EnvCommandError

  Command ['/Users/chris.rode/Library/Caches/pypoetry/virtualenvs/testing-11mUp4j6-py3.8/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/Users/chris.rode/Library/Caches/pypoetry/virtualenvs/testing-11mUp4j6-py3.8', '--no-deps', '/Users/chris.rode/Library/Caches/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz'] errored with the following return code 1, and output:
  Processing /Users/chris.rode/Library/Caches/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    error: subprocess-exited-with-error

    × Getting requirements to build wheel did not run successfully.
    │ exit code: 1
    ╰─> [25 lines of output]
        /Users/chris.rode/Library/Caches/pypoetry/virtualenvs/testing-11mUp4j6-py3.8/bin/python: No module named pip
        Traceback (most recent call last):
          File "<string>", line 38, in __init__
        ModuleNotFoundError: No module named 'pybind11'

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/Users/chris.rode/Library/Caches/pypoetry/virtualenvs/testing-11mUp4j6-py3.8/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
            main()
          File "/Users/chris.rode/Library/Caches/pypoetry/virtualenvs/testing-11mUp4j6-py3.8/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/Users/chris.rode/Library/Caches/pypoetry/virtualenvs/testing-11mUp4j6-py3.8/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
            return hook(config_settings)
          File "/private/var/folders/t1/qx21qb6n5qs_gsvftfws7h080000gp/T/pip-build-env-l3zfaczq/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
            return self._get_build_requires(config_settings, requirements=['wheel'])
          File "/private/var/folders/t1/qx21qb6n5qs_gsvftfws7h080000gp/T/pip-build-env-l3zfaczq/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
            self.run_setup()
          File "/private/var/folders/t1/qx21qb6n5qs_gsvftfws7h080000gp/T/pip-build-env-l3zfaczq/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 482, in run_setup
            super(_BuildMetaLegacyBackend,
          File "/private/var/folders/t1/qx21qb6n5qs_gsvftfws7h080000gp/T/pip-build-env-l3zfaczq/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 335, in run_setup
            exec(code, locals())
          File "<string>", line 72, in <module>
          File "<string>", line 41, in __init__
        RuntimeError: pybind11 install failed.
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.


  at ~/.asdf/installs/python/3.8.3/lib/python3.8/site-packages/poetry/utils/env.py:1476 in _run
      1472│                 output = subprocess.check_output(
      1473│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1474│                 )
      1475│         except CalledProcessError as e:
    → 1476│             raise EnvCommandError(e, input=input_)
      1477│
      1478│         return decode(output)
      1479│
      1480│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:


  Stack trace:

  5  ~/.asdf/installs/python/3.8.3/lib/python3.8/site-packages/poetry/installation/executor.py:253 in _execute_operation
      251│
      252│             try:
    → 253│                 result = self._do_execute_operation(operation)
      254│             except EnvCommandError as e:
      255│                 if e.e.returncode == -2:

  4  ~/.asdf/installs/python/3.8.3/lib/python3.8/site-packages/poetry/installation/executor.py:326 in _do_execute_operation
      324│             return 0
      325│
    → 326│         result: int = getattr(self, f"_execute_{method}")(operation)
      327│
      328│         if result != 0:

  3  ~/.asdf/installs/python/3.8.3/lib/python3.8/site-packages/poetry/installation/executor.py:446 in _execute_install
      444│
      445│     def _execute_install(self, operation: Install | Update) -> int:
    → 446│         status_code = self._install(operation)
      447│
      448│         self._save_url_reference(operation)

  2  ~/.asdf/installs/python/3.8.3/lib/python3.8/site-packages/poetry/installation/executor.py:488 in _install
      486│         )
      487│         self._write(operation, message)
    → 488│         return self.pip_install(archive, upgrade=operation.job_type == "update")
      489│
      490│     def _update(self, operation: Install | Update) -> int:

  1  ~/.asdf/installs/python/3.8.3/lib/python3.8/site-packages/poetry/installation/executor.py:123 in pip_install
      121│     ) -> int:
      122│         try:
    → 123│             pip_install(req, self._env, upgrade=upgrade, editable=editable)
      124│         except EnvCommandError as e:
      125│             output = decode(e.e.output)

  PoetryException

  Failed to install /Users/chris.rode/Library/Caches/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz

  at ~/.asdf/installs/python/3.8.3/lib/python3.8/site-packages/poetry/utils/pip.py:51 in pip_install
       47│
       48│     try:
       49│         return environment.run_pip(*args)
       50│     except EnvCommandError as e:
    →  51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       52│

It looks like it installs the dependency fine, but then cannot find that dependency in the next stage. This occurs whether or not pybind11 is installed in the poetry virtualenv or not.

pip install fasttext works fine as shown below

> pip list
Package    Version
---------- -------
numpy      1.23.3
pip        22.2.2
pybind11   2.10.0
setuptools 65.4.1
wheel      0.37.1


> pip install fasttext
Collecting fasttext
  Downloading fasttext-0.9.2.tar.gz (68 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.8/68.8 kB 1.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: pybind11>=2.2 in /Users/chris.rode/Library/Caches/pypoetry/virtualenvs/testing-11mUp4j6-py3.8/lib/python3.8/site-packages (from fasttext) (2.10.0)
Requirement already satisfied: setuptools>=0.7.0 in /Users/chris.rode/Library/Caches/pypoetry/virtualenvs/testing-11mUp4j6-py3.8/lib/python3.8/site-packages (from fasttext) (65.4.1)
Requirement already satisfied: numpy in /Users/chris.rode/Library/Caches/pypoetry/virtualenvs/testing-11mUp4j6-py3.8/lib/python3.8/site-packages (from fasttext) (1.23.3)
Building wheels for collected packages: fasttext
  Building wheel for fasttext (setup.py) ... done
  Created wheel for fasttext: filename=fasttext-0.9.2-cp38-cp38-macosx_10_16_x86_64.whl size=352631 sha256=9694a3f149a07a6ebc174637ce785cb9b067c3d8d9b266ab4608687c3a496010
  Stored in directory: /Users/chris.rode/Library/Caches/pip/wheels/93/61/2a/c54711a91c418ba06ba195b1d78ff24fcaad8592f2a694ac94
Successfully built fasttext
Installing collected packages: fasttext
Successfully installed fasttext-0.9.2

[neersighted]

This is a problem with an undeclared build-time dependency in fasttext -- pip install --use-pep517 fasttext fails as well, and shows that the problem lies upstream.

In order to resolve this, fasttext will need to specify its build time dependencies with either the deprecated setup_requires kwarg in setup.py, or in accordance with PEP 517.

This is a pretty common issue to have -- if upstream needs more guidance or if you would like a more details, let me know and I can link you to other documentation and similar previous discussions.

[zendesk-cirode]

Ahhh! Yes thankyou. Linking related fasttext issue for future interest facebookresearch/fastText#1292

[neersighted]

Aha, dupe on #6113.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.