Regression in 1.2.x when installing fasttext - can't compile due to not finding pybind and/or pip

[cfculhane]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 22.04 LTS

• Poetry version: 1.2.0b3

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/cfculhane/f8f9edee4af3ee99deab0170b814e72d

• Link to the lockfile: https://gist.github.com/cfculhane/aa093d4333e33d297a8da075d9d43e83

Issue

I've been enjoying the speed and other improvements to the 1.2.x betas, however one package is not installing, and it installs fine on 1.1.14 - fasttext.

Below is the log when installing in 1.2.0b3 with poetry install -vvv

(poetrybug-py3.8)  ~/repos/poetrybug      poetry install -vvv                                                                                                                              
Loading configuration file /home/chris/.config/pypoetry/config.toml
Using virtualenv: /home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8
Project environment contains an empty path in sys_path, ignoring.
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 1 install, 0 updates, 0 removals, 3 skipped

  • Installing fasttext (0.9.2): Pending...
  • Installing numpy (1.23.1): Pending...
  • Installing numpy (1.23.1): Skipped for the following reason: Already installed
  • Installing pybind11 (2.10.0): Pending...
  • Installing pybind11 (2.10.0): Skipped for the following reason: Already installed
  • Installing setuptools (63.4.1): Pending...
  • Installing setuptools (63.4.1): Skipped for the following reason: Already installed
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
  • Installing fasttext (0.9.2): Installing...
  • Installing fasttext (0.9.2): Failed

  Stack trace:

  2  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1454 in _run
      1452│                 )
      1453│             else:
    → 1454│                 output = subprocess.check_output(
      1455│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1456│                 )

  1  ~/.pyenv/versions/3.8.11/lib/python3.8/subprocess.py:415 in check_output
       413│         kwargs['input'] = empty
       414│ 
    →  415│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
       416│                **kwargs).stdout
       417│ 

  CalledProcessError

  Command '['/home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8/bin/python', '/home/chris/.local/share/pypoetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-22.2.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8', '--no-deps', '/home/chris/.cache/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz']' returned non-zero exit status 1.

  at ~/.pyenv/versions/3.8.11/lib/python3.8/subprocess.py:516 in run
       512│             # We don't call process.wait() as .__exit__ does that for us.
       513│             raise
       514│         retcode = process.poll()
       515│         if check and retcode:
    →  516│             raise CalledProcessError(retcode, process.args,
       517│                                      output=stdout, stderr=stderr)
       518│     return CompletedProcess(process.args, retcode, stdout, stderr)
       519│ 
       520│ 

The following error occurred when trying to handle this error:


  Stack trace:

  3  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/pip.py:49 in pip_install
       47│ 
       48│     try:
    →  49│         return environment.run_pip(*args)
       50│     except EnvCommandError as e:
       51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e

  2  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1417 in run_pip
      1415│         pip = self.get_pip_command(embedded=True)
      1416│         cmd = pip + list(args)
    → 1417│         return self._run(cmd, **kwargs)
      1418│ 
      1419│     def run_python_script(self, content: str, **kwargs: Any) -> int | str:

  1  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1707 in _run
      1705│     def _run(self, cmd: list[str], **kwargs: Any) -> int | str:
      1706│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
    → 1707│         return super()._run(cmd, **kwargs)
      1708│ 
      1709│     def get_temp_environ(

  EnvCommandError

  Command ['/home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8/bin/python', '/home/chris/.local/share/pypoetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-22.2.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8', '--no-deps', '/home/chris/.cache/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz'] errored with the following return code 1, and output: 
  Processing /home/chris/.cache/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    error: subprocess-exited-with-error
    
    × Getting requirements to build wheel did not run successfully.
    │ exit code: 1
    ╰─> [25 lines of output]
        /home/chris/.cache/pypoetry/virtualenvs/poetrybug-X9CB-wp6-py3.8/bin/python: No module named pip
        Traceback (most recent call last):
          File "<string>", line 38, in __init__
        ModuleNotFoundError: No module named 'pybind11'
        
        During handling of the above exception, another exception occurred:
        
        Traceback (most recent call last):
          File "/tmp/tmpoxq8b58i", line 363, in <module>
            main()
          File "/tmp/tmpoxq8b58i", line 345, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/tmp/tmpoxq8b58i", line 130, in get_requires_for_build_wheel
            return hook(config_settings)
          File "/tmp/pip-build-env-z78q_hl2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 177, in get_requires_for_build_wheel
            return self._get_build_requires(
          File "/tmp/pip-build-env-z78q_hl2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 159, in _get_build_requires
            self.run_setup()
          File "/tmp/pip-build-env-z78q_hl2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 281, in run_setup
            super(_BuildMetaLegacyBackend,
          File "/tmp/pip-build-env-z78q_hl2/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 174, in run_setup
            exec(code, locals())
          File "<string>", line 72, in <module>
          File "<string>", line 41, in __init__
        RuntimeError: pybind11 install failed.
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  

  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1458 in _run
      1454│                 output = subprocess.check_output(
      1455│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1456│                 )
      1457│         except CalledProcessError as e:
    → 1458│             raise EnvCommandError(e, input=input_)
      1459│ 
      1460│         return decode(output)
      1461│ 
      1462│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:


  Stack trace:

  5  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/executor.py:245 in _execute_operation
      243│ 
      244│             try:
    → 245│                 result = self._do_execute_operation(operation)
      246│             except EnvCommandError as e:
      247│                 if e.e.returncode == -2:

  4  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/executor.py:317 in _do_execute_operation
      315│             return 0
      316│ 
    → 317│         result: int = getattr(self, f"_execute_{method}")(operation)
      318│ 
      319│         if result != 0:

  3  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/executor.py:437 in _execute_install
      435│ 
      436│     def _execute_install(self, operation: Install | Update) -> int:
    → 437│         status_code = self._install(operation)
      438│ 
      439│         self._save_url_reference(operation)

  2  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/executor.py:480 in _install
      478│         )
      479│         self._write(operation, message)
    → 480│         return self.pip_install(archive, upgrade=operation.job_type == "update")
      481│ 
      482│     def _update(self, operation: Install | Update) -> int:

  1  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/executor.py:119 in pip_install
      117│     ) -> int:
      118│         try:
    → 119│             pip_install(req, self._env, upgrade=upgrade, editable=editable)
      120│         except EnvCommandError as e:
      121│             output = decode(e.e.output)

  PoetryException

  Failed to install /home/chris/.cache/pypoetry/artifacts/f7/2f/f2/478b50d736eb337c660080a287ba368aedb592ff8aef70993ea2132db5/fasttext-0.9.2.tar.gz

  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/pip.py:51 in pip_install
       47│ 
       48│     try:
       49│         return environment.run_pip(*args)
       50│     except EnvCommandError as e:
    →  51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       52│ 

  • Installing numpy (1.23.1): Pending...
  • Installing numpy (1.23.1): Skipped for the following reason: Already installed
  • Installing pybind11 (2.10.0): Pending...
  • Installing pybind11 (2.10.0): Skipped for the following reason: Already installed
  • Installing setuptools (63.4.1): Pending...
  • Installing setuptools (63.4.1): Skipped for the following reason: Already installed

And the log when the same is run on 1.1.14:

 ~/repos/poetrybug2      poetry add fasttext -vvv                                                                                                                                          
Using virtualenv: /home/chris/.cache/pypoetry/virtualenvs/poetrybug2-KxUlQ7WG-py3.8
PyPI: No release information found for fasttext-0.1.0, skipping
PyPI: No release information found for fasttext-0.6.3, skipping
PyPI: 33 packages found for fasttext *
Using version ^0.9.2 for fasttext

Updating dependencies
Resolving dependencies...
   1: fact: poetrybug2 is 0.1.0
   1: derived: poetrybug2
   1: fact: poetrybug2 depends on fasttext (^0.9.2)
   1: selecting poetrybug2 (0.1.0)
   1: derived: fasttext (>=0.9.2,<0.10.0)
PyPI: No release information found for fasttext-0.1.0, skipping
PyPI: No release information found for fasttext-0.6.3, skipping
PyPI: 1 packages found for fasttext >=0.9.2,<0.10.0
   1: fact: fasttext (0.9.2) depends on pybind11 (>=2.2)
   1: fact: fasttext (0.9.2) depends on numpy (*)
   1: selecting fasttext (0.9.2)
   1: derived: numpy
   1: derived: pybind11 (>=2.2)
PyPI: No release information found for numpy-0.9.6, skipping
PyPI: No release information found for numpy-0.9.8, skipping
PyPI: No release information found for numpy-1.0.3, skipping
PyPI: No release information found for numpy-1.0.4, skipping
PyPI: No release information found for numpy-1.0b1, skipping
PyPI: No release information found for numpy-1.0b4, skipping
PyPI: No release information found for numpy-1.0b5, skipping
PyPI: No release information found for numpy-1.0rc1, skipping
PyPI: No release information found for numpy-1.0rc2, skipping
PyPI: No release information found for numpy-1.0rc3, skipping
PyPI: No release information found for numpy-1.1.1, skipping
PyPI: No release information found for numpy-1.2.0, skipping
PyPI: No release information found for numpy-1.2.1, skipping
PyPI: No release information found for numpy-1.4.0, skipping
PyPI: 87 packages found for numpy *
PyPI: No release information found for pybind11-1.6, skipping
PyPI: 22 packages found for pybind11 >=2.2
   1: selecting pybind11 (2.10.0)
PyPI: Getting info for numpy (1.23.1) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: numpy-1.23.1.tar.gz
   1: selecting numpy (1.23.1)
   1: Version solving took 5.889 seconds.
   1: Tried 1 solutions.

Writing lock file

Finding the necessary packages for the current system

Package operations: 3 installs, 0 updates, 0 removals

  • Installing numpy (1.23.1): Pending...
  • Installing numpy (1.23.1): Installing...
  • Installing numpy (1.23.1)
  • Installing pybind11 (2.10.0): Pending...
  • Installing pybind11 (2.10.0): Installing...
  • Installing pybind11 (2.10.0)
  • Installing fasttext (0.9.2): Pending...
  • Installing fasttext (0.9.2): Installing...
  • Installing fasttext (0.9.2)

Tried a few things:

• If I start a fresh project with 1.1.14 and add/install fasttext -> no error
• If I start a fresh project with 1.2.0b3 and add/install fasttext -> fails
• If I start with 1.1.14, then update to 1.2.0b3, and add/install fasttextt -> fails
• If I start with 1.2.0n3, then downgrade to 1.1.14, and add/install fasttext -> fails

Seems to be something with the temp env created for the build? Not quite sure, and it installs totally fine with pip.

[dimbleby]

seems to have been broken at #3835, which prevents the fasttext install process from finding the already-installed (and needed) pybind11

[dimbleby]

However fasttext is probably Doing It Wrong by not declaring pybind11 as a build-time requirement.

Probably you can and should fix this over there by pointing them at https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#build-system-requirement and / or submitting an MR.

[dimbleby]

I see that facebookresearch/fastText#1223 has been open for a while so I expect that you'll have to submit an MR if you want this fixed, rather than hoping that they take it on themselves.

[dimbleby]

I just noticed facebookresearch/fastText#1270, not clear whether this is blocked on no-one being interested or on the contributor signing the CLA.

Anyway poetry is not at fault here, this can be closed.

[cfculhane]

Thanks for looking into this - I will investigate a fix and submit a MR, apologies for taking the blame to poetry!

[juhoinkinen]

While waiting the issue to be fixed on fastText side, Poetry can install fastText using the prebuilt wheels available from fasttext-wheel PyPI project.

[victorphamdeveloper]

@juhoinkinen I tried to add fasttext-wheel but poetry keeps complaining it cannot find any versions matching 0.9.2 for fasttext-wheel

[thesofakillers]

wait so how did this issue get resolved? Poetry still fails when running poetry add fasttext

[Secrus]

wait so how did this issue get resolved? Poetry still fails when running poetry add fasttext

Yes, but Poetry isn't at fault here, fasttext doesn't provide Poetry with enough info on what built dependencies they need. See PRs mentioned above.

[thesofakillers]

Thanks @Secrus ! For anyone interested I think the clearest PR to follow is facebookresearch/fastText#1292. Seems like fasttext is pretty dead.

as mentioned on there, for anyone looking for a quick workaround:

poetry add git+https://github.com/cfculhane/fastText

or what i did (ugly)

pip install fasttext
poetry add fasttext

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.