numpy version constraint not working in combination with pyworld

[mweinelt]

What did you want to do?

Fresh python 3.6 venv for a quick reproducer. I try to pin numpy<1.20 to keep python3.6 support intact.

requirements.txt

numpy<1.20
pyworld

Installed pip from master @ ab7ff0a, but can repro on 18.0 and 20.3.1.

Output

$ python3 -v
[...]
Python 3.6.12 (default, Aug 15 2020, 06:59:32) 
[...]
$ python3 -m pip --version
pip 21.0.dev0 from /home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/pip (python 3.6)
$ python3 -m pip install -r requirements.txt 
Collecting numpy<1.20
  Using cached numpy-1.19.4-cp36-cp36m-manylinux2010_x86_64.whl (14.5 MB)
Collecting pyworld
  Using cached pyworld-0.2.12.tar.gz (222 kB)
    ERROR: Command errored out with exit status 1:
     command: /home/hexa/tmp/pyworld/env/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xncb8ha5/pyworld_2d4600db54ae4664badc10f8a63244e0/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xncb8ha5/pyworld_2d4600db54ae4664badc10f8a63244e0/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-8e_ugy77
         cwd: /tmp/pip-install-xncb8ha5/pyworld_2d4600db54ae4664badc10f8a63244e0/
    Complete output (68 lines):
    Traceback (most recent call last):
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-i66eh98_/numpy-1.20.0rc1/setup.py", line 30, in <module>
        import numpy
    RuntimeError: Python version >= 3.7 required.
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-xncb8ha5/pyworld_2d4600db54ae4664badc10f8a63244e0/setup.py", line 66, in <module>
        classifiers=[],
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/__init__.py", line 142, in setup
        _install_setup_requires(attrs)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/__init__.py", line 137, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/dist.py", line 586, in fetch_build_eggs
        replace_conflicting=True,
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/pkg_resources/__init__.py", line 780, in resolve
        replace_conflicting=replace_conflicting
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1063, in best_match
        return self.obtain(req, installer)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1075, in obtain
        return installer(requirement)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/dist.py", line 653, in fetch_build_egg
        return cmd.easy_install(req)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 679, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 705, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 890, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 1158, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 1144, in run_setup
        run_setup(setup_script, args)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/sandbox.py", line 253, in run_setup
        raise
      File "/nix/store/l6zbxidx7l94r0i7k17ranxsw5id9141-python3-3.6.12/lib/python3.6/contextlib.py", line 99, in __exit__
        self.gen.throw(type, value, traceback)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/nix/store/l6zbxidx7l94r0i7k17ranxsw5id9141-python3-3.6.12/lib/python3.6/contextlib.py", line 99, in __exit__
        self.gen.throw(type, value, traceback)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/sandbox.py", line 166, in save_modules
        saved_exc.resume()
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/sandbox.py", line 141, in resume
        six.reraise(type, exc, self._tb)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/_vendor/six.py", line 685, in reraise
        raise value.with_traceback(tb)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/home/hexa/tmp/pyworld/env/lib/python3.6/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-i66eh98_/numpy-1.20.0rc1/setup.py", line 30, in <module>
        import numpy
    RuntimeError: Python version >= 3.7 required.
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Additional information

numpy >= 1.20 errors out for python < 3.7:
https://github.com/numpy/numpy/blob/31326067c2aea287a261ded44f36a7e9f78cbc79/setup.py#L29-L30

pyworld does not pin any specific numpy version
https://github.com/JeremyCCHsu/Python-Wrapper-for-World-Vocoder/blob/f2732b5490674c50df05acfee91fc294014c174c/setup.py#L51-L57

[pfmoore]

pyworld doesn't restrict numpy in the build requirements, and your requirements.txt doesn't apply there. As a result, when pip creates the build environment, it tries to install the latest numpy.

I'd have expected numpy 1.20 to have a requires-python >=3.7, and/or pyworld to have a setup_requires that restricts numpy to a compatible version on Python 3.6 (using environment markers).

I'm pretty sure the only way to control this from pip's end is to use --no-build-isolation to manage the build environment manually, and build a pyworld wheel which you can then install locally.

[mweinelt]

But numpy 1.20.0rc1 does specify a minimum python version.

https://github.com/numpy/numpy/blob/v1.20.0rc1/setup.py#L472

Is it possible that this line prevents the evaluation of that minimum version?

https://github.com/numpy/numpy/blob/v1.20.0rc1/setup.py#L29-L30

[pfmoore]

Hmm, that's definitely odd, then. Maybe pyworld isn't using PEP 517 (so it's doing a "legacy" install) in which case setuptools will be processing setup_requires using easy_install - which doesn't respect python-requires.

Yes, I just checked, pyworld doesn't have a pyproject.toml, so it's still using the legacy codepath.

[jvesely]

I'm seeing the same problem with packages like GPy and matplotlib (neither of which uses pyproject.toml).
This problem is still there for Python >=3.7 and it leads to runtime failures when using any of the extensions that were built against numpy-1.20-0-rc1:

ImportError: numpy.core.multiarray failed to import
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd

[uranusjr]

The runtime check (line 29-30) happens when setup.py is loaded, so pip (setuptools, in fact) never reaches the python_requires line (line 472). pyworld does not use pyproject.toml, so the legacy setuptools setup_requires mechanism kicks into place, triggering easy_install, which does not respect data-requires-python on PyPI, to incorrectly download the source distribution to fail the build.

So there’s nothing pip can do here. The ideal solution would be for pyworld to opt into PEP 517 (pyproject.toml).

[jvesely]

Why does easy_install use pre-release packages?

Installing the correct version of numpy ahead of other dependencies works as a workaround.
Perhaps pip can do that and install deps in multiple rounds?

[pradyunsg]

@jvesely that's what adding a pyproject.toml would do -- separate the build step for those projects, so that their build environment is better populated.