Unable to install sdist package

[SuperValou]

When using rez-pip to install the future package, the install fails.

Environment

• Windows 10.0
• rez-pip 0.2.0
• rez-pip being installed in its own virtual environment, not part of a rez installation

To Reproduce

• Rezify a python 3.7.7 interpreter through this utility
• Install rez-pip into its own virtualenv
  • open a terminal
  • run rez env python-3.7.7 -- python -m venv C:\rez-pip to generate a virtualenv that will host the rez-pip package
  • run cd C:\rez-pip\Scripts then .\activate to enter the virtualenv
  • run python -m pip install rez-pip to install rez-pip in the virtualenv
  • close the terminal
• open a terminal
• run cd C:\rez-pip\Scripts then .\activate to enter the virtualenv of rez-pip
• run .\rez-pip2.exe future
• see the error:

> rez-pip2 future
INFO     Installing requested packages for Python 3.7.7
ERROR: Could not find a version that satisfies the requirement future (from versions: none)
ERROR: No matching distribution found for future
rez_pip.exceptions.PipError: Failed to run pip command: '.\\python.exe c:\\rez-pip\\lib\\site-packages\\rez_pip\\data\\pip.pyz install -q future --disable-pip-version-check --dry-run --ignore-installed --python-version=3.7.7 --only-binary=:all: --target=/tmp/asd --disable-pip-version-check --report C:\\<tmp_folder_path>\\pip-install-output9cm63vpe'

Pip reported this:

ERROR: Could not find a version that satisfies the requirement future (from versions: none)
ERROR: No matching distribution found for future

Expected behavior
Program installs future as a rez package (as future seems compatible with python 3.7.7).

Actual behavior
The error above occurs and nothing gets installed at all. When having the error with python-3.7.7, no further python version (like 3.9.7, 3.10.8, etc.) gets attempted by rez-pip in its loop.

Investigation
This is probably a pip-trick issue. This doesn't work:

(rez-pip) PS C:\rez-pip\Scripts> .\python.exe c:\rez-pip\lib\site-packages\rez_pip\data\pip.pyz install future --disable-pip-version-check --dry-run --ignore-installed --python-version=3.7.7 --only-binary=:all: --target=/tmp/

ERROR: Could not find a version that satisfies the requirement future (from versions: none)
ERROR: No matching distribution found for future

but this seems to be okay:

(rez-pip) PS C:\rez-pip\Scripts> .\python.exe c:\rez-pip\lib\site-packages\rez_pip\data\pip.pyz install future --disable-pip-version-check --dry-run --ignore-installed --target=/tmp/

Collecting future
  Using cached future-0.18.3-py3-none-any.whl
Would install future-0.18.3

I'm not too familiar with the --python-version and --only-binary args of pip yet so I'll probably add more comments with further investigation.

[JeanChristopheMorinPerso]

This is expected, future doesn't have any wheels on PyPI. See https://pypi.org/project/future/#file.

[SuperValou]

I'm not really familiar with the implications of having a tar.gz instead of a .whl file. Isn't pip able to handle it anyway? I guess not (obviously) but as pip install future is possible it feels weird to have to "know" if a package is rez-pipable or not. I'll investigate more on this topic to better understand the issue.

The thing is, any package having a dependency to this sort of package becomes un-rez-pipable as well. As an example, rez-piping lucidity fails due to its dependency to future:

rez-pip2 lucidity
INFO     Installing requested packages for Python 3.7.7
ERROR: Could not find a version that satisfies the requirement future (from lucidity) (from versions: none)
ERROR: No matching distribution found for future
rez_pip.exceptions.PipError: Failed to run pip command: '.\\python.exe c:\\rez-pip\\lib\\site-packages\\rez_pip\\data\\pip.pyz install -q lucidity --disable-pip-version-check --dry-run --ignore-installed --python-version=3.7.7 --only-binary=:all: --target=/tmp/asd --disable-pip-version-check --report C:\\Users\\V240D~1.ESM\\AppData\\Local\\Temp\\pip-install-outputsnfkn57v'

Pip reported this:

ERROR: Could not find a version that satisfies the requirement future (from lucidity) (from versions: none)
ERROR: No matching distribution found for future

Is it an expected behaviour of rez-pip?

[JeanChristopheMorinPerso]

We use pip to get the full list of dependencies. That's the only thing we use from pip. We don't install packages with pip.

The tar file is called the source distribution (sdist). The source distribution doesn't contain any metadata that can be used to get the list of dependencies. So when you pip install future, pip will actually download the sdist, build a wheel from it, then get the dependencies from that freshly built wheel.

--only-binary=:all: tells pip to not do that and ignore sdists entirely.

[SuperValou]

Thanks a lot for the clarification, now that I dug a bit into this issue on my side as well I better understand this topic.

From what I understand, we use the download url of what is returned by pip (around here), then the whl gets downloaded (here) and unzipped/installed (here). The installation part assumes we're installing a whl file and that's why sdist aren't currently supported if I understand corretly (at least that's where everything starts to break if I try to remove --only-binary=:all: and --python-version from the command and try to run rez-pip future ).

Couldn't we think about extending the ability of rez-pip so it can install sdists as well? Maybe I could rename the issue to "Unable to install sdist packages", what do you think?

[JeanChristopheMorinPerso]

Couldn't we think about extending the ability of rez-pip so it can install sdists as well?

I unfortunately don't think we can with the current architecture. At least for cases like rez-pip2 lucidity where one of the transitive dependencies doesn't have a wheel.

There is a very specific reason why sdists aren't supported and it's currently partially (or implicitly) documented at https://rez-pip.readthedocs.io/en/latest/faq.html#why-can-t-it-install-python-2-packages.

As you can see, rez-pip does not rely on pip to install the packages. We only use pip to resolve which package we need to download and then “manually” install the wheels. This allows to fix the notorious shebang problem with pip. Pip always bakes the full Python interpreter path into the console scripts shebang. The problem is that baking the full path goes against the idea of rez. In rez, we want the use the resolve python package to executable console scripts, not the Python interpreter that was used to install the package.

We could maybe support passing an sdist to rez-pip2 (rez-pip2 /path/to/dsist.tar.gz), but this gets into the build territory, which I tried to avoid like mentioned in https://rez-pip.readthedocs.io/en/latest/transition.html#local-and-editable-installs.

For pure python packages, this seems overkill to me, considering how easy it is to create wheels (you can either use pip wheel <package> --no-deps or https://github.com/pypa/build).

I'll see if I can think about something that could simplify everyone's life, but I don't know when or if I'll be able to come up with a clear answer/plan for this.

[SuperValou]

Thanks for the detailed clarifications, I now see your point and agree that rez-pip should probably not go down the build route as well.

I'll attempt to build a future whl and see what kind of workflow I can come up with to be able to rez-pip lucidity somehow. Maybe doing something as naive as rez-pip2 /path/to/future/wheel lucidity actually works, I'll keep you posted.