QUERY: asv 0.6.2 handles uses lowest supported versions (build requirements in pyproject.toml incorrectly)

[jwodder]

We are using asv to benchmark a project whose build-system.requires field in pyproject.toml contains "setuptools >= 30.3.0". When asv 0.6.2 prepares a virtualenv for this project, it installs version 30.3.0 of setuptools (rather than the latest version) in the virtualenv (Why aren't you using build isolation?), which then causes a build failure for one of our dependencies (methodtools), which does not have a pyproject.toml but does require setuptools 39.2.0+. If we downgrade to asv 0.6.1 or remove the >= 30.3.0 from our pyproject.toml, everything is installed correctly.

[HaoZeke]

This is not an ASV issue, if the project doesn't support setuptools >= 30.3.0 it shouldn't request it. Also what is in the build matrix in the ASV configuration?

[jwodder]

• The project does support setuptools 30.3.0; it's just that asv installs setuptools 30.3.0 exactly and also doesn't isolate builds, which causes a problem with a dependency that requires a setuptools higher than 30.3.0.

• The matrix key in asv.conf.json (assuming that's what you're referring to) is commented out.

[HaoZeke]

• The project does support setuptools 30.3.0; it's just that asv installs setuptools 30.3.0 exactly and also doesn't isolate builds, which causes a problem with a dependency that requires a setuptools higher than 30.3.0.

One virtualenv is created for each matrix entry. Could you share the whole run?

* The `matrix` key in `asv.conf.json` (assuming that's what you're referring to) is commented out.

The matrix is how you should provide additional packages to install.

a dependency that requires a setuptools higher than 30.3.0.

If you have a hard dependency on something which needs setuptools higher than 30.3.0 then your code also only supports higher than 30.3.0.

[jwodder]

If you have a hard dependency on something which needs setuptools higher than 30.3.0 then your code also only supports higher than 30.3.0.

Setuptools is a build dependency for both packages, not a runtime dependency, so the version requirements are independent of each other. Normally, when pip install a package from source, it uses build isolation, creating a temporary venv for just that package into which the build dependencies are installed for use during the build, and then they're deleted once the build is done. asv, on the other hand, installs build dependencies into the same environment as it installs everything else (and, for some reason, when it sees a build requirement like setuptools >= 30.3.0, it installs v30.3.0 exactly instead of the latest setuptools version like pip would). As a result, when asv installs our package into an asv virtualenv, that virtualenv ends up with setuptools 30.3.0, and when it then attempts to install the problematic dependency (which does not have a wheel or a pyproject.toml file) by building it in the same venv, an error occurs.

I am now going to ask you point-blank: Why does asv not use build isolation when installing packages from source? That violates standard practice.

[HaoZeke]

If you have a hard dependency on something which needs setuptools higher than 30.3.0 then your code also only supports higher than 30.3.0.

Setuptools is a build dependency for both packages, not a runtime dependency, so the version requirements are independent of each other. Normally, when pip install a package from source, it uses build isolation, creating a temporary venv for just that package into which the build dependencies are installed for use during the build, and then they're deleted once the build is done. asv, on the other hand, installs build dependencies into the same environment as it installs everything else (and, for some reason, when it sees a build requirement like setuptools >= 30.3.0, it installs v30.3.0 exactly instead of the latest setuptools version like pip would). As a result, when asv installs our package into an asv virtualenv, that virtualenv ends up with setuptools 30.3.0, and when it then attempts to install the problematic dependency (which does not have a wheel or a pyproject.toml file) by building it in the same venv, an error occurs.

I am now going to ask you point-blank: Why does asv not use build isolation when installing packages from source? That violates standard practice.

Because we do not use pip as the default installation method for the benchmarked packages.

The default project build is now python -m build && python -mpip wheel -w {build_cache_dir} {build_dir.
The earlier default was python setup.py build && python -mpip wheel -w {build_cache_dir} {build_dir

A few versions ago the build process would also not use index or fetch from PyPI (using network isolating environment variables).

I'm open to a PR with a better default, but honestly the simplest approach here is to set the project build command to whatever is necessary (it sounds like pip install . would work for you).

[HaoZeke]

Also ASV has a different requirement, which is that you require different environments and the package is benchmarked accordingly. Letting pip handle dependency resolution and fetch different versions defeats the purpose.

Again, here one uses matrix to define the dependencies for configurations of the project to be tested.

[HaoZeke]

Thus if it is required to test across a set of versions, they go in the matrix, ASV then makes one virtualenv for each matrix combination, and then installs the project in each of those (preferably without updating components!). The build command is not meant to add / remove components which may be part of the benchmark matrix.

[jwodder]

Here are the logs from a run of asv run --show-stderr --verbose HEAD^1..HEAD with setuptools >= 30.3.0 in our project's pyproject.toml: https://gist.github.com/jwodder/0e5bb56107d24d783aff9d7980dcde1a

python -m build && python -mpip wheel -w {build_cache_dir} {build_dir

Both parts of that command should use build isolation by default. Is asv setting some environment variable to disable isolation?

---

We have no desire to test across different sets of versions or using benchmarking-specific dependencies. I thus don't see how matrix does anything for us.

[HaoZeke]

environment variable to disable isolation

Quite the opposite, the earlier versions of ASV used to set no-build-isolation via the environment variable which was recently removed.

ASV doesn't really make any distinctions between build time and / bench time dependencies. Each environment gets a list of things to be installed via the user defined, or default installation command. The only difference being build dependencies are not part of label.

[HaoZeke]

Is this also resolved by changing install commands as in #1389 @jwodder?