No way to specify build-time dependencies of setup.py

[pv]

pip does not currently offer a way to specify build-time dependencies for setup.py. This makes all packages using numpy.distutils (used by nearly every Python package wrapping Fortran code since Python distutils does not support that) not reliably installable via pip.

Quick reading of the pip/setuptools docs gives the impression that setup_requires is supposed to do this, and inform pip/setuptools about build-time dependencies required for running setup.py.

However, this appears incorrect --- in reality the dependencies are installed only when setup() is called, which is usually too late. The value of setup_requires appears not recorded in the *.egg-info files, and is ignored by pip (= special casing the call to egg_info will not help), also when it decides in which order to install packages.

Cf. scipy/scipy#4474 the case for Scipy. Packages using Cython.distutils apparently have the same problem, cf. gh-1958. Solution to the installation problem would be appreciated.

[pfmoore]

Typically, I would say that the correct way of handling this issue is for the projects in question (who know the build process and have the environments to build correctly) to create wheels for the packages they distribute. The pip does not need to do a build in order to install the project, and the end user does not need the build dependencies installed in order simply to use the package.

Wheels are an excellent solution for Windows, and I believe for OSX. For Linux, there are still issues to sort out around binary compatibility, so it's not yet possible to upload binary wheels for Linux to PyPI. At the moment, therefore, I would recommend uploading Windows and OSX wheels, and simply documenting the build requirements for Linux. It may also be worth hosting a project-specific index of wheels for supported Linux distributions, but that's a large overhead and so isn't really a solution for small projects.

[ionelmc]

Matplotlib has worked around this limitation (will download and build numpy if not present). They also have some specialized "Delayed" commands to delay certain operation till numpy is available (I suppose), see their source code: https://github.com/matplotlib/matplotlib

[pv]

Yes, we do wheels and document the build requirements, but it seems people are running into this problem --- it's probably made somewhat worse by the fact that apparently sometimes pip installs the packages in the right order, and sometimes not (eg when running deployed VM).

Note that a working solution for us would be if pip recorded the value of setup_requires in *.egg-info, and then respected it --- or just respected install_requires --- when determining the package installation order. This still requires a hack to make egg_info callable, but that can be done. (Or do recent pip versions respect the install_requires order?)

@ionelmc: thanks, that's interesting to know. Looks a bit painful though, and it doesn't seem to do Fortran.

[pfmoore]

@pv that's cool as far as the wheels are concerned. I presume the users having issues are likely Linux users, in that case - and wheels aren't much help there yet, unfortunately.

The problem over setup_requires/install_requires (and I should point out I'm not an expert in this area so my knowledge is pretty superficial) is that it's really a setuptools feature, and pip doesn't have a lot of control over what happens. For example, recording setup_requires in .egg-info would be something setuptools does, so that would need a PR/bug report there. And pip needs to call setup.py to get the metadata (because it's generated by the egg_info command) so by the time it could know it needs to install a dependency, it's too late. So there's a circular dependency here that's hard to break with current metadata formats.

So what I'm saying is that the build dependency stuff is a really hard problem, and so it's not likely to be solved quickly within pip/setuptools. (When Metadata 2.0 gets implemented, it'll become a lot easier). Sadly, that puts more pressure on projects in the short term to do complicated stuff in setup.py like @ionelmc referred to, which isn't really sustainable long-term.

[pv]

I think pip has control over install_requires, so a low-hanging fruit would seem to be to install dependencies before calling setup.py install/develop/bdist_wheel on packages that depend on them. This requires some song and dance on the setup.py side to make egg_info callable without the dependencies, but as a workaround, that seems somewhat simpler to implement than delaying everything (as done in matplotlib --- however, I'm not sure whether delaying the whole of numpy.distutils is possible).

In the long run, a proper way for specifying build/setup.py dependencies would be very useful. I'm not very familiar with what's going on the package metadata PEPs currently, but hopefully this is addressed.

[pv]

gh-1934 seems relevant here --- however looking at the scipy bug report they used pip 6.0.6 which apparently already had this change and it apparently didn't help, so maybe this is not very robust (does it depend on which order packages are listed in install_requires?)...

[dstufft]

It's not very robust because we currently don't really have a proper dependency graph in pip, we just have a list of things to install. Generally that list is setup (since #1934) so that dependencies are installed before the things that depend on them, but it's not perfect.

[dstufft]

I'm going to close this, pip isn't going to add build requirements on it's own it'll come via the metadata 2.0 PEPs or another PEP.

[piotr-dobrogost]

As this pops up in google when searching for build time requirements in Python I thought I would leave relevant link with current information on the subject – https://packaging.python.org/specifications/#declaring-build-system-dependencies

[alugowski]

I think this issue needs to be reopened.

This is popping up in many places now because many scientific packages have a build dependency on Cython. Also web hosts such as Google AppEngine and ReadTheDocs only support a single requirements.txt as a dependency specification. My understanding is that some Linux packaging has a similar requirement.

In other words, lack of build dependency support makes many scientific and ML packages unusable in many contexts.

@piotr-dobrogost 's note appears to be a solution, but despite claims it doesn't seem to solve the issue. Projects with a valid pyproject.toml are still not building properly.

[pradyunsg]

Could you elaborate on how pyproject.toml is not building properly?

[pfmoore]

Furthermore, rather than having a discussion on this issue that covers a load of topics, can you raise individual issues for any problems you know of with PEP 518 and pyproject.toml? General comments like "not building properly" aren't really actionable, whereas specific reproducible problems would be.

[alugowski]

@pradyunsg after some more debugging, it looks like my particular issue is less serious than it appeared at first.

The package I need to install is hdbscan, which has a Cython check at the top of setup.py. What prompted my comment is that that project does have a pyproject.toml. After more careful investigation it looks like it was added after their last push to PyPI, so in practice the file isn't there. I opened an issue with that project so they reupload

I was then faced with how to install without PyPI. The most obvious is to create my own package using sdist and install that instead. This approach doesn't work either, since sdist appears to not include pyproject.toml. The only way to make the install work is to clone and bundle the entire repo.

The other projects I found had the same issue have resolved the problem without closing the relevant issues. The tone of my earlier comment now appears exaggerated, I apologize for that.

[alugowski]

@pfmoore I understand your frustration. I brought it up here because reading the thread makes it seem like the issue was closed in a "we'll get to it eventually via another project" manner. See the progress comment above, 2 years after the issue was closed. Experience says that this pattern means that the issue has been ignored.

[alugowski]

Summary for folks who got here via Google:

Pip 10 and later support build dependencies, but they have to be specified in a pyproject.toml file. This file only appears to have an effect if downloading from an index like PyPI, it is stripped from the tarball that python setup.py sdist creates.

[pradyunsg]

it is stripped from the tarball that python setup.py sdist creates.

In that case the project is not properly packaging their sources. pyproject.toml has to be included in the sdist.

[ihnorton]

There are work-arounds. See https://github.com/Blosc/bcolz/blob/master/setup.py#L34-L78

[bskinn]

To follow up on pradyunsg's comment, one (the?) way to include pyproject.toml in the sdist is to add it to an include directive within MANIFEST.in:

include pyproject.toml

It then gets wrapped into the tarball on running python setup.py sdist.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.