Support installing requirements from arbitrary keys in pyproject.toml

[dholth]

What's the problem this feature will solve?

It can sometimes be useful to install from a pyproject.toml that has a list of install_requires = [] without having to build the whole package.

pip install -r pyproject.toml --key tool.enscons.install_requires would open pyproject.toml, look up ['tool']['enscons']['install_requires'], and install as if those items had been passed as arguments to pip install, or written out to a requirements.txt file and then installed.

It wouldn't work correctly if the package referenced its own extras, unless a second key for the extras dict was passed.

If this feature was added it would be easy to support .json as well.

[pfmoore]

I'd rather support -r -, which would then allow you to use some sort of TOML equivalent of jq¹ to read the values you want out and pass them in that way. That would be more general than a specific pyproject.toml reading feature.

¹ A quick search found https://github.com/jamesmunns/tomlq

[uranusjr]

Personally I tend to think all -r usages on “non-locked” requirement specifications should be discouraged, since it does not fit well into the recent best practice trend of lock files. I would be in favour of introducing a new requirements file format, but pyproject.toml should not be it; the lock file format should.

[pradyunsg]

I'd rather support -r -

#7822

[chrisjbillington]

I'm looking for something like this too.

My use case is making sdists in CI. pip wheel with its PEP 518 support means I no longer need to install build dependencies (from a requirements.txt or otherwise) prior to making a wheel, and I don't need to play import-guarding tricks in setup.py to make sure it is runnable without build requirements.

That's great if I'm making a wheel, but the same problems exist for sdists, which remain unsolved. Even though I'm not building anything for an sdist, some of my projects still import cython in their setup.py, they still use setuptools_scm for versioning. So to make an sdist some of these requirements are needed, and I'd have to resort to tricks again to delay importing the others, or install all build dependencies before running python setup.py sdist.

So what I would really like is a pip sdist command that does the build isolation process and installs the 'build' dependencies but makes an sdist instead of a wheel. But a pip install -r pyproject.toml would work pretty well too - no isolation but it doesn't matter on the CI server.

pip install -r pyproject.toml is a bit nicer than

pip install toml && python -c 'import toml; c = toml.load("pyproject.toml")
print("\n".join(c["build-system"]["requires"]))' | pip install -r /dev/stdin

Which is what I'm currently considering putting in my CI.

[sbidoul]

@chrisjbillington for building sdists, you could try python -m pep517.build from the pep517 project. Maybe one day pip or twine or some other tool will gain that capability, but in the meantime that should do the job.

[uranusjr]

Please do not use non-standard top-level keys in pyproject.toml. All top-level keys except tool are reserved for future use by Python packaging, according to PEP 518.

[chrisjbillington]

My problem is completely addressed by python -m pep517.build -s ., which other than the awkward name of the pep517 project has no drawbacks that I've encountered. I'm not in need of any other solutions to this problem.

[sbidoul]

@chrisjbillington in the meantime, build became a thing so the awkward name issue is resolved too.

[flying-sheep]

I wouldn’t call it a hack, it’s simply very manual. A hack is something reaching into APIs you’re not supposed to muck around with. This uses bog-standard, spec compliant ways to do what we want.

But it is awkward. Slightly less is e.g. using rq to convert it to JSON and then jq to query and transform:

rq -t <pyproject.toml | jq -r '.["build-system"]["requires"][]' | xargs -d '\n' pip install

PS: pip install has no -y flag last time I checked.

[uranusjr]

The same can be achieved by pip install . and then pip install --no-deps .. But if we’re talking about containers, installing directly from a pyproject.toml is already not optimal and it’s why lock files were being introduced.

[dudil]

@uranusjr - just to display the case in a different angle.
I am coming from pipenv concept where you have the following files:

• Pipfile --> contains the required/requested packages to install. Can look at it as an "upgraded" version of requirement.txt
• Pipfile.lock --> contains the actual packages installed / versions / hash code etc.

What I believe most community members here are asking is to be able to use pyproject.toml as a replacement for the Pipfile concept. This is not in conflict or diminishing the need for a Pipfile.lock mechanism / full protected installation through pyproject.toml, but this is just a different scenario which seems to be already covered in the proposed PEP 650 where if you to specify an installer backend will take care of that (Please correct me if wrong).

However, for developers who are not asking for locking mechanism (for their own reasons) installing from pyproject.toml as alternative of requirements.txt seems very much reasonable.

[uranusjr]

I am coming from pipenv concept where you have the following files:

• Pipfile --> contains the required/requested packages to install. Can look at it as an "upgraded" version of requirement.txt
• Pipfile.lock --> contains the actual packages installed / versions / hash code etc.

I don’t disagree with your analysis, but pipenv installs from Pipfile.lock, not Pipfile. So if you come from that, the reasonable thing wouldn’t be asking for pip install -r pyproject.toml, but something analogous to Pipfile.lock to pass after -r. This is exactly what we’re working toward instead.

However, for developers who are not asking for locking mechanism (for their own reasons) installing from pyproject.toml as alternative of requirements.txt seems very much reasonable.

The general consensus of pip maintainers working on this issue is that people should ask for locking in all cases, so we’re going for a lock file solution. People are free to disagree with this decision, but they can always work on their own solution in their own tool as well.

[dudil]

I don’t disagree with your analysis, but pipenv installs from Pipfile.lock, not Pipfile. So if you come from that, the reasonable thing wouldn’t be asking for pip install -r pyproject.toml, but something analogous to Pipfile.lock to pass after -r. This is exactly what we’re working toward instead.

It wasn't clear from previous discussions but I get your point now, that makes more sense - thank you for the elaboration.

The general consensus of pip maintainers working on this issue is that people should ask for locking in all cases, so we’re going for a lock file solution. People are free to disagree with this decision, but they can always work on their own solution in their own tool as well.

Yes - that was clear.
From what I can see even with PEP 650, still there is no support for plain vanilla pip lock file, it is just going to call pipenv / poetry etc. so if you are going for a lock file solution, I do hope it will be part of pip and not as proposed in that PEP (couldn't find it if this is indeed the case).

[uranusjr]

From what I can see even with PEP 650, still there is no support for plain vanilla pip lock file, it is just going to call pipenv / poetry etc.

Yeah, we’re working on a completing proposal to PEP 650 that does introduce the lock file. Stay tuned.

[claco]

For those in this same boat, here's the solution I ended up using.
Migrate requirements.txt into poetry.tools.dependencies and dev-dependencies in pyproject.toml

For devs, I just make a venv, then poetry install there to get everything.
For the non-dev ones, pip install -r /opt/extensions/lib . in the Dockerfile does the needful, and only knows enough to install dependencies, not dev-dependencies.

[pradyunsg]

I've just gone through and hidden a bunch of comments that are about "how to build a pyproject.toml based project into a distribution" -- which is not what this issue is about. I've also retitled this issue to better reflect the original request from @dholth.

---

Responding to the original request here: I don't think reading from pyproject.toml and installing arbitrary keys is a use case that I want pip to directly handle. It is already possible to do this, by using other tools to parse the pyproject.toml file and passing those values into pip -- as demonstrated in #8049 (comment).

To that end, I don't think this functionality belongs in pip. I can see users (ab)using this to install from project.dependencies or project.optional-dependencies to try to avoid the self-installation, which is realistically better handled with --only-deps (as noted in #8049 (comment)). I think this would serve as an excellent footgun for most usecases, and in the cases where this is directly helpful, I don't think it provides a significant-enough advantage to justify adding this footgun. I'll also flag that I don't think that -r is the right syntax for this, even if we did add this -- since it starts mixing potential input formats to the same option, overloading it and adding complexity to the UI.

[flying-sheep]

I filed #11440 to bring this forward.

[ctapobep]

Waaait a sec.. Why is this discussion about pip reading pyproject.toml dependencies on its own? For CI builds we don't need to install dependencies from pyproject.toml, we MUST install from poetry.lock or whatever the backend supports. It doesn't seem to be right that pip would decide which file to read on its own. Since the backend is used for building - that has to be the backend's responsibility.

And:

1. Either backends must be compliant with what pip thinks the installation should look like (poetry uses virtualenvs, which isn't the same as what pip does)
2. Or the job of the backends would be to gather the list of dependencies - and simply give them to pip (which is probably better as this would guarantee the consistency)

[rgommers]

@ctapobep I think you're confused here. This has nothing to do with backends or Poetry, or lock files. Poetry isn't even a backend here, it's a frontend (it offer both and more, but we're talking about installing which is a frontend feature). If you want to install dependencies with Poetry, use poetry install. In fact, it already seems that Poetry has the exact feature under discussion here: https://python-poetry.org/docs/basic-usage/#installing-dependencies-only.

[ctapobep]

@rgommers okay, I may have been late for the party, and this may not be the right issue to comment in (just the 1st one that google returned for "pip install pyproject.toml").. But the fact that pip is trying to read & install dependencies from pyproject.toml seems quite wrong.

Why do we use pyproject.toml? Because we have a project to build & distribute. For that we use a build tool which may or may not use pip under the hood. But only that build tool can possibly know which dependencies should be downloaded. pip cannot and should not guess them. Otherwise when you build things locally, you get one set of dependencies, and when you install them e.g. in Docker using pip - the set is going to be different.

This whole pyproject.toml endeavour seems to be going into the wrong direction since the implementation details (like dependencies) are leaking into the abstraction layer.

So what we get is:

1. pip can be the frontend for building
2. pip can be the backend for dependency download. Oookay, we can split pip into 2 half, confusing but possible
3. And now also pip can act on its own and determine the dependencies.

Seems 0:2 for "Python vs. build tools".

[pfmoore]

I'm going to close this issue, as I don't believe it's something we should do in the form suggested here. Most of the actionable items either have their own issues, or can be done already.

• Add --only-deps (and --only-build-deps) option(s) #11440 covers the case of installing the project's dependencies without the project itself.
• There's a general view that picking a section from pyproject.toml to install is not something we would typically want users to be doing in any case, as it's usually a poor substitute for "proper" environment management.
• Users who know what they are doing can use a jq-for-toml style tool to extract the data and supply it to pip via $(...) or xargs.
• Piping requirements into pip is covered by pip install from stdin #7822