Proposal: pipenv patterns and antipatterns for python library project

[vlcinsky]

Hacking maya I learned few lessons which resulted in my following proposal of recommended usage of pipenv in python libraries. I expect others to review the proposal and if we reach agreement, the (updated) text could end up in pipenv docs.

pipenv patterns and antipatterns for python library project

EDIT
Following is best applicable for general (mostly Open Source) python libraries, which are supposed to run on different python versions and OSes. Libraries developed in strict Enterprise environment may be different case (be sure to review all the Problems sections anyway).

END OF EDIT

TL;DR: Adding pipenv files into python library project is likely to introduce extra complexity and can hide some errors while not adding anything to library security. For this reason, keep Pipfile, Pipfile.lock and .env out of library source control.

You will be able to use full power of pipenv regardless of it's files living in .gitignore.

Python library versus python application

By python library I mean a project, typically having setup.py, being targeted for distribution and usage on various platform differing in python version and/or OS.

Examples being maya, requests, flask etc.

On the other side (not python library) there are applications targeted for specific python interpreter, OS and often being deployed in strictly consistent environment.

pipfile describes these differences very well in it's Pipfile vs setup.py.

What is pipenv (deployment tool)

I completely agree on the statement, that pipenv is deployment tool as it allows to:

• define strict requirements (Pipfile.lock) for deployment of virtual environment
• apply those strict requirements in reproducible manner on different machines

It helps when one has to deploy an application or develop in python environment very consistent across multiple developers.

To call pipenv packaging tool is misleading if one expects it to create python libraries or to be deeply involved in creation of them. Yes, pipenv can help a lot (in local development of libraries) but can possibly harm (often in CI tests when used without deeper thought).

Applying "security reasons" in wrong context

TL;DR: pipenv provides secure environment via applying approved concrete dependencies described in Pipfile.lock file and python library is only allowed to define abstract dependencies (thus cannot provide Pipfile.lock).

pipenv shines in deployment scenarios following these steps:

• define abstract dependencies (via Pipfile)
• generate from it concrete dependencies resulting in Pipfile.lock
• create (virtual) python environment reflecting those concrete dependencies
• run tests to make sure, given environment works as expected and is secure
• release the tested "golden" Pipfile.lock as definition of approved python environment
• others can use pipenv sync to apply "the golden" Pipfile.lock elsewhere getting identical python environment.

With development of python library one cannot achieve such security, because libraries must not define concrete dependencies. Breaking this rule (thus trying to declare concrete dependencies by python library) results in problems such as:

• problems to find satisfying version of shared libraries (each strict package defines exact version of shared library and it is very likely the versions will differ and prevent finding commonly acceptable version)
• concrete dependencies may depend on python version, OS or other environment markers and trying to install the package in diferent context can easily fail to satisfy some of rules defined in original abstract dependencies.

Problem: Hiding broken setup.py defined dependencies

setup.py shall define all abstract dependencies via install_requires.

If Pipfile defines those dependencies too, it may easily hide problems such as:

• missing dependency in install_requires
• Pipfile defines specific rules (version ranges etc.) for a dependency and install_requires does not.

To prevent it, follow these rules:

• library defined dependencies must not appear in Pipfile
• the [packages] section in Pipfile shall be either empty or define only single dependency on the library itself.

Problem: Pipfile.lock in repository

Keeping Pipfile.lock (typically for "security reasons") in library repository is wrong, because:

• described dependencies are likely to be invalid for different python versions or in another OS
• developers are forced to update the file not only when they add/remove some dependency, but also when other libraries are updated and may be usable within the library.

To prevent it, one should:

• remove Pipfile.lock from repository and add it into .gitignore

Problem: Competing with tox (hiding usedevelop)

If tox.ini contains in it's commands section entries such as:

• pipenv install
• pipenv install --dev
• pipenv lock

it is often a problem, because:

• pipenv install shall install only the library itself, and tox is (by default) doing it too. Apart from duplicity it also prevents of usedevelop=True and usedevelop=False in tox.ini because Pipenv is able to express it only in one variant (and tox.ini allows differencies in different environments).

To prevent it, one should:

• refrain from using pipenv in tox.ini. See requests tox.ini

Problem: Breaking builds, if pipenv fails

pipenv is under heavy development and things break sometime. If such issue breaks your CI build, there is a failure which could be prevented by not using pipenv and using traditional tools (which are often a bit more mature).

To prevent it, one should:

• think twice before adding pipenv into a CI build script, tox.ini or similar place. Do you know what value you get from adding it? Could be the job done with existing tooling?
• do not add it "just for security reasons" or because "everybody does".

Summary

Key questions regarding pipenv role in development of python library are:

• What value pipenv really brings? A: Virtualenv management tool.
• What is relevent use case for pipenv? A: Manage virtualenv.
• Shall it appear in the library repository? A: NO.

Few more details and tricks follow.

pipenv will not add any security to your package

Do not push it into project just because everybody does it or because you expect extra security. It will disappoint you.

Securing by using concrete (and approved) dependencies shall take place in later phase in the application going to use your library.

Keep Pipfile, Pipfile.lock and .env files out of repository

Put the files into .gitignore.

Pipfile is easy to recreate as demonstrated below as most or all requirements are already defined in your setup.py. And the .env file probably contains private information, which shall not be shared.

Keeping these files out of repository will prevent all the problems, which may happen with CI builds when using pipenv in situations, which are not appropriate.

pipenv as developer's private toolbox

pipenv may simplify developer's work as virtualenv management tool.

The trick is to learn, how to quickly recreate your (private) pipenv related files, e.g.:

$ cd <project_repository>
$ # your library will bring the dependencies (via install_requires in setup.py)
$ pipenv install -e .   
$ # add more dev tools you preffer 
$ pipenv install --dev ipython pdbpp
$ # start hacking
$ pipenv shell
...

Use .env file if you need convenient method for setting up environment variables.

Remember: Keep pipenv usage out of your CI builds and your life will be simpler.

Trick: Use setup.py ability to declare extras dependencies

In your setup.py use the extras_requires section:

from setuptools import setup

setup(
    name='mypackage',
    ....,
    install_requires=["jinja2", "simplejson"],
    extras_require={
        'tests': ['pytest', 'pyyaml'],
        'pg': ['psycopg2'],
    },
    ....
)

To install all dependencies declared for tests extra:

$ pipenv install -e .[tests]

Note, that it will always include the install_requires dependencies.

This method does not allow spliting dependencies into default and dev sections, but this shall not be real problem in expected scenarios.

[techalchemy]

This is very impressive, thanks a ton for compiling. Will definitely review in more detail in a bit

/cc @uranusjr @jtratner @ncoghlan

[vlcinsky]

Some references to maya issues:

• Remove Pipfile.lock from repository kennethreitz/maya#138 (RemovePipfile.lock from repository)
• Skip pipenv run in tox.ini, as the virtualenv is already activated kennethreitz/maya#139 (Skip pipenv run in tox.ini ...)
• fix pendulum>=1.0 in setup.py (was missing version) kennethreitz/maya#145 (fix pendulum>=1.0 in setup.py: version was in Pipfile but was missing in setup.py)
• Clean Pipfile, remove requirements already stated in setup.py kennethreitz/maya#143 (PR showing how pipenv issue broke whole Travis run)
• Refactor pipenv usage according to semi-official best practices. kennethreitz/maya#144 (PR Refactor pipenv usage according to semi-official best practices)

[uranusjr]

I love this too. Maybe we should add this to Pipenv’s documentation, or even the Python Packaging User Guide.

[tsiq-oliver]

The corollary of the above advice appears to be "forego deterministic/reproducible CI builds", which strikes me as a very large anti-pattern.

What are you proposing as an alternative which would still allow for determinism?

[vlcinsky]

@tsiq-oliverc Deterministic builds have their place at the moment, an application is to be built.

Imagine following attempt to perform really deterministic builds of python library:

• builds have to be based on Pipfile.lock
• each execution context (combination of each python and OS variant) may have different Pipfile.lock resulting from library abstract dependencies defined in Pipfile
• Repository would have to provide separate Pipfile.lock instances defined in the repository. Note, that building Pipfile.lock automatically during CI build does not add any determinism

This is a lot of extra effort. And what you get is a library, which will be installed in different context (e.g. a week later standard installation will pick up upgraded dependency or two) and which will not get anything from the fact, you used Pipfile.lock, which is at the moment obsolete.

The conflict is in the fact the library must never define strict dependencies inside.

If you think, there is another alternative to gain deterministic builds for python library - describe it.

[tsiq-oliver]

@vlcinsky - If a consumer of your library uses different versions of dependencies, etc. then that's out of your control. So I agree there's no feasible way for a library maintainer to manage that.

But the goal here is presumably much smaller scope. In particular, I'd see the goals for a library maintainer as the following (which are roughly equivalences):

1. If you run your CI twice, you're guaranteed to get the same result (network issues notwithstanding!).
2. You can locally recreate (and thus debug) behaviour you observe on CI, even if that means running Docker/etc. locally.
3. You can confidently say "My library behaves as expected with dependency versions X, Y, Z" to your consumers.

If any of those three things don't hold, it strikes me as antithetical to quality control.

So yes, I'd say that if you guarantee to support Python variants A, B and C to your consumer, and they behave differently enough that one lockfile (etc.) doesn't cut it, then you should have three lockfiles (or whatever).

I haven't used Pipenv enough to know how easy that would be in practice, though.

[Moritz90]

I'm currently considering adding Pipfiles to a few library projects for the CI system as well.

I absolutely need the dependency locking (+hashing) for complying with company-wide security guidelines and I currently don't need to test with different Python versions, since there's only one that's officially supported. And the fact that pipenv simplifies setting up a local development environment, including the virtualenv, is a nice side-effect.

And what you get is a library, which will be installed in different context (e.g. a week later standard installation will pick up upgraded dependency or two) and which will not get anything from the fact, you used Pipfile.lock, which is at the moment obsolete.

This is not universally true. In the world of enterprise software, you still have very specific environments that are officially supported and a security issue in a dependency results in your product being updated rather than the customer updating the dependency themselves.

(Yes, I'm talking about a library, not an application here...)

[vlcinsky]

@Moritz90 your scenario is for python library in enterprise environment and there pipenv may help because it is much more deterministic environment.

My description is aiming at general python libraries such as flask, request, maya etc. where the context is much more variable. Trying to fix couple of things in maya I got frustrated learning, that in many cases usage of pipenv introduced real problems (typically hiding problems which would be normally detected) while not providing much or any added value.

Getting deterministic builds is good thing, but it incurs costs. And if done wrong, you may pay extra for lower quality result - and this is what I wanted to prevent.

[uranusjr]

I’d argue this is one of the instances we don’t want the builds to be absolutely deterministic. If you don’t pin your dependencies with ==, you’re committing to maintain support to multiple versions by default, and should design the library that way. A dependency upgrade breaking the build on CI is actually a good thing because it exposes a bug in the library. Completely deterministic dependencies (as managed by Pipenv) would mask that. It would still be beneficial to be able to be determinitic when you want it, that is generally not the best.

[tsiq-oliver]

@uranusjr - Sure. I agree that if the desire is "non-deterministic builds", then the advice up top may well make sense. In fact, it's almost a logical equivalence, and could be stated much more succinctly: "If you don't want deterministic builds, then don't use a tool (pipenv) whose purpose is to ensure deterministic builds" 😄.

But that's certainly not a desirable goal in general.

[vlcinsky]

@tsiq-oliverc nice scope definition - it supports focused discussion. I would add one more requirement: The CI determinism shall not hide possible issues within tested library.

If we use Pipenv.lock, create virtualenv based on that and run CI test of the library, we did part of the functionality the library is supposed to do - installing proper dependencies. If the library is somehow broken in this regard - preinstalled environment would hide this problem.

To me it seems more important to detect issues within a library than to run CI in deterministic way. If there is a way to do both (e.g. running the test behind private pypi index, which could also support determinism) I have no problem, but if there is a conflict, I have my priorities.

Do not take me wrong: there is no desire to run non-deterministic builds, my desire is to run CI builds, which will detect as much issues as possible.

[Moritz90]

@vlcinsky Sure, I just wanted to share my experience to make sure that the updated documentation reflects it as well. The current documentation does a great job at explaining the tradeoffs:

For libraries, define abstract dependencies via install_requires in setup.py. [...]
For applications, define dependencies and where to get them in the Pipfile and use this file to update the set of concrete dependencies in Pipfile.lock. [...]
Of course, Pipfile and pipenv are still useful for library developers, as they can be used to define a development or test environment.
And, of course, there are projects for which the distinction between library and application isn’t that clear. In that case, use install_requires alongside pipenv and Pipfile.

(Highlighted the part that applies in my case.)

I just want to make sure it stays that way. I think your original post contains too many blanket statements without a disclaimer that you're talking about an open-source project that's going to be published on PyPI.

[vlcinsky]

@Moritz90 I completely agree. I was trying to highlight that focus but I can make it even more visible.

[vlcinsky]

@Moritz90 I added introductory note reflecting your comment.

[tsiq-oliver]

@vlcinsky - That makes sense. I understand that you don't explicitly want non-deterministic builds, but I think that it's unavoidably equivalent to what you do want (i.e. to catch issues when your upstream dependencies update).

Thinking out loud, what's the best way to resolve these two conflicting goals? One possibility is to have a two-phase CI process:

1. The deterministic phase. Leverages a Pipfile.lock in your repo, so it's entirely reproducible.
2. The non-deterministic phase. Runs pipenv update and then runs the tests, so that it pulls in the latest of all your dependencies (which is basically the same as the behaviour with no lockfile, I think?).

[vlcinsky]

@tsiq-oliverc To get deterministic builds, I would think of following setup:

• build pypi cache job: run once upon a time, producing some form of pypi index cache (as directory of files or anything similar)
• library testing job: using the pypi cache, but avoiding pipenv

Using pipenv to do the installation is similar to what installing the library itself shall do but it is definitely different because it is different code doing the work.

build pypi cache job

$ git clone <repo_url> <project_dir>
$ cd <project_dir>
$ pip install pipenv
$ $ # clean pypi cache and make it ready to cache somehow - not described here
$ pipenv install -e .[test]
$ # if we need extra testing packages in pipenv
$ pipenv install <extra_test_packages>
$ # record current requirements expressed in `Pipfile.lock`
$ pipenv lock
$ # if needed, record the `Pipfile.lock` somewhere

Outputs of such job are:

• Pipfile.lock as recorded dependencies (may help developers to reproduce the environment easily)
• pre-populated local pypi cache

library testing job

there are phases:

• configure environment to force tox, pip etc. using only our local pypi cache
• run the CI tests (avoid using pipenv)

what we get

• library is tested in deterministic environment
• library is tested incl. it's ability to install itself on it's own
• Pipfile.lock records pypi packages which were used to install the library. It can be used to reproduce environment at developer site.
• adaptation to upgraded packages on (possibly external) pypi is simple (rerun the "build pypi cache job) and is done in controlled manner (the content of pypi is recorded incl. hashes)

Another advantage is, this setup does not require developers maintaining Pipfile or Pipfile.lock. Also running the tests in different contexts is always the same (Pipfile.lock is always rebuilt in given context).

What is still missing (and can be done)

The pypi cache is the part which needs some research. I guess, simple directory would be sufficient and maybe pipenv is already ready to help with that. Maybe issue #1731 is the missing part.

[techalchemy]

As a package that does dependency resolution many of our own tests rely on deterministic builds — that is, taking known stuff and expecting a resolved graph. We use pytest-pypi for this.

Love the lively discussion on this topic. I think the nuance is important and you should always test against known dependencies as well as unpinned ones

[Moritz90]

you should always test against known dependencies as well as unpinned ones

I second this suggestion. It's a good idea to always have an explicit "known good state" for reproducible builds and to simplify debugging in case an update breaks something in addition to making sure that newer minor/bugfix versions work as well.

(In my very personal opinion, the ideal situation would be that the package manager installs the latest minor versions by default so that libraries can always specify the concrete dependency versions that they were tested with, but I realize that's a highly controversial opinion and requires everyone to follow semver.)

[vlcinsky]

@Moritz90 @techalchemy @uranusjr @tsiq-oliverc

Here is my summary from previous discussion.

Particular problems and proposed solutions

Many execution contexts - who shall maintain Pipfile.lock file(s)?

Each supported OS and python interpreter contribute to matrix of possible execution contexts.

E.g. Flask supports (at least CI stuff visible in the repository):

• OS Windows (python 2.7 and Python 3.6)
• Linux (python 2.7, 3.4, 3.5, 3.6, nightly, pypi)
• OSX (py - not sure if there are more versions)

It makes 9 different execution contexts which may differ.

Each execution context may have different Pipfile.lock.

Who shall maintain them?

Options are:

• Let developers do that manually (NO WAY)
• Maintain only one Pipfile.lock for main development platform (which platform enjoys to be ignored?)
• Automate creation via CI (YES)

Proposal: Let CI generate the file by pipenv install -e .. Do not include it in repo, help developers to pick proper Pipfile.lock as result of automated builds.

Developers need predictable environment

When fixing an issue, which may be caused by changes of dependencies on pypi, developer may need simple mean to reproduce the environment from the failing test.

Proposal:

• for many packages, the pypi dependencies changes are so rare, that they are not real problem
• to fix the environment on it's own, developer may generate the Pipfile.lock by pipenv install -e . followed by pipenv lock.
• to replicate environment from failing test, developer pick the Pipfile.lock from the failing test.
• todo: show examples, how to apply Pipfile.lock in tox.ini.

CI must reveal broken setup.py

Library setup.py may be broken (missing dependency in install_requires, missing version specifier etc.) and CI test must not hide such problem (by preinstalling omitted dependencies on it's own).

Proposal:

• trust pipenv install -e . to provide the same result as plain installation (there are currently some issues with that).
• run plain installation test (without pipenv) and possibly compare that resulting pip freeze output is subset of what is installed by pipenv.

Updated pypi dependencies may break things, CI shall detect such chases

Some dependency update may break library using it. CI shall detect failure on such problem.

Proposal:

• at least one test must run against unpinned version
• if CI always generates new Pipfile.lock, this is non-issue (as we run in unpinned mode anyway)

CI modes for different library types

In all proposed modes I tried to avoid keeping pipenv files in the repository saving developers from maintaining this really complex stuff (automation!!!).

In contrast to my original text the 2nd and 3rd mode do use pipenv in CI scripts.

Mode: Run, Forrest, Run!

Simple package with smaller number of dependencies which are not changing often.

Simply run as before the pipenv era and keep things simple to us.

Rare cases when dependencies will make troubles are easy to fix and do not justify making CI more complex.

Mode: Generate and seal

Each time the CI test is run, generate new Pipfile.lock which completely describes the environment used at the moment.

The Pipfile.lock shall become CI artefact.

If things go wrong, developer can pick Pipfile.lock from broken build, apply it locally and do the testing and fixing.

If someone wants to deploy, Pipfile.lock from last successful build can be used.

Mode: Ice Age

When changing dependencies are real problem, CI shall create Pipfile.lock once upon a time and keep it using for certain period (a month?).

This makes CI setup more difficult, as there must be at least two different jobs (one generating Pipfile.lock, the other one applying it and using in tests).

Warning: Pipfile.lock must be update also at the moment, setup.py changes dependencies.

Note, that the Ice Age requires Scrat the squirrel type of test which ignores the frozen status and checks against unpinned versions.

Closing remarks

As seen, the determinism and complexity grows mode by mode.

My proposal would be:

• start simple ("Run, Forrest, Run"). You gain efficiency and speed.
• if things get too complex because of changing dependencies, move on to "Generate and seal". You gain repeatibility in local environment.
• if things are really bad, go "Ice Age" mode. You gain (temporary) determinism.

All gains cost something.

[tsiq-oliver]

If the goal here is to update the advice in the docs, then honestly it feels irresponsible to say something dramatically different to "Follow best practice (reproducible builds) by default, until you have no choice."

[Moritz90]

@vlcinsky Under the headline "Mode: Generate and seal", it might make sense to mention that the last successful Pipfile.lock should always be kept around, e.g. by declaring it as a Jenkins artifact. With that change, it would be fine to recommend that setup for most projects. Like @tsiq-oliverc, I wouldn't recommend the first mode, ever.

[Moritz90]

The more I think about it, the more I feel like this documentation will become a section on why using pipenv for CI builds is a great idea, even if you're developing a library.

[vlcinsky]

@tsiq-oliverc vast majority of general python packages are in mode "Run, Forrest, Run". I have helped few of these packages with introducing tox and pytest, because I felt it would contribute to given package quality and because I had quite clear idea, how it could be done well.

Now there is another great tool and I wonder how to use pipenv properly in general python projects to contribute to it's quality. I want to find one or two well working recipes which are justified and easy to follow.

What would I say to Flask project?

1. Follow best practice (reproducible builds) by default, until you have no choice?
2. Add 9 Pipfile.lock files and set up policy for updating them?
3. Refactor CI scripts for Travis and Appveyor to work in two phase manner following the Ice Age mode?
4. Modify CI scripts for Travis and Appveyor to generate Pipfile.lock artefact for cases, when someone need to reproduce failing test on it's own computer?
5. no comments, apart from "Thanks a lot for Flask."

The goal is to find functional working style. If it ends in doc, nice, if not, no problem.

[Moritz90]

@vlcinsky I'd say (1) and (4) should be the recommendation for such projects. While without a pre-existing Pipfile.lock you won't know the versions used in the build in advance (which is fine outside corporate environments), you'll still get a reproducible result if you generate and archive the lock file during the build.

Edit: The tl;dr version of my recommendation would be:

• Always make sure your builds are reproducible, regardless of whether you're developing a library or an application. pipenv can help you achieve this goal.
• If you're developing an application, commit Pipfile.lock to your repository and use it for deployment. (This is already covered by the existing documentation.)
• If you're developing an open-source library, generate Pipfile.lock on-the-fly in your CI build and archive it for later.
• If you're developing a library and working in a restrictive corporate environment, maintain the appropriate number of lock files and use those in your CI builds.

(Of course, the actual documentation should have a bit more detail and examples.)

[vlcinsky]

@Moritz90 I modified the "Generate and Seal" as you proposed.

Re (1): easy to say, impossible to execute without being more specific.

Re (4): yes, I also think, that "Generate and Seal" is most feasible mode. But in case of Flask I will not dare (at least not at the moment).

Re pre-existing Pipfile.lock in enterprise environment: It has to be created somehow, either (semi)manually or automatically. I guess, in corporate environment you do not install directly from public pypi but use some private one and that provides only approved packages (devpi-server provides great service in that - multiple indexes, controlled volatility of published packages, approvals for external packages etc.) If the process of building Pipfile.lock runs in such environment, it can only use what is approved so if new version is to appear there, someone has to stand up and make it approved. Following CI build will test that it does not break things. And with pipenv check test of security issues may be also automated.

I guess such a workflow would be more secure compared to someone creating it (semi)manually. But my knowledge of enterprise environment is very limited.

[gsemet]

Hello pipenv team. I do share a lot of what is said in this text, it helps a lot any developer better understand the limitations of Pipfile/pipenv when developing a library. I do want to see this text or part of this text integrated inside the official pipenv documentation.

I do have a the following amendement I would like to discuss:

For our internal python package, fully reusable, published on our internal pypi, etc, and even for my own python packages (ex: cfgtree, txrwlock, pipenv-to-requirements), I use a package that some may already know or even use, that abstracts these details and make life of python developer easier: PBR.
PBR basicaly reads requirements.txt found at the root folder of a distribution package and inject it into the install_requires of the setup.py. Developer simply needs to maintain a requirements.txt with loose dependency declarations. Until the support of Pipfile is integrated officially inside PBR, I have to use pipenv-to-requirements that automatically generates requirements.txt from Pipfile so that they are both synchronized and both commited in source, and PBR does the injection correctly after the distribution package has been built. I think one could use pipenv to generate this requirements.txt

I work on a support of Pipfile for PBR, so that it will be able to read the Pipfile (and not the lock file) and inject it into install_requires like it does with requirements.txt.

I do not know if other similar packages exist, because it also does other things people might not want (version from git history, auto generation of AUTHORS and ChangLog).

But at the end, I really feel it is so easier to write, maintain and handle versioning of a Python library, I would be sad not to share this experience. I am promoting it as the "recommended" way of writing modern python libraries on my company.

I do recon that it is like "cheating" on all difficulties about library and pipenv, but at the end the work is done and developers are happy to use it so far. Part of the python training, I am giving to new python developer in my company, involves, first writing a python library maintaining install_requires manually, and then switching to PBR to see how it becomes easier (and frankly I am a fan of the semantic commit feature of pbr to automatically create the right semver version tag).

Part of the reason to declare the libraries dependencies using a dedicated file also for libraries is to be able to use tools such as readthedocs or pyup (even if pyup makes more sence when linked to an application).

I do not necessarily want to promote this method as the "standard" way of doing python package, it is actually the "OpenStack" way, but I would to share my experience, and if others have similar or contradictory experience, I'll be happy to ear them and update my point of view.

Team, what do you think of a kind of "community" section on the documentation? So that users like me can share his experience on how he uses pipenv, without necessarily the full endorsement of pipenv team?

PS: I can move this to a dedicated issue if you do not want to polute this thread

[tsiq-oliver]

@vlcinsky (1) is very easy to execute - put your lockfile in your repo.

I think what you instead mean is: it's impossible to give specific advice once this basic strategy is no longer sufficient. That's certainly true, but that's because the specific problem probably differs on a case-by-case basis.

Or to put it another way, the solution depends on what additional guarantees you want your CI workflow to provide.

[vlcinsky]

@gsemet you know what? All my python packages created in last two years are based on pbr - it is really great. And I follow your attempts to support Pipfile in pbr whenever I can (some thumbs up, votes etc).

In case of this issue (searching for pipenv patterns and antipatterns for general python libraries) I intentionally omitted pbr for two reasons:

• it would make conceptual discussion more complex
• some people do not like pbr for other reasons (you mentioned them) and it would sidetrack the discussion probably

On the other hand, I am really looking forward to a recipe of yours for pbr-lovers. I will read it.

[vlcinsky]

@feluxe

Sorry if this wasn't clear. My lib-dependencies are meant to be what people currently put into setup.py / install_requires. Maybe pypi-dependencies would be a better name for what I meant.

See pbr discussion in this issue. It is the effort to support library dependencies by Pipfile.

I think, that one Pipfile shall not be used for two purposes (lib and app), these things shall be done separately. If you feel it is really needed, could you describe purpose of a project using it? I usually try to keep library development and deployment projects separated as they have quite different usage in time.

[tsiq-oliver]

@vlcinsky I'm not really sure where you want to take this (I'm not sure what kind of PR you're asking for !), so I'm going to bow out of this conversation for now.

To restate the TL;DR of my position:

1. Deterministic builds are highly desirable, commonplace elsewhere in the software industry, and easy to achieve with Pipenv.
2. There are definitely some edge cases, and someone should push the state-of-the-art forward there (via workarounds or via better tooling).
3. It would be irresponsible for the Pipenv docs to blanket recommend against (1) simply because (2) affects a small subset of cases.

---

@uranusjr Got it. Though I don't think there's anything language-specific here, it's simply that different communities have settled on different heuristics for dealing with a problem with no generic solution - if you have version conflicts, you have a problem.

Maven/Java (for example) forces you to think about it at build time. The NPM way means you have runtime issues if the mismatched versions cross an interface. Runtime resolution (e.g. Python, dynamic libraries) means that a dependent may crash/etc. if the dependency version is not what it expected.

[feluxe]

@vlcinsky

See pbr discussion in this issue. It is the effort to support library dependencies by Pipfile.

pbr seems nice and all, but it falls under the category that I was trying to address with this:

I think something like this would omit the need for a chunk of anti-pattern sections and third-party tools to fill the gap.

I think such tools shouldn't be necessary in the first place.

If you feel it is really needed, could you describe purpose of a project using it? I usually try to keep library development and deployment projects separated as they have quite different usage in time.

When it comes to pypi packages, I ended up using Pipenv for handling dev-dependencies, Pipfile to describe dev-dependencies, setup.py to describe lib dependencies with install_requires and setuptools in setup.py to publish my package running pipenv run python setup.py bdist_wheel upload . This is what I consider complicated.

In other modern languages I have to learn one command line tool (package manager) plus one dependency file format. Documentation is in one place and easier to follow and a newcomer will get all this sorted out in a couple of hours. It's a matter of npm init, npm install foo --dev, npm publish. Pipenv/Pipfile can do most of it already, if it could do all of it, issues such as this one would not exist.

[gsemet]

I reinterate my call for a kind of "community" section/wiki for this discution. There are several "pattern" can are legit and some of us might want to share it "way of doing python libraries", some like me with pbr, and other might have a very good pattern. But a page inside the pipenv document, no sure if it is a good idea.

PS: to prepare migration to the new pypi, you should use twine and not python setup.py upload. Using "upload" should be considered as an antipattern.

Maybe pipenv can grow a "publish" commands ?

[zface]

@feluxe You might want to take a look at poetry. I just stumble across it and it seems that it's what you are looking for.

It does what pipenv does and more and it seems that they do it better especially regarding dependency management (at least that's what they pretend). It does dependency management, packaging and publishing all a single tool poetry.

I wonder if pipenv and poetry could gather effort to finally give Python a true package manager.

[uranusjr]

I want to reiterate myself again before this discussion goes too far. Pipenv cannot simply grow a publish command, or do anything that tries to take over the packaging duty. This would only fragment the ecosystem more because not everyone does it this way, and with app and lib dependencies being theoretically different, you cannot tell someone to merge them back together once the distinction is made in their workflow.

It may seem almost everyone is onboard with this merge , but the truth is there are a lot more people not joining this discussion because things work for them and they are doing something else. I’ve repeatedly said it: Discussion about improving the design of toolchains and file formats should happen somewhere higher in the Python packaging hierarchy, so it receives more exposure to people designing more fundamental things that Pipenv relies on. Please take the discussion there. There is no use suggesting it here, because Pipenv is not at the position to change it.

[Moritz90]

I’ve repeatedly said it: Ddiscussion about improving the design of toolchains and file formats should happen somewhere higher in the Python packaging hierarchy, so it receives more exposure to people designing more fundamental things that Pipenv relies on.

I agree that the discussion on this bug spirals out of control now that packaging and publishing came up (this bug is only about dependency management!), but could you please point us at the right place to have this discussion? People are having it here because pipenv is seen as a much-needed step in the right direction, not because they want to impose additional responsibilities upon the pipenv maintainers.

Edit: Sorry, I must have missed the post in which you did exactly that when reading the new comments the first time.

Within the scope of this proposal, however, I would suggest it to focus on the currently possible best practices, instead of going into the rabbit hole of working out a new workflow for the whole Python packaging community. It’d be more productive to propose a best practice within the current constraints, and then start the discussion for improvements.

I very much agree with this. We should first figure out what the best possible workflow for library maintainers is right now before we come up with big plans. So let's focus on that again, as we did at the start of this thread. I don't think we've reached a conclusion yet.

[Moritz90]

Back to topic: Quoting @uranusjr's post about why dependencies should be defined in a different file for libraries:

Another way to work around this is to do something clever in the packaging tools that “unpins” the dependency version. Bundler (of Ruby) sort of does this, by recommending people to not include the lock file into the gem, so Bundler can use the unpinned versions in Gemfile, instead of pinned versions in Gemfile.lock. But people tend to ignore advices and do whatever they want, so you still get pinned versions everywhere.

I was probably a bit too strong to say that it simply won’t work. But at least all previous tries have failed

I still don't see why the official recommendation for libraries for now cannot be to use pipenv for their CI builds, but keep the Pipfile.lock out of source control. Since, as a few people pointed out, pipenv doesn't currently have anything to do with the packaging process, we shouldn't run into the problem you outlined above.

And I also don't see why this is an argument against defining your abstract dependencies in the same file that applications use to define their abstract dependencies. It's okay if pipenv doesn't want to implement an elaborate solution for integrating the Pipfile with setup.py, but I don't see why that's a bad idea in general.

[Moritz90]

@vlcinsky

I think, that one Pipfile shall not be used for two purposes (lib and app), these things shall be done separately.

See my above post. Could you please elaborate on why you think that? I simply cannot see any downside in principle. Right now, it might be a bad idea to include a Pipfile, since you'll then have to define the dependencies in the same way in two different files, but I haven't yet seen any argument that explains why it would be a bad idea to use the Pipfile for dependency declarations in general.

Note that I've already agreed that Pipfile.lock should not be in source control for libraries unless you're in the same situation I'm in.

Edit: Also, if it turns out that pipenv itself actually needs to know about the difference, you might just introduce something like cargo's crate-type field before you start introducing app-dependencies and lib-dependencies - that sounds overly complicated.

[uranusjr]

@Moritz90 Several of Python’s mailing lists would be good venues to hold this discussion.

pypa-dev is the most definite for discussions centring Python packaging, and the ecosystem around it. I’d probably start here if I were to post a similar discussion.

python-ideas is a place to get ideas discussed, and has quite high visibility to the whole Python community. It would also be a good starting point if you want to push this to the PEP level (eventually you would, I think).

[vlcinsky]

@tsiq-oliverc

By PR I mean: show an example proving your concept viable.

So pick up some existing library, fork it, apply your (1) - you say it shall be easy with pipenv and show me. I tried pretty hard and have difficulties.

If your (2) means "someone else has to do the work", your PR will not exist.

In (3) you talk about "small subset of cases" without giving any real number. Are all the top libraries I described in regards to number of virtualenvs considered "small subset"?

[vlcinsky]

To conclude this discussion, I created short summary of what was found during discussion.

Focus: pipenv (anti)patterns for python libraries and applications

I changed the focus a bit: it talks not only about (general) python libraries, but also about applications as it was rather cheap to include it and it demonstrates the differences well.

I intentionally excluded anything proposing changes in existing tooling such as pipenv, tox etc.

What is pipenv and what it is not

• it is deployment tool, allowing to define and apply concrete dependencies by means of Pipfile.lock.
• it is virtualenv management tool.
• it is NOT packaging tool in the sense of generating python package.

Libraries and applications

The (python software) product is either ready to be used in another product (thus library) or it is final application ready to be run.

Personally I think, even "enterprise libraries" fall into library category (the same rules apply, only number of execution contexts is smaller).

Types of software products

• library: to be used in another product (library or an application)
• application: to be deployed and run

Installation methods:

• library: pipenv install <package> thus "get the package into play (resolving versions for other libraries around)"
• application: pipenv sync thus "apply concrete dependencies"

Abstract and concrete dependencies

software product dependencies

• abstract dependencies: must name used libraries, may restrict versions or usage, but must remain flexible enough (shall not pin versions)
• concrete dependencies: must pin versions, ideally with hashes of used libraries

pipenv artefacts:

• Pipfile: abstract dependencies
• Pipfile.lock: concrete (locked) dependencies"

Execution contexts

Typical number of different execution contexts

• library:
  • python virtualenvs on one OS: 3 to 9 (tornado using 30)
  • number of OS: 1 to 3 (Linux, OSX, Windows)
  • total number: 3 to 18
• application:
  • python virtualenvs on one OS: 1
  • number of OS: 1
  • total number: 1 (or very few)

CI goals, priorities and determinism

CI goal

• library:
  • the code incl. it's abstract dependencies allow installation and expected function within all expected variants of execution contexts.
  • when (private/public) pypi get dependency library update, fail, if it affects tested library installation of function.
• application:
  • when installed (using concrete/pinned dependencies), all expected functionality is provided within one, pre-defined execution context

Particular CI goals regarding functionality:

• library:
  • abstract dependencies declared by the library are complete and include all necessary restrictions (only where needed): library installs itself properly
  • all expected use cases function properly
• application:
  • concrete dependencies are complete and all are pinned, best incl. hashes: application installs properly
  • all expected use cases function properly

Different CI test modes

• Mode: "Run, Forrest, Run"
  • Vast majority of python libraries nowdays are tested this way.
  • use tox or similar testing SW
  • No usage of pipenv and concrete dependencies (can be fine for libraries)
• Mode: "Generate and seal"
  • No Pipfile in repository
  • Pipfile.lock created by pipenv install -e .
  • Pipfile.lock documents (seals) the environment and allows later reproduction of virtualenv for analysing issues.
• Mode: "Ice Age"
  • two phase test
    • when abstract dependencies (defined within setup.py install_requires) change or dependent package on pypi is updated, regenerate Pipfile.lock by pipenv install -e .
  • function test run: run when library code changes. Is run within virtualenv created by pipenv sync Pipfile.lock

How and by whom can be Pipfile.lock created

• manually by developer (may work for applications)
• automatically by CI build (and passing all tests, declare it verified artefact)

Priority of determinism versus flexibility

• library: flexibility (run whenever possible)
• application: determinism (run in exactly the same way within selected execution context)

What can affect determinism of installed product:

• public pypi (low determinism, packages are updated at any time)
• private pypi (higher determinism, package updates may be controlled)
• abstract requirements within libraries (shall not be used for determinism)
• concrete requirements (Pipfile.lock): total determinism

Miscellaneous

Some use cases for Pipfile.lock:

• (antipattern) define library abstract dependencies (because it must be abstract)
• (antipattern) set up virtualenv for tested library (may hide broken library abstract dependencies)
• document exact virtualenv ("seal"), where a test was run (thus allow developer to recreate it later for broken tests and experiment in it)
• set up virtualenv for tested application
• deploy application into production

Other hints

• pbr library allows definition of abstract library dependencies via requirements.txt. Update reading Pipfile is on the way.
• poetry package tries something similar to pyenv

Common issues

• "drop lockfile into repo" and you get deterministic builds:
  • Shall work for applications.
  • Will not work for libraries, as there are really numerous executions contexts and each has good potential to result in different Pipfile.lock. Seriously: flask shows on it's 11 different virtual envs (on one OS) 10 different locked dependencies. Who is going to create and commit them?
  • Note, that with "Generate and seal" CI mode, you can still get Pipfile.lock (but generated by CI script) allowing to regenerate the virtualenv elsewhere.
• Pipfile.lock in library repository
  • if used to create virtualenv, it can hide broken definition of library dependencies within setup.py.
• Pipfile in library repository
  • if it repeats abstract dependencies (which are to be defined within setup.py), it may hide broken setup.py dependency declaration.
  • recommended: Generate Pipfile by pipenv install -e . or pipenv install -e .[tests] if you need also test dependencies and they are declared as "tests" extras in the setup.py
• adding pipenv install <something> into CI scripts
  • it does not improve determinism much on it's own
  • see "Generate and seal" CI mode (then all installation into virtualenv must go via pipenv).

Conclusions

Python libraries (especially general ones) exhibit unexpectedly high number of execution contexts. The reason is, that with libraries the goal is proven flexibility under different conditions. The flexibility seems more important over deterministic builds. For people coming from "compile" world this might feel like very bad antipattern. The fact is, most (possibly all) python libraries do not provide deterministic builds (if you are aware of some, let me know) and Python is still doing very well. The reasons Python applications are stile alive might be: python as scripting language differs from compiled world. The other reason could be, that determinism can(shall) be resolved a step later as soon as and application (build from set of libraries) shall resolve the (natural and justified) requirement for determinism.

For applications the situation is just opposite and here determinism is really easy to reach with tool such as pipenv.

What to do next?

• I will not have time to deal with this for next week or two.
• I can imagine creating series of blog entries somewhere (not sure where). If you know the place (ideally allowing some discussion), it would be natural place to let the content be referenced, discussed and finally possibly (if it survives) put somewhere in stone :-).
• I propose @uranusjr to take over control of this issue (close it, decide what to do next, redirect people elsewhere or whatever seems practical)

Thanks everyone for very inspiring discussion - I feels to me as a message "I am totally lost in this topic" refactored three times - what means naturally we got better.

[zface]

@vlcinsky poetry has nothing to do with pyenv. It's a lot like pipenv (but with a much better implementation regarding the management of libraries and applications, IMO) but with the packaging and publishing part.

You have a pyproject.toml file that defines your project and its dependencies (abstract dependencies) and a pyproject.lock which describes the pinned dependencies and are pinned for any python version and platform the pyproject.toml file has specified in order to have only one deterministic lock file to avoid the problems that pipenv is facing. Only when installing, poetry will check which packages to install by checking them against the environment.

And when it packages your library, it will use the abstract dependencies (and not the pinned one) so you keep the flexibility when distributing your package (via PyPI for example).

Tha advantage of this is that it will use abstract dependencies for libraries and the lock file for applications. This is the best of both worlds.

[techalchemy]

@zface poetry not using pinned dependencies is literally defeating the entire purpose. Pipenv is idempotent and this requires reproduction of an environment. Please stop using this issue as a platform to try and sell everyone something that has listed as its first reason for why to use it over pipenv that the author doesn't like the cli. At the end of the day, our software is deployed across hundreds of thousands of machines and actually acknowledges and uses the best practices around packaging. If you don't want an idempotent environment and you do want to blur the lines between development and packaging please don't participate in this discussion because we are not moving in that direction and it will not be productive.

[techalchemy]

Essentially we spend a lot of time and effort on resiliency that small projects which make lofty claims don’t have to spend as much effort on because people aren’t hitting edge cases. If you truly believe that another tool offers you the best of all worlds then I encourage you to use it— pipenv itself is not going to handle packaging for you in the near term, if ever.

[zface]

@techalchemy I am not selling anything, really, I am merely directing towards ideas that could be used in pipenv.

And poetry does pin dependencies in the pyproject.lock, just like pipenv does in Pipfile.lock. So you have reproduction just like pipenv provides. If you have a lock file it will be used and install pinned dependency and if I am not mistaken it's also what pipenv does.

The only time it uses abstract dependencies is when it packages the project for distribution (so basically for libraries) since in this case you do not want pinned dependencies.

[uranusjr]

@vlcinsky There are still a few points that need to be sorted out, corrected, or expanded on, but I am still very keen on this going into documentation form, Pipenv or otherwise. Would you be interested in sending in a pull request? I’d be more than happy to help flesh out the article.

[uranusjr]

Regarding poetry, I am not personally a fan as a whole, but it does do many correct things. It should probably not be mentioned in Pipenv docs because it violates a few best practices Pipenv devs want to push people towards, but it should be mentioned if the discussion is held in pypa-dev or similar, to provide a complete picture of how the packaging ecosystem currently is.

poetry can also use more attention and contribution. This would be the best for the community, including Pipenv. With viable choices, people can weight on their choices instead of going into Pipenv head first and complaining it not doing what they expect. Good competition between libraries can also spur forward technical improvements in the dependency resolution front, which Pipenv and poetry both do (and neither perfectly). We can learn a lot from each other.

[vlcinsky]

@uranusjr Yes, I think few things were clarified and deserve sharing with wider audience. Your assistance is really welcome.

What about "pair documentation drafting"? I think that at this moment it would be most effective to work on it in small scale of two persons only.

Thinks to do are (possibly with one or two iterations):

• where exactly we could publish that
• identify documentation items (articles, sections)
• clarify scope and goal of each item
• agree on outline
• identify open issues
• work them out
• write the doc
• publish (and hope it would be accepted)

If you feel like writing it on your own (based on what was discussed) and have me as a reviewer, I would not complain.

I will contact you by e-mail to agree on next actions.

[uranusjr]

@vlcinsky Also I’m available as @uranusjr on PySlackers (A Slack workspace) if you prefer realtime interaction. Pipenv has a channel there (#pipenv).

[zface]

@uranusjr That's what I meant by gathering effort. Python desperately need a good package manager like cargo. The Python ecosystem pales in comparison with the other languages due to the lack of a standard way to do things. And pipenv will not help with that, I think.

What bothers me is that pipenv advertise itself as the officially recommended Python packaging tool while it's not a packaging tool, far from it, which is misleading for users. It's merely a dependency manager coupled with a virtualenv manager.

Also, you say that it was inspired by cargo, npm, yarn which are packaging tools along with dependency managers while piping is not.

And here is the flaw of pipenv, it just muddies the water since people will still make the same mistakes as before with requirements.txt vs setup.py. Projects will still be badly packaged with badly define dependencies in their setup.py because of that. That's what projects like cargo did right: they handle all the aspects of developing projects/applications to ensure a consistency while a project like pipenv does not.

And when you say:

which Pipenv and poetry both do (and neither perfectly)

What do you mean? From what I have seen, their dependency manager is much more resilient than the one provided by pipenv. The only downside is that they use the PyPI JSON API which sometimes does not have dependency information due to badly published packages.

Anyway, I think, like you said, that both projects can learn from each other.

And, one more thing, what's the future of pipenv if, ultimately, pip handles the Pipfile? Will it just be a virtualenv manager?

[techalchemy]

If the poetry dependency manager relies on the json api it’s not only sometimes wrong due to ‘badly published packages’, it’s going to be very limited in what it can actually resolve correctly. The warehouse json api posts the most recent dependencies even if you’re dealing with an old version, and that’s if it has that info at all. We used to incorporate the json api too, it was great because it was fast, but the infrastructure team told us not to trust it. It seems a bit disingenuous to call something resilient if it relies on an unreliable source to start off with.

Ultimately the challenges are around actually building a dependency graph that executed a setup file because currently, that’s how packaging works. There is just no way around it. A dependency graph that resolves on my machine may be different from one that resolves on your machine even for the same package.

It’s easy to hand wave and say ‘well doesn’t that just make pipenv a virtualenv manager if pip can read a pipfile?’ No. Pipenv is a dependency manager. It manages idempotent environments and generates a reproducible lockfile. I realize this must seem trivial to you because you are waving it away and reducing this tool to a virtualenv manager, but it isn’t. We resolve lockfiles and include markers for python versions that you don’t have, aren’t using, and keep that available so that you can precisely deploy and reproduce across platforms and python versions. We use several resolution methods including handling local wheels and files, vcs repositories (we resolve the graph there too) remote artifacts, pypi packages, private indexes, etc.

At the end of the day pip will handle pipfiles, that’s the plan, it’s been the plan since the format was created. But that is the same as asking ‘but what about when pip can handle requirements files?’ The question is basically identical. Pip can install that format. It’s not really relevant to any of the functionality I described other than that we also install the files (using pip, by the way).

[zface]

@techalchemy

The warehouse json api posts the most recent dependencies even if you’re dealing with an old version, and that’s if it has that info at all

This just plain wrong, you can get a specific version dependencies by calling https://pypi.org/pypi/{project}/{release}/json. If you just call https://pypi.org/pypi/{project}/json sure you will only get the last dependencies but you can actually get the right set of dependencies.

And the packaging/publishing part of python projects really need to be improved because in the end it will benefit everyone, since it will make it possible to use the JSON API reliably.

It manages idempotent environments and generates a reproducible lockfile.
We resolve lockfiles and include markers for python versions that you don’t have, aren’t using, and keep that available so that you can precisely deploy and reproduce across platforms and python versions.

And so does poetry. And you can make it not use the JSON API to provide the use the same resolution method as pipenv (using pip-tools). See python-poetry/poetry#37 (comment) and it will still be more resilient than pipenv (https://github.com/sdispater/poetry#dependency-resolution)

[uranusjr]

@zface I will say this one final time, please take this to somewhere higher in the hierarchy. Pipenv does not self-proclaim to be the officially recommended Python packaging tool; it says that because it is. If you feel that is inappropriate, tell it to the officials that recommend Pipenv. Please do not put these things on Pipenv dev. This is the wrong place to complain, and you cannot possibly get resolutions for your complaints here. You can also get better answers on technical questions you have there. This is an issue tracker for Pipenv, not a discussion board for Python packaging tools and how Python packaging is done.

[techalchemy]

Pipenv doesn't just rely on pip-tools for resolution, please stop reducing our software to one liners that demonstrate a lack of understanding. I know very well how the PyPI api works, I talked directly to the team that implemented it.

This just plain wrong,

This kind of attitude is not welcome here. Do not assume we don't understand what we are talking about. Please practice courtesy.

it will still be more resilient than pipenv (https://github.com/sdispater/poetry#dependency-resolution)

Pipenv does not currently flatten dependency graphs. Pointing to one specific issue where a tree has been flattened and claiming the entire tool is therefore both better and more resilient is foolish, you are proving over and over again that you are simply here to insult pipenv and promote poetry. Please be on your way, this behavior is not welcome.

[gsemet]

I agree the discussion is way off-topic, that was trying to capitalize the "good practices" arround pipenv.

However,

[...] will still make the same mistakes as before with requirements.txt vs setup.py. Projects will still be badly packaged with badly define dependencies in their setup.py because of that.

I share this opinion, getting new developers to successfully package their own Python code is actually complex, too complex, requires to read way to much online documentation.
But it's not up to pipenv or any other package dependency to deal with that entirely. We could not rewrite the history. We, as a community, need to find a way to modernize the python tool chain, step by step.

And pipenv (and probably poetry) is a very good step forward.

Having to maintain on one side Pipfile for application and setup.py for libraries on the other side, is a no brainer. No matter how hard we explain with lot of words and long articles and good practices guides, it's too complex for what it is. I completely agree it is like this for the moment, but it should not prevent us of imagining a better and safer way.
At the end, as a developer, I want a single tool, maybe with two different modes, to help me and make my life as easier as possible.

Their should be a way of extracting just the part that does the requirements.txt/Pipfilefrom libs such as PBR to propose a kind of 'easy setup.py', a Pipfile-aware wrapper arround install_requires, without all the unwanted behavior pbr brings, and package that in a dedicated setuptools wrapper that only does that.

So we would be able to have the better of each world:

• pipenv to maintain Pipfile (versionned in both libraries and applications)
• pipenv to maintain Pipfile.lock (versionned only for applications)
• one would use this magic wrapper package (pipfile_setuptools, install_requires_pipfile ?) that would be a first level dependency which job is only to inject Pipfileinto install_requires.

This another project that would not be related to pipenv, but still needs a generic Pipfile parser library. What do you think?

[uranusjr]

@gsemet From my understanding PyPA has been trying to fill that with pyproject.toml instead, led by flit. You’ll need to talk to them first (at pypa-dev or distutils-sig) about this before proceeding to use Pipfile as the source format. As for parsing Pipfile (and the lock file), that is handled in pypa/pipfile (which Pipenv vendors to provide the core parsing logic).

---

Edit: Please drop me a message if you decide to start a discussion about this in either mailing list. I do have some ideas how we can bring the two parts of Python packaging distribution together.

[gsemet]

I must admit I am a bit sad seeing dependencies declared in pyproject.toml(which takes the roles as setup.cfgdone by PBR), while PyPa also supports Pipfile....

Thanks for the pointer to flit and pipfile. There is also Kennethreitz 's pipenvlib that seems lighter.

PBR's setup.cfg seems more complete compared to the official documentation (ex: data_files) and reuse a file already shared with several tools (flake8, pytest, ...) can use the same file, reducing the number of file at the root of a python project)