virtualenv is not recreated when deps change if '-r' is used

[pytoxbot]

• Bitbucket: https://bitbucket.org/hpk42/tox/issue/149
• Originally reported by: @gavrie
• Originally created at: 2014-01-28T15:36:07.365

In my tox.ini, I have the following deps:

deps = -rpackaging/requirements.txt

Normally, when deps change, the virtualenv is recreated by tox.
However, when the contents of requirements.txt change, tox does not notice this and the environment is reused.

A workaround would be to list the deps directly in tox.ini, but that would require duplicating the contents of the existing requirements.txt that is used by other tools.

[pytoxbot]

Original comment by wbyoung

@wmyll6 thanks for the plugin. Hopefully some of that work can make its way into the core!

[pytoxbot]

Original comment by @signalpillar

Hello guys, I've created an experimental project that supports venv recreation on requirements file update.
Maybe it will be useful for you in some way: https://github.com/signalpillar/tox-battery

Implemented using plugin system that was added recently to the tox.

[pytoxbot]

Original comment by hexadite-maxk

This is rather philosophical question: if we want tox to be smart or stupid.
IF we want it stupid - adding new smart features is out of the question.

[pytoxbot]

Original comment by @merwok

This also happens if dependencies in setup.py (install_requires) change.

[pytoxbot]

Original comment by @hpk42

Within tox-1.9.1 (superseded quickly by 1.9.2) we had some code that parsed requirement files but it was buggy and caused problems.

I am all for adding proper support for dealing with requirement files in deps.

I think that tox will need to support pip==1.5.6 at least if not older versions.

[pytoxbot]

Original comment by @hpk42

If somebody can submit a PR helping with working better with "-r" lines, please do. Note that i just released tox-1.9.1 which contains some code in relation to "--force-deps" which parses requirements.

[pytoxbot]

Original comment by @bukzor

That's only slightly better. Requirement files can and do contain '-r
more.txt' lines to pull in further requirements.

To the dependency concern: tox already has a hard dependency on virtualenv,
which contains a vendored copy of pip. We can make use of this if necessary.

--phone is hard.

[pytoxbot]

Original comment by mhirota

What about recognizing the special case of deps = -r<something> and calculating the md5 based on the file contents? This would trigger recreate if anything in the requirements.txt changed... but I think that might be better than duplicating what is in pip or introducing a hard dependency on pip in tox.

[pytoxbot]

Original comment by @hashar

A way to reproduce:

$ cat tox.ini
[tox]
skipsdist = True

[testenv:venv]
deps = -rrequirements.txt

$ echo -n > requirements.txt
$ tox
...
$ cat .tox/venv/.tox-config1 
00000000000000000000000000000000 /usr/local/opt/python/bin/python2.7
1.8.1 0 0 0
00000000000000000000000000000000 -rrequirements.txt
$

Then add a new requirement and rerun tox:

$ echo pep8 > requirements.txt
$ tox -vvv
$ tox -vvv
  removing /private/tmp/toxdeps/.tox/log
using tox.ini: /private/tmp/toxdeps/tox.ini
using tox-1.8.1 from /usr/local/lib/python2.7/site-packages/tox/__init__.pyc
skipping sdist step
venv start: getenv /private/tmp/toxdeps/.tox/Venn
venv reusing: /private/tmp/toxdeps/.tox/venv
     ^^^^^^^
venv finish: getenv after 0.02 seconds
venv start: finishvenv 
venv finish: finishvenv after 0.00 seconds
...

.tox-config1 remains identical.

VirtualEnv._getresolvedeps() should be made to understand the '-rsomefile.txt' and parse its content, or even better delegate the dependencies resolution to pip itself. pip.req.parse_requirements() has all the logic :-}

[pytoxbot]

Original comment by @asottile

For what it's worth, here's a hacky workaround with a Makefile: http://stackoverflow.com/questions/23032580/reinstall-virtualenv-with-tox-when-requirements-txt-or-setup-py-changes/23039826#23039826

[dimaqq]

Any chance of porting tox-battery here?

[RonnyPfannschmidt]

@dimaqq it already is on github - @signalpillar is the maintainer/owner

[obestwalter]

Are there opinions about how we should deal with this?

I would tend to close this and recommend using tox-battery for now.

[The-Compiler]

I'll have to disagree - this is an issue I hear from various people, and it's quite unexpected. I view it as "tox does caching without invalidating the cache correctly" essentially. The core does caching, so I think it really should do this correctly.

[obestwalter]

o.k. - I stand corrected then :)

[hpk42]

we once had better support for requirements parsing but we had to revert it ... i admit i never extensively used -r requyirements.txt myself and the docs still mark that feature as experimental http://tox.readthedocs.io/en/latest/example/basic.html?highlight=requirements#depending-on-requirements-txt -- someone needs to step up to go for it more properly ... meanwhile recommending to manually use "tox -r" for recreating an env is a kludge to get around tox's ignorance.

[merwok]

FWIW, in my latest project I avoid the issue by using pip-tools. I never recreate virtualenvs, but dependencies (in one or more requirements files, e.g. run.txt and test.txt) are always synchronized, so I get fast failures before commit and CI if I forget to update requirements.

[obestwalter]

@merwok that sounds interesting, but I have no idea what you are talking about - could you link to some documentation/blog/howto/whatever with more details about this approach? Thanks!

[asottile]

@obestwalter one approach (that we're using internally, and will have a better solution for once 2.7.0 ships with my plugin changes :D) is to use https://github.com/Yelp/venv-update -- pip-faster has a --prune which will uninstall extraneous dependencies (and ensure the installed state is exactly what was on the cli)

[ulope]

@obestwalter pip-tools can be used to keep a virtualenv exactly in sync with the requirements.

It's really a great idea to use it in tox. Should have thought of that myself.

[merwok]

It is indeed https://github.com/jazzband/pip-tools

tox.ini:

[testenv]
deps =
    pip-tools == 1.8.1rc3
commands =
    pip-sync requirements.txt requirements-test.txt
    pyflakes project
    pytest {posargs}

pip-sync makes sure new dependencies or updated versions are installed in the virtualenv and removes old dependencies. I don’t duplicate dependencies in tox.ini, I never have to recreate the testenvs, my co-workers never have failures because requirements.txt changed. My deployment also uses pip-sync.

Article from the original developer of pip-tools: http://nvie.com/posts/better-package-management/

pip-tools very recently gained the ability to generate requirements.txt with pinned versions from dependencies in setup.py, so you can have a lib with loose requirements in setup.py and exact pinned versions in requirements-test.txt for your tox testing.

[obestwalter]

Very nice. Thanks for the info. I had build my self built version (something similar to what pip-tools does) for our team projects, that I might be able to throw out now :)

To get back to the original issue: in the light of the fact that this problem is better solved by a tool that can be used by tox maybe it would be better to deprecate the -r option and throw it out in the next major release? I don't use this option either as I had my own solution, but will definitely try pip-tools for this now.

[dimaqq]

I'm very glad to see this discussion coming back to life.

My 2c/experience:

pip-tools is a double-edges sword, it requires a separate pip-compile step, and voila, your one-step build is gone.

venv-update is cool! I think it replicates tox functionality in a way, but conserves resources (and thus shortens build time). Perhaps if it supports same range of interpreters, it could be integrated into tox.

pip-faster is off-topic, can be included in the above.

[gaborbernat]

@asottile can you detail? thanks!

[asottile]

venv-update aims to solve nearly the same thing as pip-tools but does so in a different implementation. Note that while confusing, venv-update and pip-faster are one-and-the-same (there has been some significant confusion in development as we've renamed the project back and forth at least once now!). The usage is almost identical to pip-tools above:

[testenv]
deps = venv-update
commands =
    pip-faster install --prune -r requirements.txt
    pip freeze

The plugin I'm writing works similarly to this ^

Internally, we suggest a dependencyless approach which involves vendoring a copy of venv-update and using it as follows:

[testenv]
commands =
    {toxinidir}/bin/venv-update venv= {envdir} install= -r {toxinidir}/requirements.txt
    pip freeze -l

venv-update currently supports cpython 2.6, 2.7, 3.5, 3.6, pypy -- it is known to work in 3.3+ (and may very well work for 3.2 and below as well as pypy3.2 but we've stopped testing against those due to the pypa group dropping support). It is likely to also work for modern pypy3 (targeting 3.5.x). No attempt to verify its behaviour has been made against ironpython / jython / etc.

There is one outstanding kinda awful bug that is similarly shared by pip itself. pip-tools avoids this by uninstall-then-install whereas venv-update does install-then-uninstall -- it's not clear if there's an easy fix for this (it's kind of a rare case and we haven't invested any time into investigating / fixing).

[merwok]

pip-tools is a double-edges sword, it requires a separate pip-compile step, and voila, your one-step build is gone.

I don’t know what «one-step build» refers to. In my usage, calling pip-compile to add a new dependency or upgrade existing deps is something I do during development when I choose to; the install step in tox testenvs or before starting the app is only pip-sync.

[gaborbernat]

Yes, and it's planned for tox 4 🤔

[WhyNotHugo]

Could tox hash any requirements.txt files that are used and cache that?

That's actually a brilliant idea. It would also make sense to order requirements before hashing, so change in order don't result in an environment recreation.

[The-Compiler]

This approach has been discussed before (probably multiple times by now?) in this thread. The main problem is that requirement files can include others via -r, which still wouldn't be fixed by a "naive" approach. However, I'd argue that even a naive approach which doesn't deal with requirement files including others would be a big benefit.

[twm]

It would also make sense to order requirements before hashing, so change in order don't result in an environment recreation.

Order is significant in requirements files (for now at least — Pip's new resolver may change that).

[janluke]

Yes, and it's planned for tox 4 🤔

Using "-r requirements.txt" in tox is so common that I'm suprised this issue has remained unresolved for >4 years instead of being solved by a hotfix. When is tox 4 planned to be released?

[gaborbernat]

Yes, and it's planned for tox 4 🤔

Using "-r requirements.txt" in tox is so common that I'm suprised this issue has remained unresolved for >4 years instead of being solved by a hotfix. When is tox 4 planned to be released?

It's a fairly complicated issue, it's more like adding a major feature than hotfix. We have an alpha version of it https://pypi.org/project/tox/4.0.0a2/, a first public release expected in a few months.

[JulienPalard]

Hi @merwok, as a user of pip-compile, I was willing to try this today, slighly modified version of you comment from 2017:

[testenv]
deps = pip-tools
commands_pre = pip-sync requirements-dev.txt
commands = coverage run -m pytest

Issue is: pip-sync uninstalls the project being tested :D

Do you still use this "trick"? If so have you found a nice and clean way to workaround this?

[nicoddemus]

Hi @JulienPalard,

I haven't tried with tox specifically, but I usually have this in my requirements.in file when using piptools:

-e file:.#egg=<my project>

(Replacing <my project> with the top level package name of your project).

This tells pip-sync to install . in develop mode.

HTH

[merwok]

Two options:

• some projects don’t need to be installed (e.g. django sites), so pip-sync is fine
• other need to be installed (Pyramid sites and other projects that need entry points), so I run flit install or pip install . from commands section.

[JulienPalard]

Thanks @merwok and @nicoddemus for sharing!

[gaborbernat]

I'm closing this as this has now been implemented in the alpha branch of rewrite. We can open more specific issue tickets on the implementation separately.

[ssbarnea]

@gaborbernat How about adding a fixed-by-tox4 label and also locking the ticket? There are lots of other similar tickets which are solved by it and which will not be addressed by old version. I you give me triage rights, I could help with that.

[gaborbernat]

You got it 👍🏻