feature: generative testenv

[quodlibetor]

I currently have an envlist that looks like {py27,py35}-{test,pylint} to get the maximum number of errors out of a CI run.

To get this to work without duplicating the commands the only thing I can think of is

[test-command]
# note: not in the envlist
commands = pytest ...

[testenv:py27-test]
commands = {[test-command]commands}

[testenv:py35-test]
commands = {[test-command]commands}

[pylint-command]
commands = pylint ...

[testenv:py27-pylint]
commands = {[pylint-command]commands}
... etc ...

This has two downsides compared to being able to do a hypothetical:

[testenv:{py27,py35}-test]
...
[testenv:{py27,py35}-pylint]
...

• It's unnecessarily verbose, and just gets worse as you increase the number of testenvs
• It also doesn't scale with more specializations per env (e.g. for pylint I use skipsdist and usedevelop, but since each of those is simple I have to decide between DRYness and simplicity since they're each one line already)

I'm not sure how complicated this would be to implement, if there is interest and the testenv expansion wouldn't take tons of knowledge of internals I might be able to devote some time to it this week or weekend.

[RonnyPfannschmidt]

@quodlibetor i beleive it woudl already help if factors could apply to more than testenv

then you would only need to declare a test and a lint factor, and use the py* factors to expand

eg

[tox]
envlist = {py27,py35}-{lint,test}

[testenv:test]
...
[testenv:lint]
...

[quodlibetor]

@RonnyPfannschmidt that seems like it should work, but I get a Commands not specified error with both 2.5 and 2.6 when I do that.

[RonnyPfannschmidt]

@quodlibetor well it currently is not implemented that way, i was suggesting a syntax that could work more easyly

[quodlibetor]

Oh! Yes, that was the first thing I tried, so it seems like it would have been obvious.

But, if you do something like that, then how do you deal with complicated env matrixes? For example, specifying some subset of {py27,py35,py36}-{test,lint}-{django1.6,django1.7,django1.11} would (I imagine) require some pretty ugly code to disambiguate which commands apply if you put in just test, whereas requiring something that expands to some set of existing names is more clear.

On the other hand, if you think that api is fine then I could be pretty easily convinced.

[RonnyPfannschmidt]

well, from my pov a sensible way to go is to pick all factors of the test env name, and apply the rest from variables below

there will be cases it wont be able to handle, but that may be a sensible limitation

[quodlibetor]

So my main concern would be the following situation:

envlist = {py27,py35}-{test,lint}
[testenv:test]
commands = ...
[testenv:py35]
commands = ...

which commands key gets selected? Giving a good error message or being able to fail seems like it requires a fair amount of tracking code. If tox is fine saying that that will be undefined behavior, or saying that it's defined but it's not a warning then I suppose that's fine, but it seems like it would be hard to debug.

[RonnyPfannschmidt]

I believe tox should fail if there is more than.one matching env

[daa]

May be I don't quite understand your problem and what you want but I think tox supports this already. Let's look at sample tox.ini:

[tox]
envlist = {py27,py35,py36,pypy}-{lint,test}
skipsdist = True

[testenv]
commands =
    lint: flake8
    test: pytest

Here if you run tox command it will execute flake8 in {py27,py35,py36,pypy}-lint and pytest in corresponding -test environments. Also If you want to add command just for one specific environment, say py35-test, you may add py35-test: some-command to commands parameter.

[RonnyPfannschmidt]

@daa this inevitably turns into a utter mess if you want to control dependencies in different envs and have more than one command

[quodlibetor]

I agree with @RonnyPfannschmidt about using that syntax making complex configs hard to read.

Here is me thinking that it also didn't work, and then realizing that it does work, just not in the way that I expected:
It also doesn't appear to work with multiline or multiple-command commands, which is the main reason that I care -- many of my commands are fairly long.

For an example of the bugs involved, this tox.ini:

[tox]
envlist = {py27,py35}-{test,lint}
skipsdist = true

[testenv]
whitelist_externals =
    echo
    time
commands =
    test: time \
        echo test \
        one
        echo two
    lint: echo lint
          echo another lint

results in some buggy output:

py27-test runtests: commands[0] | time echo test one
test one
        0.00 real         0.00 user         0.00 sys
py27-test runtests: commands[1] | echo two
two
py27-test runtests: commands[2] | echo another lint
another lint

... snip ...

py27-lint runtests: commands[0] | echo test one
test one
py27-lint runtests: commands[1] | echo two
two
py27-lint runtests: commands[2] | echo lint
lint
py27-lint runtests: commands[3] | echo another lint
another lint

... snip ...

In particular, notice how echo test one shows up in both test and lint, but the time only appears in the test...

Oh, wait, this works correctly:

[tox]
envlist = {py27,py35}-{test,lint}
skipsdist = true

[testenv]
whitelist_externals =
    echo
    time
commands =
    test: time \
    test:    echo test \
    test:    one
    test: echo two

    lint: echo lint
    lint: echo another lint

This... works. It will still cause a fairly complex testenv for my configs, but it's better than what I've got now.

I would still like to have either my original request or the first response to my proposal to tie multiple related testenv variables together. For example, I'm currently, I think, going to end up with something like the following (where each ... is possibly multiple lines):

[testenv]
whitelist_externals =
    lint: bash
usedevelop =
    lint: True
    typechk: True
deps =
    test: ..A
    lint: ..B
    typechk: ..C
install_command:
    test: ..A
    lint: ..A
    typechk: ..B
    compile: ..C
commands:
    test: ..A
    lint: ..B
    typechk: ..C

Which is definitely livable, but not as nice as it could be.

[obestwalter]

I would propose to close this as wontfix for the old config format and decide how to implement use cases like this in #999.

[quodlibetor]

oh, yay!

[gaborbernat]

As per @obestwalter suggestion.