skipif propagates through multiple inheritance

[jhamman]

• Include a detailed description of the bug or suggestion

We are seeing some unexpected behavior when using pytest.mark.skipif on test classes with multiple inheritance. Once the skipif decorator is invoked, inhereted classes are also skipped. A reproducible example is below.

• pip list of the virtual environment you are using

# packages in environment at /Users/jhamman/anaconda/envs/xarray36:
#
affine                    2.1.0                    py36_0    conda-forge
alabaster                 0.7.10                   py36_0    conda-forge
appnope                   0.1.0                    py36_0    conda-forge
argh                      0.26.2                    <pip>
attrs                     17.2.0                   py36_0    conda-forge
babel                     2.4.0                    py36_0    conda-forge
blas                      1.1                    openblas    conda-forge
bleach                    1.5.0                    py36_0    conda-forge
bokeh                     0.12.4                   py36_0  
boost                     1.64.0                   py36_4    conda-forge
boost-cpp                 1.64.0                        1    conda-forge
boto3                     1.4.3                    py36_0  
botocore                  1.4.90                   py36_0  
bottleneck                1.2.1               np113py36_0    conda-forge
bzip2                     1.0.6                         1    conda-forge
ca-certificates           2017.1.23                     0    conda-forge
cairo                     1.14.6                        4    conda-forge
cartopy                   0.15.1                   py36_4    conda-forge
certifi                   2017.1.23                py36_0    conda-forge
click                     6.7                      py36_0    conda-forge
click-plugins             1.0.3                    py36_0    conda-forge
cligj                     0.4.0                    py36_0    conda-forge
cloudpickle               0.2.2                    py36_2    conda-forge
coverage                  4.3.4                     <pip>
coveralls                 1.1                       <pip>
curl                      7.52.1                        0    conda-forge
cycler                    0.10.0                   py36_0  
cython                    0.25.2                   py36_0    conda-forge
dask                      0.15.2                   py36_0    conda-forge
dask-core                 0.15.2                   py36_0    conda-forge
decorator                 4.0.11                   py36_0    conda-forge
distributed               1.18.1                   py36_0    conda-forge
docopt                    0.6.2                     <pip>
docutils                  0.13.1                   py36_0  
entrypoints               0.2.2                    py36_1    conda-forge
expat                     2.1.0                         3    conda-forge
flake8                    3.3.0                    py36_0    conda-forge
fontconfig                2.12.1                        4    conda-forge
freetype                  2.7                           1    conda-forge
freexl                    1.0.2                         2    conda-forge
geos                      3.5.1                         1    conda-forge
gettext                   0.19.8.1                      0    conda-forge
giflib                    5.1.4                         0    conda-forge
glib                      2.51.4                        0    conda-forge
h5netcdf                  0.3.1                     <pip>
h5py                      2.7.0               np113py36_1    conda-forge
hdf4                      4.2.12                        0    conda-forge
hdf5                      1.8.18                        0    conda-forge
heapdict                  1.0.0                    py36_1  
html5lib                  0.999                    py36_0  
hypothesis                3.23.0                   py36_0    conda-forge
icu                       58.1                          1    conda-forge
imagesize                 0.7.1                    py36_0    conda-forge
ipykernel                 4.5.2                    py36_0    conda-forge
ipython                   5.2.2                    py36_0    conda-forge
ipython_genutils          0.1.0                    py36_0    conda-forge
ipywidgets                5.2.2                    py36_2    conda-forge
jinja2                    2.8                      py36_1    conda-forge
jmespath                  0.9.0                    py36_0  
jpeg                      9b                            0    conda-forge
json-c                    0.12.1                        0    conda-forge
jsonschema                2.5.1                    py36_0    conda-forge
jupyter                   1.0.0                    py36_0    conda-forge
jupyter_client            4.4.0                    py36_0    conda-forge
jupyter_console           5.1.0                    py36_0    conda-forge
jupyter_core              4.2.1                    py36_0    conda-forge
kealib                    1.4.7                         2    conda-forge
libdap4                   3.18.3                        2    conda-forge
libffi                    3.2.1                         3    conda-forge
libgdal                   2.1.4                         2    conda-forge
libgfortran               3.0.0                         0    conda-forge
libiconv                  1.14                          4    conda-forge
libkml                    1.3.0                         1    conda-forge
libnetcdf                 4.4.1.1                       5    conda-forge
libpng                    1.6.28                        0    conda-forge
libpq                     9.6.3                         0    conda-forge
libsodium                 1.0.10                        0    conda-forge
libspatialite             4.3.0a                       15    conda-forge
libtiff                   4.0.6                         7    conda-forge
libxml2                   2.9.4                         4    conda-forge
libxslt                   1.1.29                        5    conda-forge
livereload                2.5.1                     <pip>
locket                    0.2.0                    py36_1  
lxml                      3.8.0                    py36_0    conda-forge
markupsafe                0.23                     py36_1    conda-forge
matplotlib                2.0.2                    py36_2    conda-forge
mccabe                    0.6.1                    py36_0    conda-forge
mistune                   0.7.3                    py36_1    conda-forge
mock                      2.0.0                    py36_0    conda-forge
msgpack-python            0.4.8                    py36_0    conda-forge
nbconvert                 5.1.1                    py36_0  
nbformat                  4.2.0                    py36_0  
ncurses                   5.9                          10    conda-forge
netcdf4                   1.2.9                    py36_1    conda-forge
notebook                  4.4.1                    py36_0    conda-forge
numba                     0.31.0                    <pip>
numpy                     1.13.1          py36_blas_openblas_200  [blas_openblas]  conda-forge
numpydoc                  0.7.0                    py36_0    conda-forge
olefile                   0.44                     py36_0    conda-forge
openblas                  0.2.19                        1    conda-forge
openjpeg                  2.1.2                         2    conda-forge
openssl                   1.0.2h                        3    conda-forge
owslib                    0.14.0                   py36_1    conda-forge
pandas                    0.20.3                   py36_1    conda-forge
pandocfilters             1.4.1                    py36_0  
partd                     0.3.8                    py36_0    conda-forge
path.py                   10.1                     py36_0  
pathtools                 0.1.2                     <pip>
patsy                     0.4.1                    py36_0    conda-forge
pbr                       3.1.1                    py36_0    conda-forge
pcre                      8.39                          0    conda-forge
pexpect                   4.2.1                    py36_0    conda-forge
pickleshare               0.7.4                    py36_0  
pillow                    4.2.1                    py36_0    conda-forge
pip                       9.0.1                    py36_0    conda-forge
pixman                    0.34.0                        0    conda-forge
poppler                   0.52.0                        2    conda-forge
poppler-data              0.4.7                         0    conda-forge
port-for                  0.3.1                     <pip>
proj4                     4.9.3                         4    conda-forge
prompt_toolkit            1.0.13                   py36_0    conda-forge
psutil                    5.1.3                    py36_0    conda-forge
ptyprocess                0.5.1                    py36_0  
py                        1.4.34                   py36_0    conda-forge
pycodestyle               2.3.1                    py36_0    conda-forge
pyepsg                    0.3.2                    py36_0    conda-forge
pyflakes                  1.5.0                    py36_0    conda-forge
pygments                  2.2.0                    py36_0    conda-forge
pympler                   0.5                      py36_0    conda-forge
pyparsing                 2.1.10                   py36_0    conda-forge
pyproj                    1.9.5.1                  py36_0    conda-forge
pyqt                      5.6.0                    py36_2  
pyshp                     1.2.12                     py_0    conda-forge
pytest                    3.2.1                    py36_0    conda-forge
pytest-cov                2.4.0                     <pip>
pytest-mock               1.6.2                    py36_0    conda-forge
python                    3.6.0                         2    conda-forge
python-dateutil           2.6.0                    py36_0    conda-forge
pytz                      2016.10                  py36_0    conda-forge
pyyaml                    3.12                     py36_0    conda-forge
pyzmq                     16.0.2                   py36_1    conda-forge
qt                        5.6.2                h9e3eb04_4    conda-forge
qtconsole                 4.2.1                    py36_1  
rasterio                  0.36.0                   py36_0    conda-forge
readline                  6.2                           0    conda-forge
requests                  2.13.0                   py36_0    conda-forge
s3fs                      0.0.9                    py36_0    conda-forge
s3transfer                0.1.10                   py36_0  
scipy                     0.19.1          py36_blas_openblas_202  [blas_openblas]  conda-forge
seaborn                   0.7.1                    py36_0    conda-forge
setuptools                33.1.0                   py36_0    conda-forge
shapely                   1.6.1                    py36_0    conda-forge
simplegeneric             0.8.1                    py36_1  
sip                       4.18                     py36_1    conda-forge
six                       1.10.0                   py36_1    conda-forge
snowballstemmer           1.2.1                    py36_0    conda-forge
snuggs                    1.4.1                    py36_0    conda-forge
sortedcontainers          1.5.3                    py36_0    conda-forge
sphinx                    1.6.3                    py36_0    conda-forge
sphinx-autobuild          0.7.1                     <pip>
sphinx-gallery            0.1.11                   py36_0    conda-forge
sphinx_rtd_theme          0.2.4                    py36_0    conda-forge
sphinxcontrib-websupport  1.0.1                    py36_0    conda-forge
sqlite                    3.13.0                        1    conda-forge
statsmodels               0.8.0               np113py36_0    conda-forge
tblib                     1.3.0                    py36_0  
terminado                 0.6                      py36_0    conda-forge
testpath                  0.3                      py36_0    conda-forge
tk                        8.5.19                        1    conda-forge
toolz                     0.8.2                    py36_0  
tornado                   4.4.2                    py36_0    conda-forge
traitlets                 4.3.1                    py36_0    conda-forge
typing                    3.6.2                    py36_0    conda-forge
watchdog                  0.8.3                     <pip>
wcwidth                   0.1.7                    py36_0  
wheel                     0.29.0                   py36_0    conda-forge
widgetsnbextension        1.2.6                    py36_0  
xerces-c                  3.1.4                         3    conda-forge
xz                        5.2.2                         0    conda-forge
yaml                      0.1.6                         0    conda-forge
zeromq                    4.2.1                         1    conda-forge
zict                      0.1.2                    py36_0    conda-forge
zlib                      1.2.8                         3    conda-forge
zope                      1.0                      py36_0  
zope.interface            4.4.2                    py36_0 

• pytest and operating system versions

$ conda info
Current conda install:

               platform : osx-64
          conda version : 4.3.22
       conda is private : False
      conda-env version : 4.3.22
    conda-build version : 2.1.15
         python version : 3.6.0.final.0
       requests version : 2.12.4

• Minimal example if possible

import pytest
import unittest


class MixInBaseA(unittest.TestCase):

    def test_a(self):
        assert 1 == 1


class MixInBaseB(unittest.TestCase):

    def test_b(self):
        assert 1 == 1


@pytest.mark.skipif(True, reason='skipping all TestC tests')
class TestC(MixInBaseA, MixInBaseB):

    def test_c(self):
        assert 1 == 1


class TestD(MixInBaseA, MixInBaseB):

    def test_d(self):
        assert 1 == 1


if __name__ == '__main__':
    pytest.main(['test_pytest.py', '-v'])

returns

============================================ test session starts ============================================
platform darwin -- Python 3.6.0, pytest-3.2.1, py-1.4.34, pluggy-0.4.0 -- /Users/jhamman/anaconda/envs/xarray36/bin/python
cachedir: .cache
rootdir: /Users/jhamman/Dropbox/src/xarray, inifile: setup.cfg
plugins: mock-1.6.2, cov-2.4.0, hypothesis-3.23.0
collected 8 items                                                                                            

test_pytest.py::MixInBaseA::test_a PASSED
test_pytest.py::MixInBaseB::test_b PASSED
test_pytest.py::TestC::test_a SKIPPED
test_pytest.py::TestC::test_b SKIPPED
test_pytest.py::TestC::test_c SKIPPED
test_pytest.py::TestD::test_a SKIPPED
test_pytest.py::TestD::test_b SKIPPED
test_pytest.py::TestD::test_d PASSED

==================================== 3 passed, 5 skipped in 0.01 seconds ====================================

We expect only TestC tests to be skipped but in this case, TestD::test_a and TestD::test_b are also skipped.

xref: pydata/xarray#1531, pydata/xarray#1557

[nicoddemus]

Thanks for writing @jhamman.

This is a known issue, and the team (specially @RonnyPfannschmidt) is working hard towards fixing warts like this, but this is a quite long endeavor so this will take a while.

[nicoddemus]

xref: #568, #535

[jhamman]

@nicoddemus - Thank you for the quick response. Since it looks like a fix for this may be a ways off, would you consider updating pytests docs to discourage using skipif on classes that use inheritance?

@RonnyPfannschmidt - is the mark refactor mentioned in the xrefs above still slated for 4.0? Trying to gauge how long we're going to have this behavior.

[RonnyPfannschmidt]

@jhamman pretty much indeterminate - this one is a bigger one - my attemtps earlier failed and to enable a effective refactoring we are in a bad position

currnetly makrs are pretty muvh broken wrt inheritance

[RonnyPfannschmidt]

we can also close this one as duplicate of #535 and #568

[nicoddemus]

Since it looks like a fix for this may be a ways off, would you consider updating pytests docs to discourage using skipif on classes that use inheritance?

Definitely. Would you like to open a PR? We would appreciate it.

Closing this as duplicate of #535 and #568 as @RonnyPfannschmidt suggested.