--doctest-modules causes all tests to be executed even when a single test is picked

[hameerabbasi]

Issue

When --doctest-modules is in setup.cfg, it causes all tests to be run even when a single one is picked. In addition, the selected test runs twice.

Environment

• Anaconda 5.0.1 with conda update --all already run.
• Pytest 3.3.0
• macOS 10.13.2

Minimal Example

Directory Structure:

• pytest_test
  • pytest_test
    • tests
      • test_pytest.py
  • setup.cfg

setup.cfg:

[tool:pytest]
addopts = --doctest-modules pytest_test

test_pytest.py

def test_one():
    assert 1 == 1


def test_two():
    assert 1 == 2

Command line:

py.test pytest_test/tests/test_pytest.py::test_two

Expected

Just test_two should be run.

Actual

test_one is run once and test_two twice.

pip-list

$ pip list
alabaster (0.7.10)
anaconda-client (1.6.6)
anaconda-navigator (1.6.10)
anaconda-project (0.8.2)
appnope (0.1.0)
appscript (1.0.1)
argh (0.26.2)
asn1crypto (0.23.0)
astroid (1.5.3)
astropy (2.0.3)
attrs (17.3.0)
Babel (2.5.1)
backports.shutil-get-terminal-size (1.0.0)
beautifulsoup4 (4.6.0)
bitarray (0.8.1)
bkcharts (0.2)
blaze (0.11.3)
bleach (2.1.1)
bokeh (0.12.13)
boto (2.48.0)
Bottleneck (1.2.1)
certifi (2017.11.5)
cffi (1.11.2)
chardet (3.0.4)
click (6.7)
cloudpickle (0.5.2)
clyent (1.2.2)
colorama (0.3.9)
conda (4.4.6)
contextlib2 (0.5.5)
coverage (4.4.2)
cryptography (2.1.4)
cycler (0.10.0)
Cython (0.27.3)
cytoolz (0.8.2)
dask (0.16.0)
datashape (0.5.4)
decorator (4.1.2)
distributed (1.20.2)
Django (2.0)
docutils (0.14)
entrypoints (0.2.3)
et-xmlfile (1.0.1)
fastcache (1.0.2)
filelock (2.0.13)
flake8 (3.5.0)
Flask (0.12.2)
Flask-Cors (3.0.3)
gevent (1.2.2)
glob2 (0.6)
gmpy2 (2.0.8)
greenlet (0.4.12)
h5py (2.7.1)
heapdict (1.0.0)
html5lib (1.0.1)
hypothesis (3.38.5)
idna (2.6)
imageio (2.2.0)
imagesize (0.7.1)
ipykernel (4.7.0)
ipython (6.2.1)
ipython-genutils (0.2.0)
ipywidgets (7.0.5)
isort (4.2.15)
itsdangerous (0.24)
jdcal (1.3)
jedi (0.11.0)
Jinja2 (2.10)
joblib (0.11)
jsonschema (2.6.0)
jupyter-client (5.1.0)
jupyter-console (5.2.0)
jupyter-core (4.4.0)
jupyterlab (0.30.6)
jupyterlab-launcher (0.6.0)
lazy-object-proxy (1.3.1)
livereload (2.5.1)
llvmlite (0.21.0)
locket (0.2.0)
lxml (4.1.1)
MarkupSafe (1.0)
matplotlib (2.1.1)
mccabe (0.6.1)
mistune (0.8.1)
mpmath (1.0.0)
msgpack-python (0.4.8)
multipledispatch (0.4.9)
navigator-updater (0.1.0)
nbconvert (5.3.1)
nbformat (4.4.0)
networkx (2.0)
nltk (3.2.5)
nose (1.3.7)
notebook (5.2.2)
numba (0.36.2)
numexpr (2.6.4)
numpy (1.13.3)
numpydoc (0.7.0)
odo (0.5.1)
olefile (0.44)
openpyxl (2.4.9)
packaging (16.8)
pandas (0.21.1)
pandocfilters (1.4.2)
parso (0.1.1)
partd (0.3.8)
path.py (10.5)
pathlib2 (2.3.0)
pathtools (0.1.2)
patsy (0.4.1)
pep8 (1.7.1)
pexpect (4.3.0)
pickleshare (0.7.4)
Pillow (4.3.0)
pip (9.0.1)
pkginfo (1.4.1)
pluggy (0.6.0)
ply (3.10)
pockets (0.5.1)
port-for (0.3.1)
prompt-toolkit (1.0.15)
psutil (5.4.1)
ptyprocess (0.5.2)
py (1.5.2)
pycodestyle (2.3.1)
pycosat (0.6.3)
pycparser (2.18)
pycrypto (2.6.1)
pycurl (7.43.0)
pyflakes (1.5.0)
Pygments (2.2.0)
pylint (1.7.4)
Pympler (0.5)
pyodbc (4.0.21)
pyOpenSSL (17.5.0)
pyparsing (2.2.0)
PySocks (1.6.7)
pytest (3.3.0)
pytest-cov (2.5.1)
pytest-flake8 (0.9.1)
pytest-runner (3.0)
python-dateutil (2.6.1)
pytz (2017.3)
PyWavelets (0.5.2)
PyYAML (3.12)
pyzmq (16.0.3)
QtAwesome (0.4.4)
qtconsole (4.3.1)
QtPy (1.3.1)
requests (2.18.4)
rope (0.10.7)
ruamel-yaml (0.11.14)
scikit-image (0.13.1)
scikit-learn (0.19.1)
scipy (1.0.0)
seaborn (0.8.1)
setuptools (36.6.0)
simplegeneric (0.8.1)
singledispatch (3.4.0.3)
six (1.11.0)
snakeviz (0.4.2)
snowballstemmer (1.2.1)
sortedcollections (0.5.3)
sortedcontainers (1.5.7)
sparse (0.1.1, /Users/hameerabbasi/PycharmProjects/sparse)
Sphinx (1.6.5)
sphinx-autobuild (0.7.1)
sphinx-rtd-theme (0.2.4)
sphinxcontrib-napoleon (0.6.1)
sphinxcontrib-websupport (1.0.1)
sphobjinv (1.0)
spyder (3.2.4)
SQLAlchemy (1.1.15)
statsmodels (0.8.0)
sympy (1.1.1)
tables (3.4.2)
tblib (1.3.2)
terminado (0.6)
testpath (0.3.1)
toolz (0.8.2)
tornado (4.5.2)
traitlets (4.3.2)
typing (3.6.2)
unicodecsv (0.14.1)
urllib3 (1.22)
watchdog (0.8.3)
wcwidth (0.1.7)
webencodings (0.5.1)
Werkzeug (0.12.2)
wheel (0.30.0)
widgetsnbextension (3.0.8)
wrapt (1.10.11)
xlrd (1.1.0)
XlsxWriter (1.0.2)
xlwings (0.11.4)
xlwt (1.3.0)
zict (0.1.3)
zope.interface (4.4.3)

[pytestbot]

GitMate.io thinks the contributor most likely able to help you is @nicoddemus.

[hameerabbasi]

I just realized that even if I add the option in the command line and skip setup.cfg, the issue still happens.

[nicoddemus]

Thanks for the report @hameerabbasi. Might have some relation to #1756.

[nils-werner]

Selecting tests using -k EXPRESSION is unaffected:

py.test -k test_two

[hameerabbasi]

If someone can give me a brief overview of what needs to be done to fix this, I'll try my hand at a PR.

Edit: It can't be what you suggested @nicoddemus, (because that function isn't even entered into without the command line argument so technically it shouldn't even be called in the buggy case)

[nicoddemus]

@hameerabbasi thanks for the offer! 😁

Here's the code which collects a DoctestModule given a file:

pytest/_pytest/doctest.py

Lines 52 to 58 in 44fa5a7

	def pytest_collect_file(path, parent):
	config = parent.config
	if path.ext == ".py":
	if config.option.doctestmodules and not _is_setup_py(config, path, parent):
	return DoctestModule(path, parent)
	elif _is_doctest(config, path, parent):
	return DoctestTextfile(path, parent)

And as can be seen here, it will always run all doctests found inside a module, regardless of any options:

pytest/_pytest/doctest.py

Lines 244 to 246 in 44fa5a7

	for test in finder.find(module, module.__name__):
	if test.examples: # skip empty doctests
	yield DoctestItem(test.name, self, runner, test)

[Zac-HD]

This isn't actually a bug - you're actually asking for the tests to be run twice! The --doctest-modules argument runs doctests in all modules, so

pytest pytest_test/tests/test_pytest.py::test_two --doctest-modules pytest_test

means "Run test_two, run all doctests, and run all tests in pytest_test.py".

TLDR; your addopts config was running all doctests and all tests in addition to whatever you specified.