pytest test discovery tries to import conftest.py from other projects

[hsharrison]

Originally posted in #12538, I thought maybe it was the same problem, but I'm not sure.

My project structure is not the most common one, but not so unusual either, I think.

<project-root>/
|----src/
|    |----<package>/ 
|    |    |----__init__.py
|    |    |----...
|----tests/
|    |----__init__.py
|    |----conftest.py
|    |----test_xxxx.py
|    |----...
|----setup.cfg

setup.cfg contains the pytest settings such that simple running pytest from <project-root> works perfectly.

However, when I try to discover the tests in vscode, I get this error:

During handling of the above exception, another exception occurred:
/opt/conda/envs/<env-name>/lib/python3.7/site-packages/_pytest/config/__init__.py:446: in _importconftest
    mod = conftestpath.pyimport()
/opt/conda/envs/<env-name>/lib/python3.7/site-packages/py/_path/local.py:721: in pyimport
    raise self.ImportMismatchError(modname, modfile, self)
E   py._path.local.LocalPath.ImportMismatchError: ('tests.conftest', '/home/<user>/src/<another-project-dir>/tests/conftest.py', local('/home/<user>/src/<another-project-dir>/tests/conftest.py'))

It's trying to load conftest.py from two other projects!! I've confirmed that these projects are not installed in the conda env or accidentally cached anywhere (everywhere I could think to check). The project I'm working on has no reason to know that this project exists.

From the output I found this command: python /home/<user>/.local/share/code-server/extensions/ms-python.python-2020.5.86806/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir <project-path> -s

If I manually run this command in my project root, it works perfectly. If I run it from my home directory, I replicate the problem where it tries to load the wrong project. In fact, the two projects it's trying to load, are the only two that have a conftest.py anywhere within my home directory.

I tried setting python.testing.cwd to <project-path> but it didn't change anything. I can't think of anything else. In the end my best guess is that run_adapter.py should be run from the project path rather than my home directory.

Any advice is appreciated. I'm willing to believe its PEBCAK and not a bug, but either way I'd be grateful for a solution.

Originally posted by @hsharrison in #12538 (comment)

[kimadeline]

Hi @hsharrison, thank you for extracting your issue out!

A couple of things here:

• could you provide a sample repo I could use to try and replicate the issue you're having?
• could you also provide the output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)
• last but not least, could you try updating the extension to its latest version? I don't recall us changing anything in test discovery recently, but something else might have impacted it.

Thank you 🙂

[hsharrison]

Thanks for taking a look.

In the most recent version I have the problem python.setinterpreter not found. I think this issue is already reported (although I will try to find it and provide my own info; edit: #12465). I rolled back and could successfully select interpreter. That's why I'm using an older version. On the most recent version I can't even reach the point of attempting test discovery.

Anyways, I created this repo: https://github.com/hsharrison/test-repo

The repo doesn't include those other directories that it's finding, but cloning the above repo twice should replicate it.

Here's my output after following the installation instructions in the repo and then (1) setting the python interpreter to /opt/conda/envs/test-repo/bin/python then (2) running test discovery:

python /home/<user>/.local/share/code-server/extensions/ms-python.python-2020.10.332292344/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /home/<user>/src/test-repo -s --cache-clear
Test Discovery failed: 
Error: ============================= test session starts ==============================
platform linux -- Python 3.7.2, pytest-5.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/<user>/src/test-repo
plugins: cov-2.7.1, docstyle-2.0.0, flake8-1.0.6, mypy-0.4.0
collected 0 items / 1 errors

==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
/opt/conda/envs/test-repo/lib/python3.7/site-packages/_pytest/config/__init__.py:440: in _importconftest
    return self._conftestpath2mod[conftestpath]
E   KeyError: local('/home/<user>/src/<other-project-a>/tests/conftest.py')

During handling of the above exception, another exception occurred:
/opt/conda/envs/test-repo/lib/python3.7/site-packages/_pytest/config/__init__.py:446: in _importconftest
    mod = conftestpath.pyimport()
/opt/conda/envs/test-repo/lib/python3.7/site-packages/py/_path/local.py:724: in pyimport
    raise self.ImportMismatchError(modname, modfile, self)
E   py._path.local.LocalPath.ImportMismatchError: ('tests.conftest', '/home/<user>/src/<other-project-a>/tests/conftest.py', local('/home/<user>/src/<other-project-b>/tests/conftest.py'))

During handling of the above exception, another exception occurred:
/opt/conda/envs/test-repo/lib/python3.7/site-packages/py/_path/common.py:383: in visit
    for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
/opt/conda/envs/test-repo/lib/python3.7/site-packages/py/_path/common.py:435: in gen
    for p in self.gen(subdir):
/opt/conda/envs/test-repo/lib/python3.7/site-packages/py/_path/common.py:435: in gen
    for p in self.gen(subdir):
/opt/conda/envs/test-repo/lib/python3.7/site-packages/py/_path/common.py:424: in gen
    dirs = self.optsort([p for p in entries
/opt/conda/envs/test-repo/lib/python3.7/site-packages/py/_path/common.py:425: in <listcomp>
    if p.check(dir=1) and (rec is None or rec(p))])
/opt/conda/envs/test-repo/lib/python3.7/site-packages/_pytest/main.py:606: in _recurse
    ihook = self.gethookproxy(dirpath)
/opt/conda/envs/test-repo/lib/python3.7/site-packages/_pytest/main.py:424: in gethookproxy
    my_conftestmodules = pm._getconftestmodules(fspath)
/opt/conda/envs/test-repo/lib/python3.7/site-packages/_pytest/config/__init__.py:420: in _getconftestmodules
    mod = self._importconftest(conftestpath)
/opt/conda/envs/test-repo/lib/python3.7/site-packages/_pytest/config/__init__.py:454: in _importconftest
    raise ConftestImportFailure(conftestpath, sys.exc_info())
E   _pytest.config.ConftestImportFailure: (local('/home/<user>/src/<other-project-a>/tests/conftest.py'), (<class 'py._path.local.LocalPath.ImportMismatchError'>, ImportMismatchError('tests.conftest', '/home/<user>/src/<other-project-a>/tests/conftest.py', local('/home/<user>/src/<other-project-b>/tests/conftest.py')), <traceback object at 0x7fab8f51e488>))
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
========================= 1 error in 150.27s (0:02:30) =========================

Traceback (most recent call last):
  File "/home/<user>/.local/share/code-server/extensions/ms-python.python-2020.10.332292344/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/<user>/.local/share/code-server/extensions/ms-python.python-2020.10.332292344/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/home/<user>/.local/share/code-server/extensions/ms-python.python-2020.10.332292344/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

[kimadeline]

Thank you for providing a test repo! Sadly I haven't been able to replicate your issue so far 😞 I followed these steps:

• cloned your test repo in a folder named test-repo-one
• cd test-repo-one, followed your setup steps, and cd ..
• cloned your test repo in a folder named test-repo-two, which means that my root folder has test-repo-one and test-repo-two:
  
• cd test-repo-one; code ., selected the test-repo conda environment as my interpreter, and discovered tests in VS Code using pytest and the existing setup.cfg
• in my terminal, moved back to the parent directory, cd test-repo-two; code ., selected the existing test-repo conda environment as my interpreter, and discovered tests in VS Code using pytest and the existing setup.cfg

Is there something missing?

If I manually run this command in my project root, it works perfectly. If I run it from my home directory, I replicate the problem where it tries to load the wrong project.

If we take your project structure from above:

<project-root>/
|----src/
|    |----<package>/ 
|    |    |----__init__.py
|    |    |----...
|----tests/
|    |----__init__.py
|    |----conftest.py
|    |----test_xxxx.py
|    |----...
|----setup.cfg

Is that how you opened it opened in VS Code/code-server? Or did you open your home directory there (which I assume is the parent folder of <project-root>)?

I renamed the package in test-repo-two so that I could run pytest from the parent folder, and it turns out I had to run pip install -e .[test] again in test-repo-two after I renamed it or discovery would fail. After that discovery worked both in the terminal and in VS Code:

Have you tried reinstalling your package before performing test discovery again?

I also have a couple of housekeeping questions, just to be on safe side:

• Have you tried deleting the .pytest-cache folders?
• Is the python.testing.pytestArgs setting present in your .vscode/settings.json file (it shouldn't), and what is its value?

Thanks!

[hsharrison]

Thanks for helping me look into this...

Is there something missing?

Not that I can see.

Is that how you opened it opened in VS Code/code-server? Or did you open your home directory there (which I assume is the parent folder of )?

That's right, I opened it from the project directory, not my home directory. I never open vscode in the parent directory where I would see both project

I renamed the package in test-repo-two so that I could run pytest from the parent folder, and it turns out I had to run pip install -e .[test] again in test-repo-two after I renamed it or discovery would fail. After that discovery worked both in the terminal and in VS Code:

Maybe I don't follow this part, but it seems strange to me to run it from here. I always run it from the project root (also where I open vscode)

Have you tried reinstalling your package before performing test discovery again?
Have you tried deleting the .pytest-cache folders?

No luck doing these and trying again.

Is the python.testing.pytestArgs setting present in your .vscode/settings.json file (it shouldn't), and what is its value?

Nope.

---

Seems like we're a bit stuck. I still think the root problem is run_adapter.py getting called from the home directory rather than the project root, but I don't know enough how that's supposed to work.

[kimadeline]

it seems strange to me to run it from here.

Yeah, I just wanted to double-check, since I couldn't repro your issue when opening the workspace from the project root.

You mentioned a variant of it above, but can you confirm that running ~/opt/anaconda3/envs/test-repo/bin/python/home/<user>/.local/share/code-server/extensions/ms-python.python-2020.10.332292344/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /home/<user>/src/test-repo -s --cache-clear (replace <user> with your paths) manually in the terminal works?

It should return something like

[
   {
      "rootid":".",
      "root":"/Users/<user>/Documents/src/test-repo-one",
      "parents":[
         {
            "id":"./tests",
            "kind":"folder",
            "name":"tests",
            "parentid":".",
            "relpath":"./tests"
         },
         {
            "id":"./tests/test_packagename.py",
            "kind":"file",
            "name":"test_packagename.py",
            "parentid":"./tests",
            "relpath":"./tests/test_packagename.py"
         }
      ],
      "tests":[
         {
            "id":"./tests/test_packagename.py::test_add",
            "name":"test_add",
            "source":"./tests/test_packagename.py:4",
            "markers":[
               
            ],
            "parentid":"./tests/test_packagename.py"
         }
      ]
   }
]

It might also be an issue with the code-server integration. Does the issue also happen if you use an older version of the extension?

[kimadeline]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on then we will be happy to re-open this issue to pick up where we left off.

Thank you!