python.testing.cwd ignored when discovering or running tests

[botsman]

Environment data

• VS Code version: 1.40.1
• Extension version (available under the Extensions sidebar): 2019.11.49689
• OS and version: MacOS Catalina 10.15.1
• Python version (& distribution if applicable, e.g. Anaconda): 3.7.4
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv
• Relevant/affected Python packages and their versions: pytest==5.3.0
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): Jedi

Expected behaviour

Tests are displayed in the test explorer according to python.testing.cwd setting and I am able to run single test or tests from single file

Actual behaviour

python.testing.cwd is ignored for displaying tests and I am unable to run single test or tests from single file because I am getting an error file not found.

Steps to reproduce:

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

Sample project: https://github.com/pbotsman/sample_project

1. Open project
2. Open tests/py/some_test.py to start exploring tests
3. Try to execute single test test_something or tests in some_test.py file.
4. Get the error ERROR: file not found: ./tests/py/some_test.py::test_something
   (Sometimes in order to reproduce the problem I had to relaunch my VSCode)

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

> ~/.virtualenvs/sample_project/bin/python -m pytest --rootdir ~/projects/sample_project --junitxml=/var/folders/0_/6w94lk1171vc_fpwbjqx02nw0000gn/T/tmp-4746yxdv7Lw3muwB.xml -v -s --log-cli-level INFO ./tests/py/some_test.py::test_something
cwd: ~/projects/sample_project/tests/py

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)

console.ts:137 [Extension Host] Info Python Extension: 2019-11-20 14:06:31: getActivatedEnvironmentVariables, Class name = b, completed in 1ms, Arg 1: <Uri:/Users/pavel/projects/sample_project>, Arg 2: undefined, Arg 3: undefined
console.ts:137 [Extension Host] Info Python Extension: 2019-11-20 14:06:31: > ~/.virtualenvs/sample_project/bin/python -m pytest --rootdir ~/projects/sample_project --junitxml=/var/folders/0_/6w94lk1171vc_fpwbjqx02nw0000gn/T/tmp-5598WI4lpp4kFEsC.xml -v -s --log-cli-level INFO ./tests/py/some_test.py::test_something
console.ts:137 [Extension Host] Info Python Extension: 2019-11-20 14:06:31: cwd: ~/projects/sample_project/tests/py

Additional info

Looks like something has changed in the last release:

1. Previously tests would be displayed in the test explorer just as some_test.py would be root test. I mean that tests/py would be omitted in the test explorer since I have "python.testing.cwd": "tests/py" in my settings.
   Currently directories tests/py are displayed in the test explorer even though I have this setting.
2. I can't run single test because I am always getting an error file not found. It looks like something is wrong with the file paths.
   When the command
   ~/.virtualenvs/sample_project/bin/python -m pytest --rootdir ~/projects/sample_project --junitxml=/var/folders/0_/6w94lk1171vc_fpwbjqx02nw0000gn/T/tmp-5598WI4lpp4kFEsC.xml -v -s --log-cli-level INFO ./tests/py/some_test.py::test_something
   is called my current working directory is set to python.testing.cwd and that is why I am getting the error.
   So in the command mentioned above valid file path should be ./some_test.py::test_something, not ./tests/py/some_test.py::test_something.

[brettcannon]

Thanks for the bug report! We just wanted to quickly acknowledge we received it and we will triage this as soon as we can.

[kimadeline]

Hi @pbotsman 👋 Thank you for reaching out and thoroughly filling out the issue template 👍

I can confirm that I could reproduce your issue in the latest release of the extension (2019.11.49689), and that it doesn't happen in the previous release (2019.10.44104).

Seems like we don't take python.testing.cwd into account anymore when discovering tests, or running single tests using pytest.

A possible workaround while waiting for a fix would be to move the content of python.testing.cwd in the list of pytest arguments, let me know if the following works for you:

{
    "python.pythonPath": "/path/to/python",
    "python.testing.pytestArgs": [        
        "./tests/py",
        "-v",
        "-s",
        "--log-cli-level", "INFO",
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true
}

[botsman]

Hi @kimadeline,

Thanks for the workaround. This is exactly what I did 👍

[bhrutledge]

FWIW, I just stumbled upon this, and would love to see a fix that changes the working directory to python.testing.cwd before running pytest.

[kimadeline]

@bhrutledge Please upvote the original comment to help up prioritize it.

Thanks!

[martynjames]

I am running into this problem too - I am working in a team where others are using different IDEs... within their environment, their tests load a file from the same directory as the code.

Even with adding the directory to the execution path to python.testing.unittestArgs the current working directory still is not set, and I need to change the paths to the data files in order for the tests to pass.

[rndev-io]

Any progress or workarounds for this issue? Setting option "python.testing.pytestArgs": ["--rootdir=path/to/root"] doesn't help.

[Ark-kun]

Seems like we don't take python.testing.cwd into account anymore when discovering tests, or running single tests using pytest.

This looks like a bug. It also seems to affect the unittest tests. Setting python.testing.cwd has zero effect on them.

# python.testing.cwd == <workspace_root>/sdk/python

# Here is what I see during test discovery:

> .../python .../.vscode/extensions/ms-python.python-2020.4.74986/pythonFiles/pyvsc-run-isolated.py -c "
import unittest
loader = unittest.TestLoader()
suites = loader.discover(".", pattern="*test*.py")
print("start") #Don't remove this line
for suite in suites._tests:
    for cls in suite._tests:
        try:
            for m in cls._tests:
                print(m.id())
        except:
            pass"
cwd: <workspace_root>

See how the bottom line prints a different value of cwd.

[flyte]

Any progress or workarounds for this issue? Setting option "python.testing.pytestArgs": ["--rootdir=path/to/root"] doesn't help.

That seems to be because the extension sets --rootdir to ${workspaceFolder}, overriding anything you've specifically set there.

[danstreeter]

Disclaimer: I have about zero days experience with Typescript, and the vscode-python extension... however I think the issue might have stemmed from this commit:
"Force the pytest root dir to always be the workspace root."
9fe2a76

[saschahofmann]

I had a similar problem and in my case the problem was this, basically pytest doesn't add the directory to the PYTHONPATH for me solution 3 seems the most appropriate, simply create a conftest.py file in your root directory you want to run the tests from and Voila.

Also related to #10005

[henrik-wist]

This looks like a bug. It also seems to affect the unittest tests. Setting python.testing.cwd has zero effect on them.

I can confirm that it affects unittest tests as well. Interestingly, when running the whole test-suite ("Run All Tests") cwd is correctly set to the folder containing the tests files (even without setting python.testing.cwd at all). When I run a single test, cwd is set to the workspace-root, regardless of the python.testing.cwd setting.

[luabud]

@flyte sounds good, thank you so much!

[luabud]

@ChildishGiant and just to make sure, you have the tests set up to use pytest and not unittest (i.e. "python.testing.pytestEnabled": true in your settings)?

if you could share your .vscode/settings.json and your project structure, that would help a lot! 😊

[ChildishGiant]

The only relevant settings are this:

"python.testing.pytestEnabled": true,
"python.testing.pytestArgs": ["--rootdir","${workspaceFolder}/unit_tests"]

In my project all my tests are under a unit_tests folder. It reads files relative to that folder but the command used in Test Log discover pytest -- --rootdir f:\code\cloverleaf\web -s unit_tests, not --rootdir f:\code\cloverleaf\web\unit_tests. I've also tried without ${workspaceFolder} but have had no luck.

[keke8273]

I have also noticed after setting rootdir in python.testing.pytestArgs in either

"python.testing.pytestArgs": ["--rootdir","${workspaceFolder}/unit_tests"]

or

"python.testing.pytestArgs": ["--rootdir=${workspaceFolder}/unit_tests"]

The unit test results (pass or fail) are no longer updated in the TEST explorer.

Environments

[flyte]

@luabud

As @danstreeter and I pointed out before, the issue is that this extension forces the --rootdir parameter to pytest to be set to the workspaceFolder.

The fix is to perhaps have this as the default, but allow users to override the argument in their settings.

The problem was introduced here 9fe2a76

[twsl]

@karthiknadig great to see that you took over.

[karthiknadig]

Merged a fix for this just now to main. Fix is essentially what #8678 (comment) said. If there is --rootdir in python.testing.pytestArgs, we won't override it. so for the sample repo, the settings change would be:

{
    "python.testing.pytestArgs": [
        "-v",
        "-s",
        "--log-cli-level",
        "INFO",
        "--rootdir",
        "tests/py"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true
}

That should work:

Try this out with this build https://github.com/microsoft/vscode-python/suites/3858667759/artifacts/95863817 if that doesn't work please provide repro steps.

[botsman]

Works for me

Had to restart VS Code in order tests to appear in the sidebar without being nested inside tests directory

[danstreeter]

(Apologies if this is a rediculously silly question...)
Can I install the aformentioned vsix extension on my day-to-day working VSCode instance without risk of completely breakign it without any way back?!

Can it be uninstalled once installed when this fix reaches launch?

[karthiknadig]

@danstreeter You can install this and uninstall to restore your VS Code to stable extension or any extension version.

Here is how you would do it:

1. Install the VSIX using Extension tab > ... > Install from vsix... option from the "..." menu.
2. You may have to Reload VS Code (command pallet Reload Window command). Test out to see if this works for you.
3. To revert back, Go to extensions tab, find the python extension and Uninstall it.
4. Reload VS Code (command pallet Reload Window command), go to extensions tab and install from market place. This should get you back to stable build of the extension.

Note, you can also click on the 'Gear' icon next to the extension in the extensions tab to install a specific version using "Install Another version..."

If you want to try this out is complete isolation, you could install VS Code Insiders (which can install side by side with your stable one) and install this vsix there and test it out. then uninstall the whole thing. This way it won't touch your day-to-day VS Code instance.

[danstreeter]

Thanks @karthiknadig , I've got it installed in my main 'day-to-day' instance however still not quite working for me...

Using the default provided config after 'configuring tests', I end up with the following in my settings file:

    "python.testing.pytestEnabled": true,
    "python.testing.pytestArgs": [
        "tests"
    ],

This matches my project layout, as all tests are stored in the ./tests directory with all actual code in ./src

Running my tests with my normal workflow still works fine by running make test which actually runs:

export PYTHONPATH=./src
pipenv run python -m pytest tests

When performing discovery with the above config, I get the following in the Python output:

~/path-to-code/.venv/bin/python3.8 ~/.vscode/extensions/ms-python.python-2021.10.1267871984-dev/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir  -s --cache-clear tests
Error 2021-09-24 19:51:20: Error discovering pytest tests:
 [r [Error]: ============================= test session starts ==============================
platform darwin -- Python 3.8.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /
plugins: asyncio-0.15.1, html-3.1.1, metadata-1.11.0
collected 0 items

I also get the following warning:

PytestCacheWarning: could not create cache path /.pytest_cache/v/cache/stepwise

Also, if I add any --rootdir to the config as follows:

    "python.testing.pytestEnabled": true,
    "python.testing.pytestArgs": [
        "--rootdir",
        "my-root-dir-fake-name",
        "tests"
    ],

The error change as 'expected' with the root directory and trys to target a path on root:

~/path-to-code/.venv/bin/python3.8 ~/.vscode/extensions/ms-python.python-2021.10.1267871984-dev/pythonFiles/testing_tools/run_adapter.py discover pytest -- -s --cache-clear --rootdir my-root-dir-fake-name my-root-dir-fake-name
Error 2021-09-24 19:53:03: Error discovering pytest tests:
 [r [Error]: ERROR: Directory '/my-root-dir-fake-name' not found. Check your '--rootdir' option.

Why is it trying to target root?

[danstreeter]

OOH OOH OOH - I FIXED IT!!!

I needed an absolute path in the rootdir... Using my pseudo, sanitised example above - this now works!

    "python.testing.pytestEnabled": true,
    "python.testing.pytestArgs": [
        "--rootdir",
        "/full/path/to/my/path-to-code",
        "tests"
    ],

--- Update ---

This is much nicer:

    "python.testing.pytestEnabled": true,
    "python.testing.pytestArgs": [
        "--rootdir",
        "${workspaceFolder}",
        "tests"
    ],

[karthiknadig]

@danstreeter can you try:

"python.testing.pytestEnabled": true,
    "python.testing.pytestArgs": [
        "--rootdir",
        "./tests"
    ],

[danstreeter]

Doing the above gives:

> ~/path-to-code/.venv/bin/python3.8 ~/.vscode/extensions/ms-python.python-2021.10.1267871984-dev/pythonFiles/testing_tools/run_adapter.py discover pytest -- -s --cache-clear --rootdir ./tests ./tests
Error 2021-09-24 20:00:44: Error discovering pytest tests:
 [r [Error]: ERROR: Directory '/tests' not found. Check your '--rootdir' option.

[karthiknadig]

@danstreeter Are you missing the '.' before the './tests'? Just checking because '/tests' will not work.

[danstreeter]

Definately there:

The config in the last part of my previous post up a bit (#8678 (comment)) works perfect though.

[karthiknadig]

I see. Thanks for the input. This should work for most case with this:

    "python.testing.pytestEnabled": true,
    "python.testing.pytestArgs": [
        "--rootdir",
        "${workspaceFolder}",
        "tests"
    ],

I will tweak it, it seems like for some reason, the cwd gets set to `` (empty) string instead of . which seems to be the problem here. But that is unrelated to this.