python3 cannot be used to create venvs on recent Windows images

[ptheywood]

Description

The python3 executable errors when creating virtual environments, unlike the python executable on windows images.

python3 -m venv env leads to the following error Error: [WinError 2] The system cannot find the file specified.

python -m venv env successfully creates the venv.

This is only an issue since the 20210202.1 version of the windows-2019 environment which introduced the python3 symlink (#2461, actions/python-versions#78).

Here is an action run which illustrates this problem.
https://github.com/ptheywood/cmake-ctest-pytest/runs/1882133904?check_suite_focus=true

We only encountered this issue as we have a cmake process that creates a virtual environment to install a python package into for testing. CMakes module to find python prefers the executable containing the version name.

Area for Triage:

Python

Question, Bug, or Feature?:
Bug

Virtual environments affected

• Ubuntu 16.04
• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11.0
• Windows Server 2016 R2
• Windows Server 2019

Image version
20210202.1

Expected behavior
Virtual environment successfully created.

Actual behavior
Error: [WinError 2] The system cannot find the file specified

Repro steps

1. python -m venv py-env - works
2. python3 -m venv py3-env - errors

Example: https://github.com/ptheywood/cmake-ctest-pytest/runs/1882133904?check_suite_focus=true

[vsafonkin]

Hi @ptheywood , looks like venv is not working with symlinks by default, you can try to use --symlinks flag:

python3 -m venv py3-env --symlinks

https://github.com/vsafonkin/cmake-ctest-pytest/runs/1885272400?check_suite_focus=true

[dsame]

@vsafonkin I confirm python3 -m venv py3-env --symlinks works
@ptheywood does the suggested solution work for you?

[Robadob]

Have you tried replacing the symbolic link wtih a hard link inside the image? If this works it would be far more durable than having user's track down that they've hit an edge case that requires adding --symbolic.

The documentation for venv --symbolic doesn't even make it clear why it should fix the problem, it merely makes venv create symlinks instead of copies. The documentation even has an explicit note recommending against using --symlinks on windows.

If having python3 as a symbolic link breaks python3, surely creating a whole venv as symbolic links could break other components of the created venv.

I can't reproduce this on a local windows machine both --symbolic and --copies work when I setup python3 as a symbolic link as shown in actions/python-versions#78. So I'm unable to test my theory that a hard link will work as expected.

Edit: This would suggest hard links are broken too, https://github.com/Robadob/cmake-ctest-pytest/runs/1886181603?check_suite_focus=true

I suppose the remaining option would be to simply copy the whole executable. That way python3 should be identical to python .

Edit: Even a copy fails the same as a symlink, this is very unusual. https://github.com/Robadob/cmake-ctest-pytest/runs/1886195635

Thanks

[vsafonkin]

@Robadob , thank you for your help, looks like venv can't copies symlink to the created local environment and if we use --symlinks it just creates symlinks to binary files from original:

Run python3 -m venv py3-env --symlinks

    Directory: D:\a\cmake-ctest-pytest\cmake-ctest-pytest\py3-env\Scripts

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---           2/12/2021  9:34 AM           2316 activate
-a---           2/12/2021  9:34 AM            990 activate.bat
-a---           2/12/2021  9:34 AM           1521 Activate.ps1
-a---           2/12/2021  9:34 AM            368 deactivate.bat
-a---           2/12/2021  9:34 AM         106390 easy_install.exe
-a---           2/12/2021  9:34 AM         106390 easy_install-3.7.exe
-a---           2/12/2021  9:34 AM         106381 pip.exe
-a---           2/12/2021  9:34 AM         106381 pip3.7.exe
-a---           2/12/2021  9:34 AM         106381 pip3.exe
la---           2/12/2021  9:34 AM              0 python.exe -> C:\hostedtoolcache\windows\Python\3.7.9\x64\python.exe
la---           2/12/2021  9:34 AM              0 python-3.7.9-amd64.exe ->
                                                  C:\hostedtoolcache\windows\Python\3.7.9\x64\python-3.7.9-amd64.exe
la---           2/12/2021  9:34 AM              0 python3.dll ->
                                                  C:\hostedtoolcache\windows\Python\3.7.9\x64\python3.dll
la---           2/12/2021  9:34 AM              0 python3.exe ->
                                                  C:\hostedtoolcache\windows\Python\3.7.9\x64\python3.exe
la---           2/12/2021  9:34 AM              0 python37.dll ->
                                                  C:\hostedtoolcache\windows\Python\3.7.9\x64\python37.dll
la---           2/12/2021  9:34 AM              0 pythonw.exe ->
                                                  C:\hostedtoolcache\windows\Python\3.7.9\x64\pythonw.exe
la---           2/12/2021  9:34 AM              0 vcruntime140.dll ->
                                                  C:\hostedtoolcache\windows\Python\3.7.9\x64\vcruntime140.dll

UPD: but I can't understand why it doesn't work with copy of executable file.

[Robadob]

https://stackoverflow.com/questions/61669873/python-venv-env-fails-winerror-2-the-system-cannot-find-the-file-specified

Stack overflow suggests it's a common problem when you rename python.exe (when python 2 is also on the path?).

Edit: Disabling windows defender/Windows defender controlled file access doesn't appear to help. And python2 isn't on the path, so that seems unrelated.

[ptheywood]

@vsafonkin I confirm python3 -m venv py3-env --symlinks works
@ptheywood does the suggested solution work for you?

Yes using --symlinks does work, although as pointed out by @Robadob the venv docs reccomend against this on windows.

Note While symlinks are supported on Windows, they are not recommended. Of particular note is that double-clicking python.exe in File Explorer will resolve the symlink eagerly and ignore the virtual environment.
venv docs

We've got a workaround for CMake finding the symlinked executable, by telling it to use the relevant python.exe via -DPython3_ROOT_DIR and -DPython3_EXECUTABLE=.

[dsame]

@ptheywood does the workarounds work for your builds?

[Robadob]

@ptheywood does the workarounds work for your builds?

@dsame Refer to his previous comment if you haven't already noticed it.

This is the commit he made to address it: FLAMEGPU/FLAMEGPU2@0b4f2f1

[vsafonkin]

Hi @ptheywood, unfortunately, we didn't find any ways to resolve this issue. You can use either python -m venv to create local environment or python3 -m venv --symlinks to create environment through python3 symlink. Probably, it will be resolved from venv side someday. I'm going to close the issue because we can't do anything from our side.

[wildmichael]

@vsafonkin, it would be really helpful if this made it into the documentation of both, the Windows image and the actions/setup-python action.

[astarche]

It does look like this is fixed in cpython venv code, but not in 3.10. python/cpython#25216

[EDSPBOT]

@Robadob @vsafonkin Tried to run: virtualenv ..venv -p python, but got Fatal error in launcher: Unable to create process using '"C:\Users\Name\AppData\Local\Programs\Python\Python311\python.exe" "C:\Users\name\AppData\Local\Programs\Python\Python311\Scripts\virtualenv.exe" ..venv -p python': The system cannot find the file specified.
Please help me fix this