Virtualenv pinned to <20

[freundTech]

In #8387 virtualenv was pinned to <20 to fix failing tests.

virtualenv <20 will be broken by python 3.10. Therefor I tried to unpin it. I haven't quite managed to make it work yet, but found the cause of the problem and wanted to document my findings.

testpep561 creates a virtualenv, installs some packages into it and then runs mypy, while telling it to search for stub packages in that virtualenv.

At first I couldn't reproduce the error. As it turns out it is related to the patches debian (and it's derivates) apply to python.

virtualenv<20 ships it's own site.py file to build sys.path. This site.py file doesn't contain the getsitepackages and getusersitepackages methods, causing mypy's sitepkgs.py to fall back to using distutils to get the python library path.
This works.

virtualenv>=20 doesn't ship it's own site.py file, so the python site.py file is used. Debian patches this file to return paths ending in dist-packages instead of site-packages. Pip in a virtualenv however still installs to site-packages, causing mypy to not find the installed packages.

This is only a problem with python2. Debian doesn't seem to patch python3 site.py in this way.

[freundTech]

I got two patches into virtualenv to fix this and using virtualenv>=20.4.6 should no longer break on debian.

Tests are still broken on windows. If I get that fixed virtualenv will no longer have to be pinned to <20

[hauntsaninja]

Thanks for working on this!

[emmatyping]

@freundTech if you open a PR I can look at the Windows errors if you want.

[freundTech]

I'd prefer to not open a PR yet, as I regularly force-push to my branch to test things on the CI. You can however look at the branch directly. CI is enabled on my fork.
https://github.com/freundTech/mypy/tree/unpin-virtualenv

The tests on windows output "C:\Users\RUNNER~1\AppData\Local\Temp<random characters> is in the PYTHONPATH. Please change directory so it is not." to stderr and quit running after that.
This message comes from mypy/mypy/modulefinder.py, line 722. The directory is probably the virtualenv created by the test.

That's as far as I have currently investigated.

[JukkaL]

Are there any updates on this? Perhaps somebody could help, as this seems to be one of the biggest remaining issues blocking 3.10 support, I think.

[emmatyping]

This is only a problem with python2. Debian doesn't seem to patch python3 site.py in this way.

Personally, I'd prefer we just skip these tests on Python 2 since it seems that is the "issue". Our Python 2 support is pretty much best efforts anyway, and I don't think we should bend over backwards for a test that is broken because of Python 2 weirdness.

[JukkaL]

Personally, I'd prefer we just skip these tests on Python 2 since it seems that is the "issue".

Yeah, this sounds reasonable. This only seem to affect a specific Python 2 use case only. Installing packages in the same virtualenv that runs mypy still works, and I'd expect to be the most common use case by far. If users complain about this, we can consider fixing this.

Would anybody want to create PR that switches to a more recent virtualenv and skips the relevant tests?

[emmatyping]

Oh hm, looking closer it seems the issue on Windows may not be Python 2 related and probably deserves actually being looked into. I'll try to get to this sometime today.

[hauntsaninja]

Well-timed, looks like something has started causing these tests to fail on master (e.g., see #10845). Looked at a couple different packages and didn't see any incriminating releases in the relevant timeframe.

[pranavrajpal]

I think I know why those tests started failing suddenly. virtualenv released version 16.7.11 today, which bumped the versions of pip, setuptools, and wheel (see pypa/virtualenv@ca04bf7).

That appears to bump pip from 19.3.1 to 21.1.3, which breaks on python 2 by requiring the typing module (see pypa/pip@0945809), which is where the 'No module named typing' error is coming from.

That still leaves some unanswered questions though:

1. That virtualenv commit also included a copy of pip 20.3.4, presumably so that it could keep python 2 support. Why isn't that being used here?
2. The Travis CI setup appears to install typing on python2, so why isn't python finding that when pip imports typing?