ModuleNotFoundError: No module named 'pip' with 3.x-dev on Travis CI

[hugovk]

If submitting a BUG please provide:

• Minimal reproducible example or detailed descriptions
• OS and pip list output

OS: Xenial
pip list: don't know, exception when running pip

1. Build on Travis CI with .travis.yml:

language: python
dist: xenial

env:
  - VIRTUALENV_SEEDER=pip

python:
  - 'nightly'
  - '3.9-dev'
  - '3.8-dev'
  - '3.7-dev'
  - '3.6-dev'
  - 'pypy'
  - 'pypy3'
  - '3.8'
  - '3.7'
  - '3.6'
  - '3.5'
  - '2.7'

install:
  - pip install -U tqdm

script:
  - echo "$DIST"
  - lsb_release -a
  - python --version
  - python ./test.py

Expected result

Passes

Actual result

On 3.x-dev and nightly, there's a ModuleNotFoundError: No module named 'pip'.

Setting environment variables from .travis.yml
$ export VIRTUALENV_SEEDER=pip
0.00s$ source ~/virtualenv/pythonnightly/bin/activate
$ python --version
Python 3.9.0a3+
$ pip --version
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.9-dev/bin/pip", line 6, in <module>
    from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip'

https://travis-ci.org/hugovk/test/builds/648978823

Also happens without VIRTUALENV_SEEDER=pip.

See also

The [Travis CI] build process does go through changes, of course, but for the most part they are pretty stable. If the *-dev version behaviors change, the changes are most likely due to the changes upstream.

https://travis-ci.community/t/python-development-versions-no-longer-include-pip/7180

[hugovk]

This looks like it coincides with the release of virtualenv 20.

Passing build, yesterday:

• https://travis-ci.org/hugovk/test/builds/648322611

Failing build, today:

• https://travis-ci.org/hugovk/test/builds/648976678

[gaborbernat]

@hugovk I don't see any virtualenv creation here, just activation? Who and where creates the virtual environments? They would need to use the pip seeder; or also cache the app-data folder, or we can copy the images.

[hugovk]

@hugovk I don't see any virtualenv creation here, just activation? Who and where creates the virtual environments? They would need to use the pip seeder; or also cache the app-data folder, or we can copy the images.

@BanzaiMan / @native-api Hello, please could you answer this? ^

[native-api]

I already identified the reason. @hugovk should have probably linked to that message for reference.

It’s a change in virtualenv‘s behavior in the latest version.
It now saves a couple of required files – including pip installation – outside the virtualenv’s subtree. Travis’ CPython compilation worker saves a premade virtualenv into a tarball to avoid recreating it each time – which now doesn’t include those files.

AFAICS, a workaround is to use --seeder=pip.

---

For the reference, with the now-default app-data seeder, is there a way to save a virtualenv into a tarball that guarantees its integrity and no conflicts when restored onto another machine?
It's being saved together with the corresponding pyenv-installed Python installation, and the other machine has the same arch and distro version.

[gaborbernat]

@native-api you would have a few options, in an hour or so once #1579 is merged I'll release it part of 20.0.2:

• this allows moving the app-data to something you also cache - app-data copies by default #1578
• the new default now copies rather than symlinks by default, so things should be good out of box Make app data override variable public #1559

[gaborbernat]

This now has been fixed on our side, so I'll close the issue. Once we do the bugfix release I'll come back and post a message here.

[native-api]

@gaborbernat I confirm that with HEAD version, the virtualenv created with default options looks tarballable.

I'm though okay with using --seeder=pip since the speedup with app-data is very noticeable and our use case is indeed an unusual one.

So, what are your plans here? Do you plan to deprecate/remove --seeder=pip in the foreseeable future? Do you wish to eventually return symlinking as the default (just not so drastically)?

[gaborbernat]

I think the pip seeder will be kept forever... Going ahead we'll keep copy for the app-data one the default, and allow users to change to symlink manually. 🤷‍♂ so the plan is to keep the status quo post this release for this major version (20.x series).

[gaborbernat]

Hello, a fix for this issue has been released via virtualenv 20.0.2; see https://pypi.org/project/virtualenv/20.0.2/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-2-2020-02-11) . Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.

[native-api]

We went the --seeder=pip route, but nevertheless, confirming that the new tarballs are now live and usable.

[hugovk]

Confirmed working, thank you very much for the quick fix, and also for maintaining and updating virtualenv! I've seen about a 30 second speedup running tox on a Mac.