when executed as a wheel, pip creates a temporary file containing ca certificates which is not cleaned up

[asottile]

Description

a temporary file is leaked during the execution of pip as a wheel. repeated execution consumes ~250KB per execution which quickly fills up disk if run in a loop

Expected behavior

I expect the temporary file to be cleaned up instead of left around after execution

pip version

21.3.1

Python version

3.8.10

OS

ubuntu 20.04

How to Reproduce

python3 -m pip download pip==21.3.1
PYTHONPATH=pip-21.3.1-py3-none-any.whl python3 -m pip download astpretty

Output

$ du -hs /tmp/tmp*
du: cannot access '/tmp/tmp*': No such file or directory
$ PYTHONPATH=pip-21.3.1-py3-none-any.whl python3 -m pip download astpretty
Collecting astpretty
  Using cached astpretty-2.1.0-py2.py3-none-any.whl (5.3 kB)
Saved ./astpretty-2.1.0-py2.py3-none-any.whl
Successfully downloaded astpretty
$ du -hs /tmp/tmp*
256K	/tmp/tmpf4qxtczw

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

Using pip directly from a wheel is not supported, as far as I'm aware.

[notatallshaw]

I can reproduce but only by running it exactly as you listed (which is setting the wheel as the pip module?):

 PYTHONPATH=pip-21.3.1-py3-none-any.whl python3 -m pip download astpretty

If I simply run:

python -m pip download astpretty

I can not reproduce.

[asottile]

Using pip directly from a wheel is not supported, as far as I'm aware.

this is (for instance)

• how virtualenv bootstraps dependencies: https://github.com/pypa/virtualenv/blob/21bd7cbf379ff6c6c80276875f84ff2d8695334f/src/virtualenv/seed/wheels/acquire.py#L99-L121
• how ensurepip runs pip: https://github.com/python/cpython/blob/77195cd44b2506cda88a3cfc98918526068b1d46/Lib/ensurepip/__init__.py#L80-L93

[asottile]

here's a reproduction using venv (via ensurepip):

$ du -hs /tmp/tmp*
du: cannot access '/tmp/tmp*': No such file or directory
$ python3.10 -m venv venv
$ du -hs /tmp/tmp*
256K	/tmp/tmpq8131b28cacert.pem

(the filename is a little different as python3.10.1 has pip 21.2.4)

[pradyunsg]

Ugh. Can someone check whether --use-pep517 is functional with the wheel-based execution?

https://www.python.org/dev/peps/pep-0427/#is-it-possible-to-import-python-code-directly-from-a-wheel-file

[asottile]

yep looks like that works fine:

$ ~/opt/venv/bin/python -m pip download pip
Collecting pip
  Using cached pip-21.3.1-py3-none-any.whl (1.7 MB)
Saved ./pip-21.3.1-py3-none-any.whl
Successfully downloaded pip
$ python3 -m venv --without-pip venv
$ PYTHONPATH=pip-21.3.1-py3-none-any.whl venv/bin/python3 -m pip install isort --no-binary :all:
Collecting isort
  Downloading isort-5.10.1.tar.gz (174 kB)
     |████████████████████████████████| 174 kB 5.3 MB/s            
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: isort
  Building wheel for isort (pyproject.toml) ... done
  Created wheel for isort: filename=isort-5.10.1-py3-none-any.whl size=103431 sha256=eea14d9ea804d971049b4a1f35d1fde5a1988a3d6adc52a01f84f6ed4afca079
  Stored in directory: /home/asottile/.cache/pip/wheels/17/33/b8/4531f2a5211bf911323d26bbf429c922ef72f0483567135943
Successfully built isort
Installing collected packages: isort
Successfully installed isort-5.10.1

[pfmoore]

Ensurepip doesn't need the certificates, as far as I know (it installs what's bundled with Python). And ensurepip is closely tied to pip, as well as being essentially part of pip's delivery mechanism, so it's a special case. Virtualenv may be doing the same, or it may need the certificates. But it's relying on unsupported behaviour, regardless.

Note that happens in our patch to certifi, which uses importlib.resources.path to extract the file to a temporary location if needed. If someone really wants to modify that patch (we don't call __exit__ on the path context manager) then feel free to have a go. I don't promise that we'd accept such a patch (an even more complicated patch to a vendored library, just to support an unsupported use case, isn't an obvious win).

Or you could submit an issue to certifi, asking them to support running from a zipfile. Then we could remove our patch altogether. That would probably be the most reliable solution, if you care enough to go down that route.

[asottile]

get-pip also uses this mechanism: https://github.com/pypa/get-pip/blob/6043201e89b1193f7bf050979436eff6a7337247/templates/default.py#L130-L136

and I'm pretty sure it needs the certificates in that case since it reaches out to pypi to download the dependencies

[pfmoore]

It looks like certifi already has the importlib.resources code, and they are aware of the cleanup issue - see here. It's definitely worth discussing this problem with them, not with us.

Also, get-pip uses different code which does do the cleanup¹. But the certifi/importlib approach is independent, and get-pip can't fix that.

Footnotes

1. You're right, get-pip needs the certificates - that's why we have that code in there. I believe it predates the importlib.resources mechanism. ↩

[vstinner]

Hi, I also noticed this bug and I found the root issue. It's a bug in Python importlib.resources: python/cpython#93353

[vstinner]

I think that this pip issue since importlib.resources has been fixed in Python.

[pfmoore]

Thanks @vstinner - that was a complicated one to find, thanks for digging into it to work out what the root cause was!