windows in-place pip upgrades

[qwcode]

not sure of the status of pip install --ugrade pip on windows.

need this for PEP453

@pfmoore ?

see #188 for an old discussion on this.

[pfmoore]

Use python -m pip install --upgrade pip. I think @vsajip was considering some form of magic in the exe wrappers to mean that self-replacement of the exe could be made to work. But I don't know if he did anything in the version of distlib we're using. I'm not actually sure it's something that could be done without help from the application (pip). And I'm pretty sure no code has been added to pip for this.

Personally, I consider this to be a minor issue - telling people to use `python -m pip`` seems to me to be a perfectly acceptable answer, given that the issue is fundamental to how Windows handles deletion of open files.

Ultimately it's going to be a question of how much complexity (with the consequent maintenance cost) and risk we want to take on in order to solve this.

[dstufft]

Personally i'm ok with pip install --upgrade pip printing a message to use python -m pip install --upgrade pip. I just don't think we should throw an error about open file handles or whatever the exception is.

[pfmoore]

I was just about to say the same thing.

Here's a question - would you want that message to only appear on Windows, or would it be reasonable to say the same on Unix (for a consistent user experience). Personally, I'd like to be consistent across platforms (because it's easier to document that way).

Also, we should trap any command that attempts to (re-)install pip - pip install -I pip is another one, for example.

[dstufft]

Personally I think it should be windows only. I view it as a work around for the way the Windows platform works, and something that's a target to eventually remove for something nicer down the road.

[pfmoore]

I knew you were going to say that :-(

Frankly, I don't believe we will ever get "something nicer", basically because we don't have enough Windows expertise to develop and maintain it. I know of no Windows software with a satisfactory self-replacement of executables solution. I certainly don't have an answer, and I wouldn't expect any of the Unix-based developers to be able to help. Even if we did get something, it's likely to be complex and so a probable source of bit rot and obscure bugs.

Making a solution that works on all platforms the "official answer" seems to me to avoid sending confusing messages to the user. Back it up with a warning as described and we have a robust solution, even if it's not quite as perfect as getting pip install -U pip to work.

Please excuse a certain level of mild grumpiness in this reply, caused by a feeling that it's always the Windows platform that has to be the one to compromise - I don't mean to make an issue out of this. In practice, I'll go with whatever the consensus is. I don't think it's a big deal.

[dstufft]

FWIW my goal after PEP453 is done is to get our own testing infrastructure stood up which includes Windows, so hopefully that'll solve some of the windows bitrot problems.

I can understand the grumpyness though :) If we were designing this system brand new I'd be a lot more amendable to that but I don't think it's user friendly to break pip install -U pip for everything non Windows in order to maintain consistency. If this was a brand new thing then there's no (or less at least) user epectation :]

[pfmoore]

OK, that sounds fair. You're right that historical usage is more important here.

I actually have no feel at all for what the user experience on Unix is, so I'm probably just as guilty of one-sided thinking as anyone else. I must try to see how using Python on Linux feels - is there a good "Python on Linux for beginners" resource that shows how to set up a Python (3) environment, get the various tools set up, etc? [One thing I always find doesn't get covered, is how to make "python" run Python 3 by default, how to make "virtualenv" build Python 3 virtualenvs by default, all that sort of thing - it seems to me that Python 3 is much more second-class on Unix than on Windows, oddly enough.]

[dstufft]

Typically you don't want to make python point to python3 on *nix because the system uses Python and not all of their libraries are going to be ported to Python3. However there are some Linux distros that have Python3 as a default now... I think Arch? And Maybe the next Fedora?

[pfmoore]

Hmm, I'm not talking at a system level, but really you can't choose what to use for Python in your personal account? That definitely makes Python 3 second-class, to my way of thinking... But it does explain why Unix users don't seem to find using things like "pip3" or "pip3.3" as ugly as sin :-)

(Anyway, we're way off-topic here, so never mind).

[dstufft]

You can setup your shell so it has ~/bin/ before the system bin and add a symlink there from python to python3, but yea offtopic. I'm happy to help though if you're actually interested in getting things set up :] IRC or Email.

[vsajip]

My work which I believe Paul was referring to was just to implement the 'deleteme' dance for replacing files in use (i.e. exe wrappers) - that's in distlib Python code, not in the .exe wrappers. This is in the distlib.scripts module and should be exercised whenever that module is used to create a script.

[dstufft]

So perhaps this is already solved for Wheels? And it's a setuptools problem elsewise?

[pfmoore]

Sounds possible. I'll do some tests today, if I can.

[pfmoore]

Sadly not, at least with "pip install -I pip". (It's hard to test an actual upgrade, as only the absolute latest version of pip has script wrapper support):

>pip install -f $x -I pip --use-wheel --no-index --pre
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking pip
Installing collected packages: pip
  Found existing installation: pip 1.5.dev1
    Uninstalling pip:
      Successfully uninstalled pip
Cleaning up...
Exception:
Traceback (most recent call last):
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 365, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'c:\\users\\uk03306\\appdata\\local\\temp\\pip-zhbsiy-uninstall\\work\\scratch\\xxyy\\scripts\\pip.exe'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\basecommand.py", line 121, in main
    status = self.run(options, args)
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\commands\install.py", line 258, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\req.py", line 1330, in install
    requirement.commit_uninstall()
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\req.py", line 557, in commit_uninstall
    self.uninstalled.commit()
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\req.py", line 1743, in commit
    rmtree(self.save_dir)
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\util.py", line 42, in rmtree
    onerror=rmtree_errorhandler)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 460, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 362, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 362, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 362, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 362, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Work\Scratch\xxyy\lib\shutil.py", line 367, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Work\Scratch\xxyy\lib\site-packages\pip\util.py", line 61, in rmtree_errorhandler
    func(path)
PermissionError: [WinError 5] Access is denied: 'c:\\users\\uk03306\\appdata\\local\\temp\\pip-zhbsiy-uninstall\\work\\scratch\\xxyy\\scripts\\pip.exe'

Storing complete log in C:\Users\uk03306\pip\pip.log

[vsajip]

So pip's not using distlib.scripts to install scripts ... I guess it doesn't fit in the pip workflow.

[pfmoore]

I suspect it's because we do an uninstall then an install, rather than an install overwriting what's there (for obvious reasons).

[agronholm]

Windows doesn't allow overwriting or deleting open exe files, but it does allow renaming/moving them.

[rgbkrk]

Well, with some tomfoolery with the Win32 API you can close the handle to pip.exe much like you can when using Sysinternal's process explorer.

[pfmoore]

I don't think you can close all handles to the currently running exe. But if you have working code as an example, I'd love to see it, as this has long been a problem that people have debated and the consensus has always been that it's not possible (except by tricks like "rename and delete later"). Working code that does this would be extremely useful to the community.

[techtonik]

Run into the same issue.

As a quick hack, can pip.exe detect that it is being upgraded prior to execution and restart itself with python -m pip {{ sys.argv }}?

[dstufft]

I don't think that actually works.

But maybe a stupid thought, afaik the .exe doesn't need to be updated, can we just special case pip.exe and leave it alone? Would possibly leave a pip.exe laying around after an install though.

[pfmoore]

@dstufft correct, "restarting yourself" isn't as simple as @techtonik makes it sound on Windows, where there's no equivalent of the Unix exec call.

Technically, pip.exe may need updating if the entry point changes or if the distlib wrapper code is changed. That's not common, but it can happen (early on, the distlib code changed a fair bit till things settled down). What it may be possible to do would be to write the new pip.exe to a temporary location, then if it's identical to the current one, skip it. You wouldn't avoid errors when it did need updating, but in the 99% of cases when it didn't, you'd be fine.

BTW, we're going round in circles here. According to Vinay's earlier comments, this is solved in distlib (which is where I think it should be solved) but pip doesn't use the API in a way that allows it to work. If we could use the distlib solution, then great. Otherwise, we need someone to write a pip-specific solution (and I remain skeptical that this can be done in a robust, maintainable manner that isn't far more complex than the issue warrants).

Suggestion:

1. Someone add a section to the pip docs at http://www.pip-installer.org/en/latest/installing.html#install-or-upgrade-pip noting that the python -m pip workaround is needed on Windows when upgrading pip with itself. I might get a chance to do this at some point, but PRs from people who have contributed to this issue would be gratefully accepted :-).
2. If anyone has a viable solution, and is prepared to implement it, they should raise a PR and we can review actual code rather than proposed approaches.

I think we're at the stage where actual working code is the only thing that will move this issue forward.

[techtonik]

I am -1 on bloating the docs without trying other solutions.

[qwcode]

I'm +1 on documenting what works right now.

[techtonik]

Even with doc comment it is still workaround - not a solution, and you've got command prompt for Windows wrong.

[mikofski]

Maybe this is crazy but I always just do this:

$ curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python

of course this requires one to have libcurl, but there are windows binaries available here.

[pradyunsg]

@ofek #4490 made a change in pip, printing a better message, IIRC.

[ihormelnyk]

So for now if I've installed python without pip I will not be able to install pip module in future?

c:\Python27>python.exe get-pip.py ERROR: To modify pip, please run the following command: c:\Python27\python.exe -m pip

c:\Python27>c:\Python27\python.exe -m pip install c:\Python27\python.exe: No module named pip

[pradyunsg]

Hey @ihormelnyk!

That's not the case. The conditional for this was too broad -- it blocks using any script with the word "pip" in it from modifying pip. You can try renaming get-pip.py to not contain the word "pip", to work around this issue.

A 10.0.1 release (which will likely happen this week) will fix this issue.

[AlexArcPy]

@pradyunsg thanks for sharing the tip. I had the same issue as @ihormelnyk and renaming the get-pip.py file to a name that does not contain the word pip helped.

[BJQ98]

I print pip in cmd inwindows,but it showed
Fatal error in launcher: Unable to create process using '""d:\program files (x86)\python3.6\python.exe" "D:\Program Files (x86)\python3.6\Scripts\pip.exe" '
I have already set environment for pip,but it doesn't work,how to figure it out?

[pradyunsg]

@BJQ98 You'd want to look at #5223.

[supermansaga]

pradyunsg said 10.0.1 would fix this, but i don't know how. Thanks.

C:\Users\clarkk\AppData\Local\Programs\Python\Python36>python -m pip install -U pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 9.0.3
    Uninstalling pip-9.0.3:
      Successfully uninstalled pip-9.0.3
  Rolling back uninstall of pip
Exception:
Traceback (most recent call last):
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "C:\Users\yangg\AppData\Roaming\Python\Python36\site-packages\pip\req\req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\req\req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\wheel.py", line 462, in move_wheel_files
    generated.extend(maker.make(spec))
  File "C:\Users\yangg\AppData\Roaming\Python\Python36\site-packages\pip\_vendor\distlib\scripts.py", line 372, in make
    self._make_script(entry, filenames, options=options)
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\_vendor\distlib\scripts.py", line 276, in _make_script
    self._write_script(scriptnames, shebang, script, filenames, ext)
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\_vendor\distlib\scripts.py", line 212, in _write_script
    launcher = self._get_launcher('t')
  File "C:\Users\clarkk\AppData\Roaming\Python\Python36\site-packages\pip\_vendor\distlib\scripts.py", line 351, in _get_launcher
    result = finder(distlib_package).find(name).bytes
AttributeError: 'NoneType' object has no attribute 'bytes'
You are using pip version 9.0.3, however version 10.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

[pradyunsg]

That'll be the same as #5343.

[supermansaga]

Thanks pradyunsg, Someone else's solution turns out to be simpler. Just performed "python -m pip uninstall pip" first, then "python -m pip install -U pip"

Requirement already up-to-date: pip in c:\user\clarkk\appdata\local\Programs\Python\Python36\lib\site-packages (10.0.1)

[littleOStar]

https://pip.pypa.io/en/stable/installing/#upgrading-pip

[nidhi100]

      Personally I think it should be windows only. I view it as a work around for the way the Windows platform works, and something that's a target to eventually remove for something nicer down the road.

I am getting error while running above command as syntax error to the word get what might be the solution for this

[pfmoore]

@AProNoob You should be running the commands at a command prompt (cmd or powershell), not at a Python prompt.

[AProNoob]

Thanks! That worked!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.