Feature/improve package info handling #161

reorx · 2018-06-25T11:19:18Z

Goals:

Fix the problem of list command fails with FileNotFoundError (package_info.json) #124
Improve package_info related code (involved in show packages version with new switch: pipsi list --versions #78), make the code more robust and easier to read, add comments to important functions, improve function consistency.

re-org the relationship of the functions below:

save_package_info: receives data from installation process, use make_package_info to create the dict, save it to file
make_package_info: in charge of creating the package_info dict from scratch
get_package_info: if the json file exists, get and parse its data; if not, call make_package_info to get the dict

Eventually, get_package_info's behavior becomes consistent and predictable, so now we can use it in list_everything easily and intuitively.

Other changes:

Add an uninstall-pipsi.py script
Add -e to pip when running get-pipsi.py .
Remove duplicated list_everything call

TODO:

Define a class to handle read/write about package_info

for the frequent need to install/uninstall pipsi, also useful for users who just want to uninstall/reinstall.

fix no scripts display when `list --versions`

re-org the relationship of the functions below: - save_package_info: receives data from installation process, use `make_package_info` to create the dict, save it to file - make_package_info: in charge of creating the package_info dict from scratch - get_package_info: if the json file exists, get and parse its data; if not, call `make_package_info` to get the dict Eventually, `get_package_info`'s behavior becomes consistent and predictable, so now we can use it in `list_everything` easily and intuitively.

pass the right scripts to `save_package_info`

as we already have get_package_info to provide us all we need, this function is just confusing and useless

to make it not so confusing when `normalize_package` exists

- move make_package_info to PackageInfo.create_from_venv_path - serialize/deserialize package info by PackageInfo methods to_json and create_from_json - get attributes by dot instead of slice

remove `from os.path import ...`, all functions are now referred by `os.path.xxx()`

because pipsi does not handle multiple version of a same package now, ultimately `pipsi install xxx=1.0.0` and `pipsi uninstall xxx=1.0.0` are seen as `xxx` package under the hood, so we should prompt `xxx is not installed`, `do you want to uninstall xxx` instead of `xxx=1.0.0` with version number.

rename to `list_package_infos`, change its return to list of PackageInfo, change the output rendering in `list_cmd`

smancill · 2018-08-11T21:05:03Z

I confirm this was the only way I got pipsi working.

The command I used was a combination of #162 and this branch:

$ curl https://raw.githubusercontent.com/reorx/pipsi/41ec097047ede9af0381d86080ca4b59cbe51062/get-pipsi.py | python3 - --src 'git+https://github.com/reorx/pipsi.git@41ec097047ede9af0381d86080ca4b59cbe51062#egg=pipsi'

reorx · 2018-08-24T13:53:17Z

@mitsuhiko @RonnyPfannschmidt guys, reviewing of this project has been very inactivate recently, and I'm sad to see that. Pipsi is a very great tool, I personally use it a lot, I think it can make better difference, and it still has a lot of improvement space. I'm wondering if I can be one of the maintainers, I have at lease one day free to do open source coding each week. I really want to push pipsi forward, to release a stable and mature version for all the people who care about it. How do you think? Looking forward to your reply :)

sahensley · 2018-09-06T03:49:37Z

After installing Pipsi from this branch, I now have a functioning Pipsi installation which is backed by Python 3.6.6. I can also verify that the virtual environments are being made via the venv module instead of virtualenv.

I'd love to see @reorx added as a maintainer and possibly a few others to help keep this project afloat.

sciyoshi · 2018-09-27T15:57:35Z

Thanks @reorx, the package_info.json issue causes problems when pipsi shares its venv directory with other tools like pew. 👍 to merging this as well as expanding the set of maintainers if that's something @mitsuhiko would support.

sciyoshi · 2018-09-27T16:04:05Z

@reorx I did notice an issue however:

Packages and scripts installed through pipsi:
  Package "awscli":
Traceback (most recent call last):
  File "/Users/sciyoshi/.local/bin/pipsi", line 11, in <module>
    load_entry_point('pipsi', 'console_scripts', 'pipsi')()
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/decorators.py", line 27, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/Users/sciyoshi/.local/venvs/pipsi/src/pipsi/pipsi/__init__.py", line 607, in list_cmd
    click.echo('    ' + script)
TypeError: must be str, not list

It seems like the file gets re-written with scripts that look like this: {"name": "httpie", "version": "0.9.9", "scripts": [["/Users/sciyoshi/.local/venvs/httpie/bin/http", "/Users/sciyoshi/.local/bin/http"]]}

reorx · 2018-09-28T05:21:34Z

@sciyoshi hmm, I find that too, thanks for reporting, to make pipsi list work temporarily, you can change line 607 to click.echo(' {}'.format(script)). I'm digging further into this problem to see what causes scripts to be a list in list.

change link_script method return to be list of dest path, this makes less confusion, and won't affect script when calling upgrade command

sort paths before removing so that files could be removed prior to dirs

reorx · 2018-09-28T06:26:44Z

@sciyoshi the new push should fix that issue, please check it out. For the infected package_info.json files, please manually fix them by removing the redundant [ and ] ([["/Users/sciyoshi/.local/venvs/httpie/bin/http", "/Users/sciyoshi/.local/bin/http"]] should become ["/Users/sciyoshi/.local/venvs/httpie/bin/http", "/Users/sciyoshi/.local/bin/http"]).

It was a bug in upgrade command that caused the json file to be wrong.

sciyoshi · 2018-09-28T19:32:35Z

  File "/Users/sciyoshi/.local/bin/pipsi", line 11, in <module>
    load_entry_point('pipsi', 'console_scripts', 'pipsi')()
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/decorators.py", line 27, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/Users/sciyoshi/.local/venvs/pipsi/src/pipsi/pipsi/__init__.py", line 570, in upgrade
    if repo.upgrade(package, editable):
  File "/Users/sciyoshi/.local/venvs/pipsi/src/pipsi/pipsi/__init__.py", line 466, in upgrade
    old_scripts = set(info.scripts)
TypeError: unhashable type: 'list'

I also got this issue when upgrading, which I was able to fix by removing and reinstalling.

reorx · 2018-10-02T08:50:10Z

@sciyoshi could you try upgrading using the the updated version of this PR (ensure scripts in package_info.json is not list of list first, see my previous comment)? The latest two commits should have fixed the upgrading issue.

aiguofer · 2018-10-21T15:28:19Z

I was having issues upgrading pipsi installed packages (using master). I can confirm that this branch fixes those issues.

cs01 · 2018-10-21T15:35:35Z

Just wanted to note that I have started pipx which combines the functionality of pipsi and npx. I have implemented several of the features requested in pipsi, including upgrading all packages with one command (pipx upgrade-all) and reinstalling all packages w/ a different Python executable with one command (pipx reinstall-all PYTHON). pipx works with python3.6+ only, and uses venvs only.

I am actively responding to issues and PRs. Would love to hear what you think -- https://github.com/cs01/pipx.

reorx added 16 commits June 25, 2018 16:35

add uninstall-pipsi script

fe624dd

for the frequent need to install/uninstall pipsi, also useful for users who just want to uninstall/reinstall.

ignore Pipfile

b3b8722

add -e to pip when running get-pipsi.py .

5adce37

add debug print for run function

fa1b727

fix duplicate list_everything call

593a088

fix no scripts display when `list --versions`

enhance linked_scripts result process

0c495f1

pass the right scripts to `save_package_info`

remove get_package_scripts

e669dd9

as we already have get_package_info to provide us all we need, this function is just confusing and useless

add check_package_installed

53759b0

rename func normalize -> normalize_path

9a9b79b

to make it not so confusing when `normalize_package` exists

get_package_path returns both package_path and package_name

d5d3208

add PackageInfo class

895f73d

- move make_package_info to PackageInfo.create_from_venv_path - serialize/deserialize package info by PackageInfo methods to_json and create_from_json - get attributes by dot instead of slice

unify variable name venv_path and path

d48f70e

unify os.path functions calling

8324df5

remove `from os.path import ...`, all functions are now referred by `os.path.xxx()`

rename list_everything, use list of PackageInfo as its return value

41ec097

rename to `list_package_infos`, change its return to list of PackageInfo, change the output rendering in `list_cmd`

reorx added 2 commits September 28, 2018 14:19

fix upgrade command causes scripts to be list in list

c1a017c

change link_script method return to be list of dest path, this makes less confusion, and won't affect script when calling upgrade command

enhance error handling for uninstall when removing paths

d31fad0

sort paths before removing so that files could be removed prior to dirs

sciyoshi mentioned this pull request Sep 28, 2018

Better fallbacks #177

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/improve package info handling #161

Feature/improve package info handling #161

reorx commented Jun 25, 2018 •

edited

Loading

smancill commented Aug 11, 2018

reorx commented Aug 24, 2018

sahensley commented Sep 6, 2018

sciyoshi commented Sep 27, 2018

sciyoshi commented Sep 27, 2018

reorx commented Sep 28, 2018

reorx commented Sep 28, 2018

sciyoshi commented Sep 28, 2018 •

edited

Loading

reorx commented Oct 2, 2018

aiguofer commented Oct 21, 2018

cs01 commented Oct 21, 2018

Feature/improve package info handling #161

Are you sure you want to change the base?

Feature/improve package info handling #161

Conversation

reorx commented Jun 25, 2018 • edited Loading

smancill commented Aug 11, 2018

reorx commented Aug 24, 2018

sahensley commented Sep 6, 2018

sciyoshi commented Sep 27, 2018

sciyoshi commented Sep 27, 2018

reorx commented Sep 28, 2018

reorx commented Sep 28, 2018

sciyoshi commented Sep 28, 2018 • edited Loading

reorx commented Oct 2, 2018

aiguofer commented Oct 21, 2018

cs01 commented Oct 21, 2018

reorx commented Jun 25, 2018 •

edited

Loading

sciyoshi commented Sep 28, 2018 •

edited

Loading