split jedi into external process, v2

[nakamuray]

It is second version of jedi RPC work. (v1, #507)

This time, RPC version of jedi object preserve original jedi compatible API.
I also added config option to toggle RPC code (disabled by default).

[blueyed]

Can this be made configurable in Vim?
May 'use_rpc' could be changed to not use this RPC method when empty, but would otherwise be the path / command to use?

[nakamuray]

Yes, possible.
I'll do that.

[blueyed]

Very interesting, thanks!

How does it compare with regard to performance?
Does it have any benefit over Neovim's method, which also uses an external Python process?

[nakamuray]

How does it compare with regard to performance?

Apparently, it has some performance penalty.
I choose JSON as a serialization method because it is in the standard library. (and I doubt pickle could use to communicate between python 2 and 3.)
But it is not a fastest serialization format.

It may be possible to get some speed by including known-to-be-used variables to the RemoteObject, which reduce RPC round trip.

Does it have any benefit over Neovim's method, which also uses an external Python process?

It works on vim!!!

... In fact, there is not so much benefit over neovim's. If I had to say anything,

• neovim seems to switch python interpreter at :python (or :python3) level. So, if you have vim plugin which need python3.4, but want to use python3.5 to get completion candidates, you can't satisfy both need.
  My code can switch python version independently.
• It may be possible to extend it to run jedi process for each buffers using multiple python versions.
  For example, you'll spawn python3.4 jedi process for buffer A, python3.5 jedi for buffer B and python3.5 with another virtualenv for buffer C to get appropriate completion result.
• Currently, it is not require any external packages to run. No need to install "neovim" package for each virtualenvs you work on.

[davidhalter]

Thanks for all the work!!

I think this is quite an ok solution. However I don't see what it actually improves besides calling arbitrary python versions (which is great!).

What I would want for an async solution is an async server that is not living within jedi-vim. Such a server should be a part of Jedi! There's a good reason for it: This http server can be used for only within jedi-vim, while it could benefit a few other plugins as well.

Hence once we're talking about a more general solution, we would need to talk about session handling (because some parts of the API are lazy). How do we keep that state? How do we do garbage collection on such sessions... etc. etc.

In conclusion: I will not be merging this. It's just - not what we need in the long term. I might merge something that at least contains an async function that makes completion "faster". But I don't see big usage for this.

BTW: JSON will not be the bottleneck at all. It's going to be Jedi itself. We don't have to pick a "fast" protocoll. Jedi is going to be literally a million times slower. Pickle might still be an option though (there are pickle versions aren't there? Or do they collide with python versions?) in case we have or even just pure http/rest.

[nakamuray]

I think this is quite an ok solution. However I don't see what it actually improves besides calling arbitrary python versions (which is great!).

Yes, it only fix version problem, and it is very important for me.
I want my vim which is linked with python 2.7 to get completion from virtualenv I worked on, which is created for python-3.4.
Current jedi.vim state, completion only from standard library of different version of python, is not what I want to get.

What I would want for an async solution is an async server that is not living within jedi-vim. Such a server should be a part of Jedi! There's a good reason for it: This http server can be used for only within jedi-vim, while it could benefit a few other plugins as well.

No, it's not http server. It uses line based JSON protocol on a pipe.
I put this code on jedi.vim, not jedi itself, because it is vim specific needs.
Other editors, such as emacs and atom, already implement some sort of RPC for their need, because they can't load python modules directly.

Hence once we're talking about a more general solution, we would need to talk about session handling (because some parts of the API are lazy). How do we keep that state? How do we do garbage collection on such sessions... etc. etc.

Yes, state management is important. So, why do you stick to use HTTP, stateless procotol?
We can use plain TCP or unix domain socket. In this case, connection closed mean "client exited" and safe to release resources created by the client.

In conclusion: I will not be merging this. It's just - not what we need in the long term. I might merge something that at least contains an async function that makes completion "faster". But I don't see big usage for this.

Sad. Then, I'll try implement it as a separate vim plugin, which override "jedi_vim.jedi" to RPC versioned one.

Anyway, thanks to provide such a great tool!

[davidhalter]

Current jedi.vim state, completion only from standard library of different version of python, is not what I want to get.

It's also not what I want. However, I want a long term solution not something that I have to maintain and throw away again.

Other editors, such as emacs and atom, already implement some sort of RPC for their need, because they can't load python modules directly.

True. But this is not the point. It's all about providing a good API on the side of Jedi to avoid everyone handling VIRTUALENV and different python versions separately.

Yes, state management is important. So, why do you stick to use HTTP, stateless procotol?
We can use plain TCP or unix domain socket. In this case, connection closed mean "client exited" and safe to release resources created by the client.

That is true - I'm no expert in that area. I have just suggested HTTP, because it's used a lot at the moment, but a TCP socket would probably be way better.

How do unix domain sockets and TCP sockets compare?

Oh and for the experts: Is it possible to forward TCP sockets over SSH?