How about joining up for Python code completion? (YouCompleteMe) #119
Comments
This was my answer:Hey Val! I don't have much time right now, I will answer you later in detail. I really thank you for bringing this up. That's basically the idea of open source. Working together and improving things. It also comes in quite handy for me, because I don't really have time to maintain jedi-vim a lot, because of all the things going on in Jedi. My main question would be here: Where does jedi-vim fit in? Would you deprecate jedi-vim in favor of YouCompleteMe? Or would it be more intelligent to let them co-exist and let YouCompleteMe do the work if available? The question is also relevant because some features in jedi-vim are probably Python-specific. Additionally jedi-vim plans to integrate a few refactoring things. What do you think about this? ~ David PS: Could I post this mail and my answer on github? So that everybody can be a part of the discussion? |
I'd love to see refactoring commands in YCM! This is the reason why the OnUserCommand API was created. I know I'll be building more IDE features for C-family languages in YCM using libclang. I think this makes complete sense: since YCM already has an in-memory AST of your C/C++/ObjC file and has access to a library that provides a complete semantic understanding of the codebase, why not expose those features? It would be great if features like these were also implemented in YCM for non-C-family languages like Python too. Naturally, those features would be Python specific; this is fine, there will be C++-specific commands too. The user would access them through If you decide to start contributing to the jedi_completer in YCM, what happens to jedi-vim is up to you. It seems like you don't have time to maintain it and I've mentioned that I think there's a better way than duplicating the work for a good code completion Vim infrastructure. It's already built in YCM. So jedi-vim... I don't know. Do you want to continue to develop it in parallel? That's certainly up to you. If you'd rather focus on jedi the library and jedi_completer in YCM and would rather turn down jedi-vim, put a deprecation notice at the top of your README pointing out that it's not being maintained anymore and point people to YCM (I feel like an egotistical dick just mentioning this). Again, it's up to you. If you want to work on both in parallel, I'm fine with that. |
|
I haven't contributed much to jedi-vim, but I think this makes a lot of sense. I haven't tested YCM yet, but the description and GIF look awesome. One important thing is not yet answered though: Is it possible to install YCM without all the C/C++ specific parts (e.g. Clang)? I'm using those things from time to time, but I'm pretty sure there are many people that don't, and that wouldn't want to install unneeded dependencies to complete their Python code. As for the "migration process", I'd try to fix any important open issues and then add a notice that jedi-vim is not being developed anymore and that people should migrate to YCM. Anyways, I'll try out YCM when I find the time to do so, but generally I'm all in favor of replacing jedi-vim with YCM, as long as the issue above is guaranteed. If there are any issues with YCM, I'll post them here or at the YCM github page. |
Yes, the default install does not build the clang parts. YCM still has a compiled component (ycm_core.so) but that's what makes YCM completions super-fast (installation is trivial and amounts to running |
|
Sounds great :) |
You could probably use Python threads. But I never used them. At the moment I'm not sure if it's a good idea to close jedi-vim (because YCM) is nowhere where jedi-vim is (pydoc support, function display, etc.). The first step would be to use YCM if available (if you know what I mean). My personal problem is at the moment that I don't even have the time to do that (look at my commits in the last few months in jedi-vim). I only do Jedi stuff at the moment. So for me the big issue at the moment is my time. A thought for you @Valloric: YouCompleteMe is VIM specific, why not create an engine (like Jedi), that could be of general use? This wouldn't necessarily mean that you need to split YCM. I guess there are quite a few IDEs which would love to have good autocompletion and do that all by themselves. |
The jedi_completer does in fact use a Python background thread. But you still need a native code binary to get fast subsequence-based matching.
Yep, those features will come.
I completely understand; YCM takes up a lot of my time too. Feel free to send a pull request at some point if you feel the need/have time.
I haven't discounted this option actually. Some emacs guys I know have talked about getting something similar for emacs by building on YCM as a base. But it's not really a current goal. |
what's that? |
|
From the docs:
|
|
Yes, subsequence-based matching is awesome. It's actually the main reason why I started YCM; in a past life I used IDEs and a lot of the good ones have subsequence-based matching. The GIF on YCM's page shows how it works in practice: I can type |
|
+1 for getting jedi through YCM! |
|
👍 While this is happening, I'd love it if I could at least load YCM and jedi-vim simultaneously and get the best of both worlds, without the two stepping on each other's feet. (Currently, this doesn't work properly--completions appear out of nowhere and such.) |
Absolutely. That's the important goal for now. |
|
Just an update: since a few days ago, YCM now supports GoTo commands for Python, using the Jedi library (same as jedi-vim). See the YCM README for details. Am I correct in saying that the last major features that jedi-vim has for Python that YCM needs to gain are:
|
|
Refactoring commands have not been implemented yet. But that will probably soon happen. I think your right. That's about it. |
|
@Valloric great, in that case I need to update immediately :) |
|
I added YCM to the plugin list in the Jedi README: davidhalter/jedi@06cd975
|
|
Blogpost: http://blog.dbrgn.ch/2013/5/27/using-jedi-with-ymc/ (Correct me if something's wrong or missing.) |
|
Thanks for adding it to the docs! I don't see anything wrong in the blog post (thanks for writing that too). |
|
Just as a side-note: jedi-vim will not be deprecated. I'm probably including more features than YCM will. Also I'm much more flexible to test things. That doesn't mean that I don't think YCM is great, it is! Are there any issues at the moment, that those two wouldn't work well together? |
|
YCM uses Jedi internally, but conflicts with jedi-vim. I don't see how it would be possible for the user to have both plugins running at the same time in a Python filetype. YCM is still missing a few jedi-vim features, but I intend to add them all eventually. |
|
how can i make sure i enable jedi-vim for python and others and ycm for ruby? |
|
I don't know. It's easy to disable jedi-vim features. I don't know how that would work with ycm. |
|
I am using jedi-vim together with YCM... In order to prevent conflicts, I've disabled most of jedi-vim that is related to completion. " jedi-vim {
let g:jedi#auto_vim_configuration = 0
let g:jedi#popup_on_dot = 0
let g:jedi#popup_select_first = 0
let g:jedi#completions_enabled = 0
let g:jedi#completions_command = ""
let g:jedi#show_call_signatures = "1"
let g:jedi#goto_assignments_command = "<leader>pa"
let g:jedi#goto_definitions_command = "<leader>pd"
let g:jedi#documentation_command = "<leader>pk"
let g:jedi#usages_command = "<leader>pu"
let g:jedi#rename_command = "<leader>pr"
" }I get to fully use YCM and still earn some extra nice jedi-vim features such as better argument completion. |
|
To disable YCM for python i've put the following snippet in my .vimrc: let g:ycm_filetype_specific_completion_to_disable = { 'python' : 1 }
let g:ycm_filetype_blacklist = { 'python' : 1 }So far seems like everything is going as planned. How is the conversation about a merger/integration going? You probably already thought about this, but how hard it is to make jedi-vim a module of YCM? |
|
@josecostamartins Both jedi-vim and ycm are using Jedi. Jedi is doing almost all of the work. The difference of jedi-vim and ycm in behavior and features is small. I wrote a small answer detailing why I like jedi-vim to be a different project, you can read it below. However, the reason why I won't merge those two projects is basically my limited time. I'm working like 99% on Jedi and 1% on jedi-vim. I'm not really doing anything on jedi-vim, because I think Jedi is way more important for the future of the Python ecosystem and nobody else is going to do that work in the near future. I'm very happy to see people improving YCM and jedi-vim. Seriously. I don't even look at them as competitors anymore. They are two ways of implementing Jedi. Both are fine. But if you're really interested in good autocompletion you'll start improving Jedi :-)
On a side note, I still think that YCM hasn't tried to "reach the stars", by limiting itself to be a VIM autocompletion engine. I would much rather have see a library/engine like |
I agree with this 100%. Several months ago YCM was split into a client-and-server model where the vast majority of the YCM logic lives in the ycmd completion server which has nothing to with Vim. There's also a thin client that runs in Vim that talks to ycmd. Some time in the near future, ycmd will be split into a separate project so that other editors can build clients that talk to it. I see ycmd as the one completion server that editors talk to; ycmd itself ships with and connects to various semantic engines (Jedi, libclang, omnisharp etc.) but provides one common interface to talk to all of them. That's the idea at least. |
|
Awesome! :-) Well maybe one day we can merge our efforts... However, davidhalter/jedi#385 is going to be implemented first anyways. |
davidhalter/jedi#385 sounds like a great idea! I'd love to connect to a Jedi server instead of importing Jedi as a Python library. YCM already uses OmniSharpServer instead of calling into the OmniSharp library directly. Having a server for this is the right approach. Some advice: use JSON over HTTP as the RPC interface. It's very easy to use and the overhead is negligible for local calls. This combo works great for ycmd, OmniSharpServer and Tern.js (server for JS code completion). |
|
Thank you for the detailed analysis. Doesn't surprise me at all. Since I know what Jedi does internally (it's like a crazy amount of things), the server won't eat too much. The big issue is actually implementing this for different Python versions and setting up a good protocol. Is there a protocol for autocompletion that you think looks like it should? If not, what's typically missing in those protocols? Examples? |
Honestly I'd recommend looking at the ycmd's protocol for inspiration. It isn't documented anywhere but eventually will be when ycmd becomes it's own repository. Parts of the API won't be relevant, but a lot of it will be. If you want to examine it, the server handlers are here. You should be able to follow the logic from there. The tests are certainly also worth taking a look. Here's what the completion tests look like. This API is sound and has been working well for months now. The one thing I want to change before releasing the server and the API externally is make line and column numbers in the API be 1-based instead of 0-based since editor API's are all 1-based as are all diagnostic messages (compilation errors & warnings) shown to the user. |
|
I'd also like to stress how important it is to focus on making your server API easy to interface with for clients. This is why HTTP+JSON is such a good combo here: every (sensible) language has great support for HTTP and JSON and both of those are very well understood by developers worldwide. Going with something like Thrift, zeromq or any kind of binary protocol makes no sense since you do not need that kind of performance (as proven by both the benchmarks I linked to and experience with ycmd over the last 6 months), but would still have to pay the cost of ease of use. |
Yes I fully agree. Thank you for bringing this up again, it will make my life way easier defending this position. Just to quote Knuth again:
HTTP/JSON is a good option. I don't even think there's a lot of good alternatives to transfer data like this. The only thing I would actually consider is to use Python's pickle, which is comfortable if your client is also written in Python (but I would prefer something language agnostic).
Any other protocols? Or do you have any examples of how no to do it? |
|
Jumping in the thread via davidhalter/jedi#385
Sounds great. It would be nice to have Emacs plugin to talk to ycmd.
The reason why I proposed ZMQ in davidhalter/jedi#385 is that you need many-to-many communication not for performance, especially for Python/Jedi. Read davidhalter/jedi#385 for why. That said, I agree it is a good idea to have HTTP server "frontend" to communicate with editors ("translator") for easily accessible API. Probably that can be in ymcd: Jedi wrapper written around ZMQ and HTTP-based frontend in ymcd translating ZMQ to HTTP. |
|
Why do we need many-to-many connections? Sorry I don't understand the argument in #385. |
|
Let's say you have two processes running Jedi using Python 2 and 3, P2 and P3, and you have two editor processes E1 and E2. Connecting to E1 to P2 and P3 and E2 to P2 and P3 requires many-to-many connection. If you don't care about memory then you can launch a set of P2 and P3 for each editor but you still need many-to-one. Furthermore, if you want to mix the result of P2/P3 and doing subseq match on-the-fly then you'd want to something like iterator via RPC. I don't say it's impossible to do that with HTTP. It's just it's better to use a right tool at the right place; as I said, for editor-service communication HTTP is a good idea. |
|
@Valloric BTW, if you consider refactoring, then maybe you'd need a dialog-like UI. In that case, bi-directional RPC rather than HTTP could be a better option. Again, you can do that in HTTP with some more code. But I think HTTP is not a right tool to bring in this case too. |
|
@tkf You're assuming that you want to have one Jedi server for several clients. I thought of the same for YCM, and it was unnecessarily complex (and limiting). The simplest and easiest way is for each client to start up its own copy of the completion server. This approach has been working for months without any issues for YCM. If you try to have a single (or multiple) shared copies of the completion server, this limits what each client can do with it. Each client may want to start the completion server with different options to serve different needs. You may also want (or more likely, be forced) to store state on the server between completion requests. If you do store state, knowing that there's only one client talking to you simplifies things greatly. My advice boils down to the KISS rule and YAGNI. Don't try to engineer something that works with multiple clients that you're unlikely to have; also, don't use a complicated protocol like zeromq that's designed for super-low overhead which isn't your concern. In other words, don't design a cathedral. They're really hard to build. Design a small hut and build that, and if you need a cathedral later on (unlikely), extend the hut. |
|
@Valloric I consider your point already in my comment. See the last half after "If you don't care about memory..." about many-to-one connection. And don't miss the next comment about bi-directional RPC. BTW, I understand and aware of most of your point about simplicity. My aim in davidhalter/jedi#385 was to start discussion. Nobody puts crazy ideas so I had to put ones.
Funny you say that. Isn't ycmd going to be a cathedral? I mean, if it is going to kill huge amount of birds (tools for many languages and many editors/IDEs) then it's a good cathedral :-) |
|
(Came here from jedi and found the discussion interesting) Re http, isn't there an issue that on some systems (e.g. windows on a heavily locked-down network) using http even on localhost runs afoul of anti-virus/firewalls. Perhaps you could offer the option of using pipes as an alternative to http for RPC? |
|
Hey all, coming in as a spectator here... this discussion hasn't been updated in 5 years. What is the current state of things? Is the jedi integration into YCM complete to the point where jedi-vim is "deprecated", or is that not true and jedi-vim still has features for python that YCM does not have? Asking as someone with both plugins currently enabled, and suffering issues, in their .vimrc! |
|
I've been using YCM with Jedi for years now, it's working great 🙂 On the other hand I only use completion, goto-definition and show-docs. I think the only other feature that jedi-vim supports and YCM does't(?) is renaming, right? In any case, I much prefer YCM's speed and complete-as-you-type over an explicit completion trigger. |
|
I think there are still a few differences. jedi-vim does a few additional things like call signatures, There's also deoplete-jedi if you're looking for async completions. This is what feels faster with YCM. @blueyed might know a bit more about the differences than me. YCM itself is not faster, it's just asynchronous. As long as you wrap Jedi, Jedi will be the botteneck. jedi-vim itself is a very small wrapper around Jedi. |
|
I came to this issue because I have been having very bizarre issues on my Mac after using Vim. For example, my main OS X copy paste getting obliterated and pasting random python code whenever I hit command-V! Sometimes the python code contains the word "jedi". So I started investigating whether I possibly have something bad going on in my .vimrc and started commenting out certain plugins and researching whether some of my plugins might "overlap", and sure enough I came here. And I seem to have solved the clipboard problem and so I believe some bad combination of plugins was causing the issue. I would have to take a video to see show bizarre the paste becomes after using Vim, sometimes killing So that aside, my goal isn't to stand behind one tool or the other and say it's better, it's only to have multiple language support (python, js, erlang, c++, etc) in a set of non-overlapping orthogonal plugins. If the simplest solution to that problem is to just use YCM that's my vote. If it's to use jedi-vim for python and other plugins for other languages , that's great too. So I was just looking for guidance. For now I will roll with YCM but I was open to suggestions in using jedi-vim + other plugins for other languages. Oh, and thank you both for taking the time to make vim-as-an-IDE better! |
|
@tommyjcarpenter |
|
@blueyed Empirically, commenting out jedi-vim has solved the issue. This is not a thorough analysis because I have not slowly investigated this issue by turning on/off plugins one by one to find the exact culprit. My current hunch is that jedi-vim alone is not to blame, but rather some unfortunate combination of non-orthognal plugins causing an unexpected issue. Since sticking to just YCM I have not seen the issue, yet. |
|
I think that it's save to say that we have in some ways worked together. Jedi is used within YCM and jedi-vim has always been a side-project. The times have been changing and language servers have emerged as the new market leader. I still think that language servers have a bit too much overhead (both in complexity and resources), but that's probably not going to stop people from using them. So I'm glad YCM, jedi and jedi-vim have become what they are now. Cheers to a few more years! |
[Replicating our private email discussion in a more public forum]
I'm the author of a YouCompleteMe (YCM), a recently released (Feb 2013) OSS code completion system for Vim. It's become very popular very quickly (~1300 stars on GitHub).
YCM comes with out-of-the box support for semantic completion for C-family languages like C/C++/Objective-C/Objective-C++ (using libclang as the engine) and also has a language-agnostic identifier completer. The GIF on the project page is a thousand words and there's also an extensive introduction there so I'll spare you the details that you can easily grab.
Getting to the point, an external developer contributed a Jedi-based semantic completer for YCM today. YCM actually provides a generic framework for writing custom semantic completers (for any language) that can then take advantage of the code completion infrastructure that YCM provides (non-blocking operation that lets Vim continue to process events, a smart caching layer, the subsequence-based completion filtering system and the advanced completion ranking algorithm etc). You can see just how small the file for jedi-based semantic Python completion is in YCM.
I see that Jedi (which is freaking awesome btw, congrats on that) also has other features too, like goto etc. Any chance on us teaming up to bring those features to YCM? I see that you have written the jedi-vim plugin. While I haven't tried it I'm sure it's awesome (hey, it's using Jedi) but I think it's sad that we have this duplication of work. I'd also like to see YCM become a common framework for code completion in Vim, much like Syntastic is the common framework for displaying compilation/lint errors.
YCM will expand into more IDE features with go-to-definition/declaration. Basically all the useful features that one can build when one has an engine that provides a semantic understanding of the codebase. These features would be accessed through the new
:YcmCompleter [completer sub-command] [sub-command arguments]interface that also recently landed. That command calls OnUserCommand on the Completer object with the arguments the user provided after the sub-command name. From then on, the Completer writer can do anything he wishes.So my idea is to expand the functionality of YCM's jedi_completer until it does everything jedi-vim can. That's it. I'll probably end up doing all this myself if no one else does but I'd love it if you could contribute pull-requests. Correct me if I'm wrong, but your goal is to build the best possible Python code completion system. You shouldn't have to spend time on the VimScript parts and other Vim plugin boilerplate. You should be able to focus on making Jedi, the library, even better.
If you're not interested, that's perfectly fine. I just think it's sad that there's this duplication of efforts; I spent a ton of time writing the ycm_core library in C++ so that completions in Vim can be super-fast. I'd love to see others leverage that work to improve editing in Vim even more.
The text was updated successfully, but these errors were encountered: