Rust Language Server (IDE support)

[nrc]

This RFC describes how we intend to modify the compiler to support IDEs. The
intention is that support will be as generic as possible. A follow-up internals
post will describe how we intend to focus our energies and deploy Rust support
in actual IDEs.

There are two sets of technical changes proposed in this RFC: changes to how we
compile, and the creation of an 'oracle' tool (name of tool TBC).

Thanks to Phil Dawes, Bruno Medeiros, Vosen, eddyb, Evgeny Kurbatsky, and Dmitry Jemerov for early feedback.

[killercup]

Rendered

[killercup]

Line break → link break

[liigo]

'oracle' is not a perfect name here, since when you google 'rustlang oracle', Google don't know what you intend to search, a compiler tool or a database api?

[daniel-vainsencher]

Seems like "incremental" here is use to describe push-driven in contrast to the lazy pull-driven compilation. Which is more efficient depends strongly on the use case, hence supporting both and combinations is probably useful. If the dependencies, changes (the pushes) and interests (the pulls) are managed explicitly, combining the strategies should be feasible. I would use the word "incremental" instead to describe all of these partial compilations.

[nrc]

The two are orthogonal - you can have lazy incremental, eager incremental, lazy non-incremental, and eager non-incremental.

[daniel-vainsencher]

I'm saying that the reference to incremental in line 101 seems to define it as eager, conflicting the orthogonality set up in lines 110-115. Or maybe that's just the way it reads to me.

[Ericson2314]

@daniel-vainsencher I think the intended distinction is that "incremental" describes what is done (only what hasn't been done before), while "lazy" describes the algorithm of figuring what to do (start at end goal and work through dependency dag). Laziness without caching is non-incremental, and the "push-driven" approach you mention is a different algorithm for incremental compilation than the lazy one. Does that clarify the orthogonality?

[eddyb]

I forgot to comment this on the pre-RFC thread (sorry @nrc), but I prefer rider as the name of the oracle tool, and keeping racer (with @phildawes' permission, of course) as the one-shot completion (and perhaps quickcheck) tool.

The way I see it, racer "races" to give an useful result each time, whereas rider "rides" along an IDE, for as long as there are Rust projects to be handled by it, and can have a slow start without impacting UX.

[kud1ing]

I don't have a name suggestion (yet), but i expect calling it "Oracle" will lead to confusion and legal trouble.

[killercup]

(I like how this RFC concentrates on the name first 😉)

How about rustcage? Either pronounced rust cage or rust sage.

[phildawes]

Personally I'm not entirely sold on the oracle/rider concept yet. I think the fundamental requirement is that we need a stable interface to rustc to support IDEs and plugins. I'm not yet sure whether a long running database process is required/desirable.

Some things that might cause problems:

• The oracle concept implies a single view of the sourcecode, and I think this jibes a bit with the tools that will perform refactoring, reformatting and suggestions. I suspect they will want interactive access to the compiler (compile this pre-processed snippet, what errors occur if I do this?)
• If the oracle were to support completion plugins directly from its database, it would need to have very low latency update turnaround (i.e. on every keypress, in <100ms). I'm not sure how well this would work with it being separate from rustc and not driven by the plugins.

It may be that we have an oracle in addition to a tools-oriented interface to rustc.

(aside: I noticed that the go oracle isn't used by the gocode completion tool, but I don't know the reason for this, it could just be historical)

[daniel-vainsencher]

This style of API, where the IDE queries for what it needs right now, is hard for doing push-driven updates (based on a file changing and being saved), because who knows what exactly the IDE is interested in?

An alternative is to allow IDEs to register interest in some queries (for a typical heavy IDE "all definitions in this project", but in Racer, only a fast changing "whatever is under this cursor location"), and then:

• Use the registrations to notify IDE only of interesting changes.
• Use positive registrations to know someone cares about this at all (even before they ask for it).

[nrc]

Could you explain the push-driven update a bit more please? I am assuming the IDE plugin is pretty much state-less with respect to the oracle's data and whenever the user performs an action (right click, hover, hotkey, etc.) then it will query the oracle for the data it needs (thus why the oracle must be quick).

[daniel-vainsencher]

Two things can drive the oracle to do work:

• Code changed (two major usecases: a drip as in editing or massive as in git pull/automatic refactoring/cargo update). So IIUC, what you've called eager evaluation would be to react to the change in code immediately, I called this push driven. This can easily be a waste of time when you are recomputing something that nobody cares about. However, if in the API the IDE said "I care until X,Y,Z and any updates on them until further notice" then you can be correctly selective.
• IDE asked for something (say, a definition) and not everything has been precomputed in advance. Triggers some lazy (I called this pull driven) computation. This is never a waste of time, but may have unreasonable latency (whoops, I should have d/led those new versions of two crates before, huh?) and might also lose opportunities for doing things in parallel.

[nrc]

I think that for the push-based changes, the IDE calls the update functions (for the big changes, that is why we need to invalidate whole files or directories). But I don't think the result of any of that has to be communicated back to the IDE (other than error messages, maybe) - the IDE won't keep any state about the program - that is all kept in the oracle, which will be updated by the compiler (or the two are integrated together).

You should of the oracle as part of the IDE really, the IDE shouldn't manage state of its own about the program.

Does that sound right? Or am I missing something about the work flow?

[bungcip]

Found some paper related with IDE integration:
http://wasdett.org/2013/submissions/wasdett2013_submission_10.pdf

[daniel-vainsencher]

Apologize for linking my own paper. [1] is a (proto-) pattern language describing methods to support multiple analyses over a changing source base (some from Smalltalk designs, some also used then in Eclipse). Basically, something like the oracle discussed here. One major decision point is: what objects from the analysis persist when the program text is not completely valid? for example, adding a "{" early in a file can be seen to invalidate every scope after it, do we then not use those function definitions in auto complete? Smalltalk solves this by giving most definitions (classes, methods) a persistent identity over time. The text edited only corresponds to a single definition, and updates the canonical version only when saved (and syntactically valid). Rust currently seems to encourage large files, which makes this more difficult, though not necessarily impossible. For example, if we have both a recent valid version of the source and the changed span, we can use the valid old definitions except where valid current versions are available.

[1] http://hillside.net/plop/2006/Papers/ACMConferenceProceedings/Intimacy_Gradient/a15-vainsencher.pdf

[olivren]

Preliminary notes: I'm commenting on this RFC because I started writing a Rust plugin for the QtCreator IDE. This plugin is in a very preliminary state and it does nothing useful for the moment. I don't know much about compilers in general, and I'm discovering the QtCreator API as I write my plugin. So, be suspicious of anything I could say!

What is the rationale for proposing a daemon process + an IPC API? I fail to see an advantage compared to the oracle just being a library with both a Rust and a C API, that I could call directly from my IDE plugin code. It would remove the pain of writing communication code. it would also allow me to instanciate two services at the same time with isolated content.

The only pros I can see for the daemon approach are:

1. Share information between multiple IDE instances.
   It seems like a rare use case, and I doubt there are much data to share.
2. Be easier to integrate in languages with awful FFI capabilities.
   The IPC API could be designed on top of an existing library, if the need is real.

When I investigated to integrate Racer to my IDE plugin, I came to the conclusion that it would be easier to use Racer as a library rather than invoking it as a process and parsing the output.

[eddyb]

Having an API means all sorts of stabilization concerns, and also passing structured data through a C API.

[phildawes]

Would we not have to apply the same stablization concerns to the IPC interface?
(genuine question, what's the difference between a rust api and an ipc api wrt stabilization?)

[nrc]

The major reason is parallelism - we want the IDE to carry on doing its thing whilst the compiler (coordinated by the oracle) compiles the non-urgent parts of changed source. Then the oracle needs to update its database with the results. Although the IDE could handle the threading to make all this work, it is simpler (and more reusable) just to do it all in a separate process.

With a decent IPC library, having an IPC API is not a lot more complex than having an FFI API. And in turns of implementation having a totally separate process is marginally easier.

In terms of stability, I don't think there is much difference between the two. An IPC API might be a little more robust because using an intermediate data structure and having parsing/serialisation adds a little abstraction.

[olivren]

At least for QtCreator, the extension API is already designed with asynchrony in mind. Having to deal with asynchronous IO would be a lot more painful for me than having a simple synchronous API. In fact, if you give me an IPC API, the first thing I'll do will be to wrap IO and parsing into a simple synchronous API. In the end there will just be an unnecessary
indirection layer.

The oracle library will of course deal with its own update work in a dedicated thread, but you will have to have such a thread anyway, with a process-based design.

[nrc]

Hmm, interesting, that suggests implementing the oracle as a library might be a better solution than as a separate process. I'm not 100% convinced, but it seems like a reasonable alternative to consider. Let me have a think about it...

[nrc]

Does anyone know how this would work out in the Java world? How does a JNI/FFI interface compare to an IPC interface?

[eddyb]

I guess this got lost in the bikeshed: the reason I independently came up with the IPC idea is that compartimentalization and serialization is unavoidable.

I have implemented a system where rustc threads are spawned in the background, and there is a duplex request/response channel to control the thread and get data from it.

The messages are owned ADTs, which means rustc-specific data (e.g. an interned string) has to be converted to a general representation (String).

To avoid tying the in-memory format to Rust or C-based representations, an actual serialization format can be used.
Cap'n Proto, for example, had built-in versioning and extensibility.

And, finally, for added reliability and to get rid of the FFI surface, the threads can be moved to a separate process, which is how we end up with the oracle/rider model.

[eddyb]

@phildawes To remove as much latency as possible and to prevent rustc stabilization concerns, I believe that the oracle should use rustc's internal APIs and be behind the stability wall.
That is, the oracle ships with rustc and uses its libraries directly.

I really see no point in having some output format from rustc itself, that's added design complexity and inefficiency for no gain (in case of the oracle, at least).

[phildawes]

Hi @eddyb!

I agree with that idea, if oracle is to be a thing then it makes sense to put oracle behind the stabilization wall and link it directly to rustc.

However I'm worried about the whole oracle thing. Definitely we need a stable interface to rustc. My concerns with oracle being the stable interface to rustc are:

• 'database of code' is seductive, but imposing this top-down abstraction could easily result in incidental complexity (overengineering)
• it might turn out that building the various refactorings and functionality require an interface that doesn't fit well into the database-of-code paradigm.

It already appears to me that completions maybe don't fit naturally into this design, and we've barely got started.

I'd prefer to see us drive the interface out bottom-up from the ide plugins. Try to build stuff using rustc directly, understand what stable interface is required.

[nrc]

'oracle' is not a perfect name here

@liigo oracle is a terrible name! But it does have some precedent from Go, and I can't think of a better one. Suggestions welcome!

[nrc]

@bruno-medeiros syntex is maintained by @erickt, who is a core member of the community (community and moderation teams, as well as just being generally involved), but not part of the compiler team. It is pretty much just a straight clone of libsyntax from the compiler, so improvements to the compiler's parser show up in syntex quite quickly.

Part of the plan with procedural macros/syntax extensions is to present a stable interface for them to work on, at which point syntex gets a lot less necessary (only useful for tools). In the long term I'd like to stabilise enough of libsyntax that tools don't need it either.

There is work going on to make the parser panic-less, it no longer panics on error. I've also been doing some work on error recovery, for example rust-lang/rust#31065 adds some error correction for missing identifiers. Ideally it should be panic-free under normal use and recover from most errors within the next few months.

[nrc]

@nikomatsakis I did not intend the DB to be used for the code completion suggestions. It is useful for find all references, also queries like find all impls of a given trait, find all sub-traits, etc.

[bruno-medeiros]

Still, it seems to me that even in code completion, there is perhaps a role for an oracle

Just to be clear, the Oracle, as in, "the resident process that is responsible for serving requests of various sorts to the IDE", it should handle code completion as well. Even if the data structures that are used to determine the results of code completion are entirely different from the data structures used for say, find-references, it makes no sense for this to be two different processes, or two different tools. This is because at the very minimum, these two operations can share cached AST data, not to mention that eventually (and sooner that later) we will want the Oracle to support functionality to manage/supply dirty editor buffers (ie, use a document that is being edited in memory in an IDE, but has not yet been persisted to disk). Even the functionality I'm coding in the Rainicorn tool should ideally also eventually be integrated into an oracle.

Of course, as an early prototype, it's okay for different operations to be handled by different tools, etc. but the end goal should be to integrate everything in the oracle, all-knowing that it is. 😉 (gotta hand it to the Go guys, they choose the perfect name - apart from the trademark thing.. 😝 )

@nrc BTW, I was just looking at the Nim language, and to my surprise found out they have an "oracle" tool already: http://nim-lang.org/docs/idetools.html
And by the looks of it quite more advanced than the Go-oracle, it actually seems to achieve all those key aspects that were mentioned before:

• Handles incorrect/incomplete code well
• Is a resident process
• Supports managing dirty buffers
• Supports all sorts of IDE query operations in a single tool: resolve-definition ("Definitions"), code-completion ("Suggestions"), find references ("Symbol usages"), etc.. (Something like parse-analysis seems to be missing though)

[bruno-medeiros]

Ideally it should be panic-free under normal use and recover from most errors within the next few months.

Sweet 👍

[alexcrichton]

🔔 This RFC is now entering its two-week long final comment period 🔔

[lilianmoraru]

Don't know if it is of any help but I will throw it out there.
QtCreator(mainly C++ IDE) has a plugin that uses Clang to offer C++ code completion.
Here is the code:
http://code.qt.io/cgit/qt-creator/qt-creator.git/tree/src/libs/clangbackendipc
http://code.qt.io/cgit/qt-creator/qt-creator.git/tree/src/plugins/clangcodemodel

QtCreator also has an older custom C++ code model(that is being replaced by the clang one) that was a lot faster and seems like it is accepted in the community that the code model that uses the compiler will be considerably slower, but it offers more information and is more accurate.
I mention the speed because I saw @phildawes mentioning the hard requirements on the compiler to deliver the information in <100ms and I am not sure if we should expect it to be fast.

[DemiMarie]

@lilianmoraru The reason the compiler needs to deliver the information so quickly is that the GUI is waiting on it. The user is expecting the completion to appear as soon as the user presses . and that requires the compiler to respond fast.

[michaelwoerister]

I'm in favor of accepting this RFC. The approach seems worth exploring and it does not preclude improving the compilers amenability for being used as a library. On the contrary, I think the compiler's APIs will benefit from trying to build the RLS on top of it and it can only help if people on the compiler team are actual clients of their own APIs. The RFC leaves many open questions when it comes to specifics and we just need a prototype implementation and the experience that comes from building that in order to decide how to proceed further. Worst case, we'll learn a bunch of stuff on what doesn't work :)

[erkinalp]

1-based line numbers and 0-based column numbers please.

[nrc]

@erkinalp why?

[bruno-medeiros]

@erkinalp why?

I was wondering the same, why is a mixed format being used (1-based in one, and 0-based for the other)

[erkinalp]

Emacs uses 0-based column numbers.

[ticki]

@erkinalp That's a perfect reason for not doing that 😜 .

[erkinalp]

@nrc @bruno-medeiros: And it uses 1-based line-numbers https://gcc.gnu.org/bugzilla/show_bug.cgi?id=19165#c21

[nrc]

afaik, there is no standard for editors to use 1-based line numbers. Having both 0-based seems like the least confusing thing to do, it's easy enough for editors to add one to each line number.

[dgrunwald]

Compilers (including rustc) tend to use 1-based line and column numbers in their error messages. Following that standard seems like the least confusing thing to do. I certainly wouldn't expect 0-based line numbers.

But the concepts of "line" and "column" are ill-defined anyways.
Does a vertical tab (\v) count as a new line? What about form feed? What about all the other exotic Unicode newlines?
Does a tab count as N columns (where N is configurable; usually 4 or 8?), does it advance to the next multiple of N columns, or does it count as only 1 character?
Are full-width characters like 'ｘ' one or two columns wide?
Or does the editor count each Unicode codepoint as 1 column? Maybe a 'column' really is a UTF-8 byte offset within the line? Maybe it's measured in UTF-16 code units? Grapheme clusters?

It's difficult to find two editors that agree in their line/column counting for all possible input files.

[Valloric]

I've integrated 5+ semantic engines in ycmd, and the only thing that makes sense is 1-based line and column numbers. Columns are byte offsets in UTF-8. Done.

it's easy enough for editors to add one to each line number.

But why should they? Line & column numbers coming from your oracle will be shown to the user and they expect 1-based numbering.

there is no standard for editors to use 1-based line numbers.

And yet they ~all do use 1-based numbers in the user interface. When you put your caret on the first line in the file, the editor doesn't say the line number is 0, it says it's 1. Same for columns.

[erkinalp]

Vertical tab means skip one line below and continue from same column offset.
CR, CR+LF, LF, LS and NEL are regular line feeds.
FF and PS count as two lines instead of one.

[bruno-medeiros]

I've integrated 5+ semantic engines in ycmd, and the only thing that makes sense is 1-based line and column numbers. Columns are byte offsets in UTF-8. Done.

Why byte offsets and not Unicode character offsets? It's not like an error or position for a Rust symbol will ever start in the middle of a Unicode character.

there is no standard for editors to use 1-based line numbers.

And yet they ~all do use 1-based numbers in the user interface. When you put your caret on the first line in the file, the editor doesn't say the line number is 0, it says it's 1. Same for columns.

Because the internal API for lines and columns can be 0-based, despite the UI being 1-based. This is certainly the case for Eclipse, for IntelliJ, and probably for most IDEs/editors out there. It would not surprise me if Vim is the odd one out... 😆

[adelarsq]

Keep it simple. Just use 0-based for lines and columns.

[alexcrichton]

This RFC was discussed during the tools team triage today and the decision was to merge. This RFC is still at a somewhat high level and some minor details can continue to be ironed out in the implementation over time, but there seems to be widespread agreement about the body of the RFC here.

Thanks again for the discussion everybody!