[READY] Java support using jdt.ls

.circleci/config.yml

@@ -31,6 +31,7 @@ aliases:
         - ~/.pyenv
         - clang_archives
         - third_party/racerd/target
+        - third_party/eclipse.jdt.ls/target/cache
   restore-cache: &restore-cache
     restore_cache:
       key: v3-ycmd-{{ .Environment.CIRCLE_JOB }}

.circleci/install_dependencies.sh

@@ -104,4 +104,15 @@ echo "export PATH=${CARGO_PATH}:\$PATH" >> $BASH_ENV
 
 npm install -g typescript
 
+#################
+# Java 8 setup
+#################
+
+java -version
+JAVA_VERSION=$(java -version 2>&1 | awk -F '"' '/version/ {print $2}')
+if [[ "$JAVA_VERSION" < "1.8" ]]; then
+  echo "Java version $JAVA_VERSION is too old" 1>&2
+  exit 1
+fi
+
 set +e

.gitignore

@@ -67,3 +67,6 @@ coverage.xml
 # API docs
 docs/node_modules
 docs/package-lock.json
+
+# jdt.ls
+third_party/eclipse.jdt.ls

.travis.yml

@@ -64,3 +64,5 @@ cache:
     - $HOME/.pyenv  # pyenv
     - $TRAVIS_BUILD_DIR/clang_archives  # Clang downloads
     - $TRAVIS_BUILD_DIR/third_party/racerd/target  # Racerd compilation
+      # jdt.ls download
+    - $TRAVIS_BUILD_DIR/third_party/eclipse.jdt.ls/target/cache

JAVA_SUPPORT.md

@@ -0,0 +1,227 @@
+This document briefly describes the work done to support Java (and other
+Language Server Protocol-based completion engines). 
+
+# Overview
+
+The [original PR][PR] implemented native support in ycmd for the Java language,
+based on [jdt.ls][]. In summary, the following key features were added:
+
+* Installation of jdt.ls (built from source with `build.py`)
+* Management of the jdt.ls server instance, projects etc.
+* A generic (ish) implementation of a [Language Server Protocol][lsp] client so
+  far as is required for jdt.ls (easily extensible to other engines)
+* Support for the following Java semantic engine features:
+  * Semantic code completion, including automatic imports
+  * As-you-type diagnostics
+  * GoTo including GoToReferences
+  * FixIt
+  * RefactorRename
+  * GetType
+  * GetDoc
+
+See the [trello board][trello] for a more complete picture.
+
+## Overall design/goals
+
+Key goals:
+
+1. Support Java in ycmd and YCM; make it good enough to replace eclim and
+   javacomplete2 for most people
+2. Make it possible/easy to support other [lsp][] servers in future (but, don't
+   suffer from yagni); prove that this works.
+
+An overview of the objects involved can be seen on [this
+card][design]. In short:
+
+* 2 classes implement the language server protocol in the
+  `language_server_completer.py` module:
+ * `LanguageServerConnection` - an abstraction of the comminication with the
+   server, which may be over stdio or any number of TCP/IP ports (or a domain
+   socket, etc.). Only a single implementation is included (stdio), but
+   [implementations for TCP/IP](https://github.com/puremourning/ycmd-1/commit/f3cd06245692b05031a64745054326273d52d12f)
+   were written originally and dropped in favour of stdio's simplicity.
+ * `LanguageServerCompleter` - an abstract base for any completer based on LSP,
+   which implements as much standard functionality as possible including
+   completions, diagnostics, goto, fixit, rename, etc.
+* The `java_completer` itself implements the `LanguageServerCompleter`, boots
+  the jdt.ls server, and instantiates a `LanguageServerConnection` for
+  communication with jdt.ls.
+
+The overall plan and some general discussion around the project can be found on
+the [trello board][trello] I used for development.
+
+## Threads, and why we need them
+
+LSP is by its nature an asyncronous protocol. There are request-reply like
+`requests` and unsolicited `notifications`. Receipt of the latter is mandatory,
+so we cannot rely on their being a `bottle` thread executing a client request.
+
+So we need a message pump and despatch thread. This is actually the
+`LanguageServerConnection`, which implements `thread`. It's main method simply
+listens on the socket/stream and despatches complete messages to the
+`LanguageServerCompleter`. It does this:
+
+* For `requests`: similarly to the TypeScript completer, using python `event`
+  objects, wrapped in our `Response` class
+* For `notifications`: via a synchronised `queue`. More on this later.
+
+A representation of this is on the "Requests and notifications" page
+of the [design][], including a rough sketch of the thread interaction.
+
+### Some handling is done in the message pump.
+
+While it is perhaps regrettable to do general processing directly in the message
+pump, there are certain notifications which we have to handle immediately when
+we get them, such as:
+
+* Initialisation messages
+* Diagnostics
+
+In these cases, we allow some code to be executed inline within the message pump
+thread, as there is no other thread guaranteed to execute. These are handled by
+callback functions and state is protected mutexes.
+
+## Startup sequence
+
+See the 'initialisation sequence' tab on the [design][] for a bit of background.
+
+In standard LSP, the initialisation sequence consists of an initialise
+request-reply, followed by us sending the server an initialised notification. We
+must not send any other requests until this has completed.
+
+An additional wrinkle is that jdt.ls, being based on eclipse has a whole other
+initialisation sequence during which time it is not fully functional, so we have
+to determine when that has completed too. This is done by jdt.ls-specific
+messages and controls the `ServerIsReady` response.
+
+In order for none of these shenanigans to block the user, we must do them all
+asynchronously, effectively in the message pump thread. In addition, we must
+queue up any file contents changes during this period to ensure the server is up
+to date when we start processing requests proper.
+
+This is unfortunately complicated, but there were early issues with really bad
+UI blocking that we just had to get rid of.
+
+## Completion
+
+Language server protocol requires that the client can apply textEdits,
+rather than just simple text. This is not an optional feature, but ycmd
+clients do not have this ability.
+
+The protocol, however, restricts that the edit must include the original
+requested completion position, so we can perform some simple text
+manipulation to apply the edit to the current line and determine the
+completion start column based on that.
+
+In particular, the jdt.ls server returns textEdits that replace the
+entered text for import completions, which is one of the most useful
+completions.
+
+We do this super inefficiently by attempting to normalise the TextEdits
+into insertion_texts with the same start_codepoint. This is necessary
+particularly due to the way that eclipse returns import completions for
+packages.
+
+We also include support for "additionalTextEdits" which
+allow automatic insertion of, e.g.,  import statements when selecting
+completion items. These are sent on the completion response as an
+additional completer data item called 'fixits'. The client applies the
+same logic as a standard FixIt once the selected completion item is
+inserted.
+
+## Diagnostics
+
+Diagnostics in LSP are delivered asynchronously via `notifications`. Normally,
+we would use the `OnFileReadyToParse` response to supply diagnostics, but due to
+the lag between refreshing files and receiving diagnostics, this leads to a
+horrible user experience where the diagnostics always lag one edit behind.
+
+To resolve this, we use the long-polling mechanism added here (`ReceiveMessages`
+request) to return diagnostics to the client asynchronously.
+
+We deliver asynchronous diagnostics to the client in the same way that the
+language server does, i.e. per-file. The client then fans them out or does
+whatever makes sense for the client. This is necessary because it isn't possible
+to know when we have received all diagnostics, and combining them into a single
+message was becoming clunky and error prone.
+
+In order to be relatively compatible with other clients, we also return
+diagnostics on the file-ready-to-parse event, even though they might be
+out of date wrt the code. The client is responsible for ignoring these
+diagnostics when it handles the asynchronously delivered ones. This requires
+that we hold the "latest" diagnostics for a file. As it turns out, this is also
+required for FixIts.
+
+## Projects
+
+jdt.ls is based on eclipse. It is in fact an eclipse plugin. So it requires an
+eclipse workspace. We try and hide this by creating an ad-hoc workspace for each
+ycmd instance. This prevents the possibility of multiple "eclipse"  instances
+using the same workspace, but can lead to unreasonable startup times for large
+projects.
+
+The jdt.ls team strongly suggest that we should re-use a workspace based on the
+hash of the "project directory" (essentially the dir containing the project
+file: `.project`, `pom.xml` or `build.gradle`). They also say, however, that
+eclipse frequently corrupts its workspace.
+
+So we have a hidden switch to re-use a workspace as the jdt.ls devs suggest. In
+testing at work, this was _mandatory_ due to a slow SAN, but at home, startup
+time is not an issue, even for large projects. I think we'll just have to see
+how things go to decide which one we want to keep.
+
+## Subcommands
+
+### GetDoc/GetType
+
+There is no GetType in LSP. There's only "hover". The hover response is
+hilariously server-specific, so in the base `LanguageServerCompleter` we just
+provide the ability to get the `hover` response and `JavaCompleter` extracts the
+appropriate info from there. Thanks to @bstaletic for this!
+
+### FixIt
+
+FixIts are implemented as code actions, and require the diagnostic they relate
+to to be send from us to the server, rather than just a position. We use the
+stored diags and find the nearest one based on the `request_data`.
+
+What's worse is that the LSP provides _no documentation_ for what the "Code
+action" response should be, and it is 100% implementation-specific. They just
+have this `command` abstraction which is basically "do a thing".
+
+From what I've seen, most servers just end up with either a `WorkspaceEdit` or a
+series of `TextEdits`, which is fine for us as that's what ycmd's protocol looks
+like.
+
+The solution is that we have a callback into the `JavaCompleter`  to handle the
+(custom) `java.apply.workspaceEdit` "command".
+
+### GoToReferences
+
+Annoyingly, jdt.ls sometimes returns references to .class files within jar
+archives using a custom `jdt://` protocol. We can't handle that, so we have to
+dodge and weave so that we don't crash.
+
+### Stopping the server
+
+Much like the initialisation sequence, the LSP shutdown sequence is a bit
+fiddly. 2 things are required:
+
+1. A `shutdown` request-reply. The server tides up and _prepares to die!_
+2. An `exit` notification. We tell the server to die.
+
+This isn't so bad, but jdt.ls is buggy and actually dies without responding to
+the `shutdown` request. So we have a bunch of code to handle that and to ensure
+that the server dies eventually, as it had a habbit of getting stuck running,
+particularly if we threw an exception.
+
+[PR]: https://github.com/valloric/ycmd/pull/857
+[jdt.ls]: https://github.com/eclipse/eclipse.jdt.ls
+[lsp]: https://github.com/Microsoft/language-server-protocol/
+[eclim]: http://eclim.org
+[javacomplete2]: https://github.com/artur-shaik/vim-javacomplete2
+[vscode-javac]: https://github.com/georgewfraser/vscode-javac
+[VSCode]: https://code.visualstudio.com
+[destign]: https://trello.com/c/78IkFBzp
+[trello]: https://trello.com/b/Y6z8xag8/ycm-java-language-server
+[client]: https://github.com/puremourning/YouCompleteMe/tree/language-server-java

README.md

@@ -37,9 +37,9 @@ Building
 **If you're looking to develop ycmd, see the [instructions for setting up a dev
 environment][dev-setup] and for [running the tests][test-setup].**
 
-This is all for Ubuntu Linux. Details on getting ycmd running on other OS's can be
-found in [YCM's instructions][ycm-install] (ignore the Vim-specific parts). Note
-that **ycmd runs on Python 2.6, 2.7 and 3.3+.**
+This is all for Ubuntu Linux. Details on getting ycmd running on other OS's can
+be found in [YCM's instructions][ycm-install] (ignore the Vim-specific parts).
+Note that **ycmd runs on Python 2.6, 2.7 and 3.3+.**
 
 First, install the minimal dependencies:
 ```
@@ -74,7 +74,8 @@ API notes
   header. The HMAC is computed from the shared secret passed to the server on
   startup and the request/response body. The digest algorithm is SHA-256. The
   server will also include the HMAC in its responses; you _must_ verify it
-  before using the response. See [`example_client.py`][example-client] to see how it's done.
+  before using the response. See [`example_client.py`][example-client] to see
+  how it's done.
 
 How ycmd works
 --------------
@@ -86,11 +87,12 @@ provided previously and any tags files produced by ctags. This engine is
 non-semantic.
 
 There are also several semantic engines in YCM. There's a libclang-based
-completer that provides semantic completion for C-family languages.  There's also a
-Jedi-based completer for semantic completion for Python, an OmniSharp-based
-completer for C#, a [Gocode][gocode]-based completer for Go (using [Godef][godef]
-for jumping to definitions), and a TSServer-based
-completer for TypeScript. More will be added with time.
+completer that provides semantic completion for C-family languages.  There's
+also a Jedi-based completer for semantic completion for Python, an
+OmniSharp-based completer for C#, a [Gocode][gocode]-based completer for Go
+(using [Godef][godef] for jumping to definitions), a TSServer-based completer
+for TypeScript and a [jdt.ls][jdtls]-based server for Java. More will be added
+with time.
 
 There are also other completion engines, like the filepath completer (part of
 the identifier completer).
@@ -338,3 +340,4 @@ This software is licensed under the [GPL v3 license][gpl].
 [vscode-you-complete-me]: https://marketplace.visualstudio.com/items?itemName=RichardHe.you-complete-me
 [gycm]: https://github.com/jakeanq/gycm
 [nano-ycmd]: https://github.com/orsonteodoro/nano-ycmd
+[jdtls]: https://github.com/eclipse/eclipse.jdt.ls

appveyor.yml

@@ -39,3 +39,5 @@ cache:
   - '%USERPROFILE%\.cargo'  # Cargo package deps
   - '%APPVEYOR_BUILD_FOLDER%\clang_archives'  # Clang downloads
   - '%APPVEYOR_BUILD_FOLDER%\third_party\racerd\target'  # Racerd compilation
+     # jdt.ls download
+  - '%APPVEYOR_BUILD_FOLDER%\third_party\eclipse.jdt.ls\target\cache'