Handle communication between 2 language servers -> language client registry

[angelozerr]

When a language server requires some Java features (ex: track change of classpath for Spring Tools LS, collect properties computed by scanning some Java annotations from a project for Quarkus LS), it uses JDT LS the Java Language server:

• it registers custom command on JDT LS server by using Eclipse extension point
• the external language server call this custom command.

This issue is about calling this custom command. Here a simple schema which show you how Quarkus LS communicates with JDT LS:

The basic idea is that Quarkus LS request the Quarkus vscode language client and this client call a vscode java workspace command (coming from the vscode-java language client) which call the JDT LS. See https://github.com/redhat-developer/vscode-quarkus/blob/master/src/extension.ts#L49

In other words the 2 language client instances are used to delegate communication between the 2 language servers.

This mechanism is working but it requires some extra glue and it is not very clean. I would like to improve this mechanism to use custom LSP services instead of executing workspace command.

To do that I need that the 2 language client instances can be available in a language client registry to override languageClient#onRequest (from vscode-quarkus) and delegate the request to the other languageClient#sendRequest (vscode-java).

I create this issue here, because it is related to the vscode language client. In other words it should be very fantastic if vscode-languageclient could provide this languageclient registry kind to register a languageclient instance with an id and get a language client instance from an id.

Many thank's for your help. @martinlippert I think you could be interested with this idea.

[martinlippert]

@angelozerr thanks for adding me to this, I am indeed very interested in a better solution for this scenario. The Spring Boot language server that we work on has to exchange information with the JDT LS and we do this in the same way, but it would be indeed fantastic if there would be a better and easier mechanism to do that.

Maybe there could even be an option/way in the language server protocol itself to send commands/requests to other language servers and let the client implementation of the LSP take care of that automatically. That would make the whole mechanism more explicit within the protocol.

[angelozerr]

@martinlippert many thanks for your answer.

Maybe there could even be an option/way in the language server protocol itself to send commands/requests to other language servers and let the client implementation of the LSP take care of that automatically.

Indeed it should be the ideal solution, but is it possible? @dbaeumer what do you think about this idea?

Perhaps in the first step we could manage communication between 2 servers with language client instances (like today) and implement language client registry?

[dbaeumer]

Maybe there could even be an option/way in the language server protocol itself to send commands/requests to other language servers and let the client implementation of the LSP take care of that automatically. That would make the whole mechanism more explicit within the protocol.

I am not a fan of this since it assumes that language features are implemented using language servers. This is IMO not necessarily the case.

So IMO this always have to go through the extension code that manages the language server, either using commands or better defined API.

To better understand what you needs can you provide a concrete example?

[angelozerr]

To better understand what you needs can you provide a concrete example?

Let me try to explain the Quarkus LS. In Quarkus we have an application.properties file where you can configure some properties:

quarkus.datasource.url=vertx-reactive:postgresql://localhost:5432/quarkus_test
quarkus.datasource.username=quarkus_test
quarkus.datasource.password=quarkus_test
myapp.schema.create=true

Each property are declared in a Java class field. For instance myapp.schema.create comes from this Java class field annotated with @ConfigProperty annotation:

public class FruitResource {

    @ConfigProperty(name = "myapp.schema.create", defaultValue = "true")
    boolean schemaCreate;

}

The Quarkus LS provide a support to manage completion, validation, hover, definition for those properties in application.properties. See the following demo:

In technical point of view, we have a Quarkus LS written in Java and a vscode-quarkus extension which consumes this language server. This vscode-quarkus extension embed the Java JDT LS Eclipse plugin com.redhat.quarkus.jdt.core.jar which provides a delegate command handler (registered with quarkus.java.projectInfo id) which scans all JAR and Java sources files from a given project to collect all Quarkus properties (that you can see in completion for instance). See https://github.com/redhat-developer/vscode-quarkus/blob/master/package.json#L44

"contributes": {
    "javaExtensions": [
      "./jars/com.redhat.quarkus.jdt.core.jar"
    ]
}

This com.redhat.quarkus.jdt.core.jar JAR is loaded by JDT LS Java language server to register a new workspace command in the server.

Here a schema which shows how the 2 language (JDT Java LS <-> Quarkus LS) server communicates:

When completion is triggered in application.properties, the vscode-quarkus language client call the Quarkus LS LSP completion service. This service call the quarkus/projectInfo request of the vscode-quarkus language client and executes the proper java workspace command quarkus.java.projectInfo:

languageClient.onRequest('quarkus/projectInfo', async (params: QuarkusProjectInfoParams) =>
       <any> await commands.executeCommand("java.execute.workspaceCommand", 'quarkus.java.projectInfo', params)
    );

See https://github.com/redhat-developer/vscode-quarkus/blob/master/src/extension.ts#L49

The java.execute.workspaceCommandis a vscode (client) command coming from vscode-java which call the JDT LS language server with the proper command . In Quarkus LS context, it call the command registered with id quarkus.java.projectInfo, scan all JAR, Java sources files and returns an array of Quarkus properties.

As you can see, the quarkus language client communicates with the java language client with workspace command.

I would like to avoid doing that and manage communication with LSP standard services. In other word, the com.redhat.quarkus.jdt.core.jar Eclipse plugin should register dynamicly a custom standard LSP service (named with quarkus./projectInfo request) in the JDT Java Language Server and when Quarkus LS call the quarkus./projectInfo request, the vscode-quarkus language client which receives this request delegate this request to the vscode-java language client which call the JDT LS Java Language server.

In other words, the quarkus./projectInfo request must be bound between the 2 language client vscode-quarkus and vscode-java.

We are experimenting this idea and it starts working well. I will give you more information once we will clean more our code. But it's not generic (the bindRequest method is linked to vscode-java), but I think it should be very good if vscode-languageclient could provide this bindRequest feature.

[martinlippert]

The examples in the Spring Boot language servers are:

• the spring boot language server needs project and classpath information for the java projects in the workspace
• the spring boot language server needs detailed information about certain java types

Since the Spring Boot language server is not strictly speaking a "language" server, but more an framework-specific extension to existing languages (Java, XML, property files, in our case), we try to not duplicate the base language tooling in our language server, but rely on other language servers (or language support extensions) to do the work for us.

At the very beginning, we had our own Java project understanding included in our language server, which resulted in, for example, that both language servers (the Java one and ours) were doing the same work twice (identifying the projects, using Maven or Gradle to resolve dependencies, etc). This involves the inherent danger to have subtle differences in the way we handle those projects, compared to the Java language server. Therefore we decided to depend on the existing Java tooling to do this work once (and to be the only source of truth) and send us the information (via commands, as described above).

The downside of this is that our language server directly depends on the Java . We even have to contribute an extension to the Java language server.

(I think it would be much better to handle all this at a higher level, so that our language server would not need to know anything about the Java language server, but that sounds like a totally different game. However, would be interesting to investigate for sure.)

[angelozerr]

@martinlippert thank a lot for your feedback.

We have created a little POC with jdt.ls, vscode-java and quarkus ls, vscode-quarkus whith which simplifies the communication between the 2 language servers. On client side (it's the side which is interested by this issue, we bind request between 2 language client (from vscode-quarkus to vscode-java, see poc at https://github.com/xorye/vscode-java/tree/registry).

You can see more informations and link to the POC at https://docs.google.com/document/d/1HHC39OVcenfF7bxhhaQO8rgMgyX_YVBzx-arUKpDnCE

[kdvolder]

I want to point out that the 'trick' of using commands as a communication tool to send data between language servers is actually a bit of hack and technically isn't even guaranteed to work within the LSP protocol as it is currently written. It works in vscode but maybe this more of a 'accident' than actually by design. I recall some heated discussions with Atom folks about their interpretation of the spec in this regard and how their interpretation essentially keeps command registered to a language server specifically limited to be called by that same language server. This makes it so its impossible to use these commands to send information between two different servers.

See (amongst others):

atom/atom-languageclient#183 (comment)

I also raised a ticket in lsp protocol about this and we (@dbaeumer and me) had a long exchange of ideas already but I don't think its really resolved. See:

microsoft/language-server-protocol#432

Ultimately that issue was really about the same underlying need: Sometimes you will want a language server to be able to exchange information with another instead of re-implementing all the functionality that is needed to compute that information on its own.

[angelozerr]

I want to point out that the 'trick' of using commands as a communication tool to send data between language servers is actually a bit of hack

I agree with you, have you read our proposition https://docs.google.com/document/d/1HHC39OVcenfF7bxhhaQO8rgMgyX_YVBzx-arUKpDnCE ?

The basic idea is that you can register with an Eclipse plugin a custom LSP service (by using LSP4J annotation @JSonRequest) to the JDT LS Java language server and the external language client (ex: vscode-quarkus) delegates the request to vscode-java. The request is a standard LSP request. It's just a delegation. Do you like the idea? @kdvolder please give us your feedback, thanks!

[kdvolder]

@angelozerr I did take a peek at it but I don't understand much of what it is proposing. So take mu opinion with a grain of salt. It looks to me like a rather complex solution that is very much specific to lsp4j/lsp4e and requires custom handling in every client implementation and server. So not sure this is really better than using the 'exploit command as inter-server messages'. At least the 'hack' requires no new protocol and only requires reading slightly more into the spec than it really says.

The idea of a mechanism for defining 'services' sounds nice though... if it could be fitted into lsp protocol as a proper and standardized extension of the protocol.

[kdvolder]

Perhaps what we need is a kind of 'message bus'. Some way for a server to declare (as a server capability) that it accepts messages of a given custom type. And also some protocol that allows (other) servers to send these messages with the understanding that the server who declares it handles the message type will receive them when another server sends them. Effectively that is the kind of thing that we are simulating with commands today.

[angelozerr]

@angelozerr I did take a peek at it but I don't understand much of what it is proposing.

That's shame, we should improve our documentation,but it is linked to JDT LS. The main idea of the proposition is:

• extends JDT Java language server with standard LSP services (by using Eclipse plugin), it's out of the scope of this issue.
• binding request between 2 language clients. It's this feature that I would like to promote in this issue.

Forget the extension feature and takes an another example with standard LSP request like textDocument/documentSymbols. Imagine you have a language server X which manages *.x file. The MyClass.x is bound to MyClass.java file and contains fields declared in the class.

For instance:

• MyClass.java looks like this:

public class MyClass {

   private String name;
   private int age;
}

• MyClass.x looks like this:

name=xxxx
age=10

Now you want to manage completion inside MyClass.x file for name, age fields coming from MyClass.java. You don't want develop a JDT LS extension with delegate command handler or with standard LSP4J annotations (like explained in our proposition)

The textDocument/completion from the language server X, could call the standard LSP textDocument/documentSymbol from the Java JDT language server and extract fields kind from the document symbols and manage their own completion by using those collected fields. And imagine the language server X is written in TypeScript.

connection.onCompletion((textDocumentPosition, token) => {
	return runSafe(() => {
		const document = documents.get(textDocumentPosition.textDocument.uri);
		if (!document) {
			return null;
		}
                // replace .x with .java extension
                const javaUri = textDocumentPosition.textDocument.uri.replace('.x', '.java');
                // request the X language client with   java/textDocument/documentSymbols
                const params = {'textDocument': {'uri': textDocument.uri.r
                const symbols = connection.sendRequest('java/textDocument/documentSymbols', params);
               // Loop for Java symbols and return the proper comletion based on those Java symbol fields kind.
               return ...

The call of

const symbols = connection.sendRequest('java/textDocument/documentSymbols', params);`

send a request to the language client X. This language client X must delegate the request to the JDT LS language server like this:

xLanguageClient.onRequest('java/textDocument/documentSymbols', async (params: any) =>
       // delegate the request by sending the standard LSP request 'textDocument/documentSymbols' to the java language client (vscode-java language client)
       return javaLanguageClient.sendRequest('textDocument/documentSymbols', params);
    );

It's the idea of the binding request between 2 language clients instances. As vscode-java doesn't expose their language client instance javaLanguageClient, a language client registry is a good candidate to write only

registry.bindRequest('x', 'java/textDocument/documentSymbols', 'java', 'textDocument/documentSymbols');

where :

• 'x' is the id of the from language client
• 'java/textDocument/documentSymbols' is request id of the from language client
• 'java' is the id of the to language client
• 'textDocument/documentSymbols' is request id of the to language client

In other words, with request binding features you can bind any request (standard or custom (with extension capability)) between 2 language clients instances. The communication between 2 language servers uses their language client to communicate each other by using request binding feature provided by the language client registry.

So take mu opinion with a grain of salt. It looks to me like a rather complex solution that is very much specific to lsp4j/lsp4e and requires custom handling in every client implementation and server.

What is complex?

I try to show in the document that the language server delegates a custom LSP request to the client.

• If the client have the capability to do it (ex: in Eclipse IDE with LSP4E, the language client have the capability to manage Java parsing and scan of annotations, it can do it, so it manage it).

• if the client have not the capability, it delegates the request to the proper language server bu using their own language client instance (in vscode context, the vscode-quarkus language client written in TypeScript cannot parse Java file, so it delegates this feature to the vscode-java). In this case it call a custom LSP services (an extension) which is registered with Eclipse plugin extension point mechanism (out of the scope of this issue).

So not sure this is really better than using the 'exploit command as inter-server messages'.

It uses standard LSP service and you can do that with other technologie than JDT LS (ex: yaml language server could expose some custom LSP services with an another extension mechanism).
At least the 'hack' requires no new protocol and only requires reading slightly more into the spec than it really says.

The idea of a mechanism for defining 'services' sounds nice though... if it could be fitted into lsp protocol as a proper and standardized extension of the protocol.

It's an another topic, provide a specification to extend a language server with custom LSP services. This issue is about language client side to manage bindings between request.

Perhaps what we need is a kind of 'message bus'. Some way for a server to declare (as a server capability) that it accepts messages of a given custom type. And also some protocol that allows (other) servers to send these messages with the understanding that the server who declares it handles the message type will receive them when another server sends them. Effectively that is the kind of thing that we are simulating with commands today.

The binding request feature should take care of that and you can manage communication between 2 language servers which are written in different technologies.

[kdvolder]

@angelozerr Okay I think I'm starting to understand a bit better what direction you are going with this. I think I could probably get behind something like this. But I also think it is quite a long way to go from that linked google doc to something that can be folded into LSP protocol with a 'model' implementation in vscode client.

[tsmaeder]

In my opinion, this whole affair could quite easily be solved by promoting "registerCommand" down do LSP. One LS could register a command, the other could invoke it. Of course, there would have to be some rules on how to handle collisions, etc.

[angelozerr]

Command is indeed a generic mean, but as you can send any data, it's hard to know what you receive, send.

You could manage the whole LSP protocol (completion, hover, etc) with Command. But code would be ugly (no Pojo params).

We have a completion with clean Pojo params, result, why we could not have this same support for custom services? For me Command is like Java reflection API. Using Java reflection is generic but provides awful code.

[tsmaeder]

@angelozerr because you're asking for a fundamental change of architecture (LS being recognized as a concept in VS Code) and a lot of machinery to be implemented to achieve compile-time type safety. If type-safety were the only consideration, I might agree with the approach, but it is not.
Don't forget that every client (there are quite a lot: https://microsoft.github.io/language-server-protocol/implementors/tools/) will have to implement your proposal for language servers to work with those clients. I believe that implementing command registration has a much greater probability of adoption any time soon.
I am generally a fan of type-safety, but in this instance, I think it's quite simple for language servers to document the actions they implement and what the parameters should be. They could even offer descriptions (json schema, *.d.ts) that allows to check and convert parameters automatically and offer type-safety to languages where that applies.
But you don't have to convince me, you have to convince the good folks from MS, like @dbaeumer. Until we get some resolution from that side, I don't think we should invest in the approach.

[kdvolder]

In my opinion, this whole affair could quite easily be solved by promoting "registerCommand" down do LSP. One LS could register a command, the other could invoke it. Of course, there would have to be some rules on how to handle collisions, etc.

I tend to agree. Effectively it is how the vscode client works today. So perhaps we just need the spec to be a bit more specific about the fact that there is a shared command registry.

I've tried to have this kind of 'clarification' added to the spec (i.e. see the issue I also linked above already microsoft/language-server-protocol#432). I understand there is some reluctance to add that kind specificity into the spec.

[tsmaeder]

It even seems we have @dbaeumer on board: microsoft/language-server-protocol#432 (comment) Any idea why that issue has not been addressed (apart from "other stuff to do")?

[kdvolder]

Any idea why that issue has not been addressed

Just guessing here, but I think that its not really so clear what the right solution is.

• The use of commands works today but is more of a hack not guaranteed by the spec. And doesn't feel like a nice solution. (I think that's why there is reluctance to enshrine it in the spec)
• A better solution seems like something complex to design and build. For example Dirk's suggestion to make it part of the client. may not work unless all servers actually use the same client / version.

TBH. I kind of agree with you @tsmaeder that updating the spec to guarantee that the 'commands hack' works for all conforming LSP clients... but only because its the least complex thing to realize. Not because it feels like a 'properly designed' solution. And because a 'properly designed soultion' seems so complex right now that it may not be realistic to expect an expedient resolution via that route.

[gurusharan2]

Hi
Is there any update on this ?
or is there any hack other than using commands to do so ?