Extensible custom completion, validation, hover, ... support

[angelozerr]

I create this issue because there are more and more language server (written in Java according my knowledge) which requires YAML support like:

• camel language server: @apupier could give you more information.
• Spring Boot language server: @martinlippert could you give you more information.
• Quarkus LS, I'm working on this language server, so I will get you more information about our need.

Today Quarkus uses application.properties file to customize some properties. Properties are dynamic and are collecting according the maven dependency (ex: if you have a Quarkus hibernate in your pom.xml, you will need just configure hibernate properties). Our Quarkus LS manages completion, hover, validation for thoses properties and when classpath changed (or Java sources changed), those properties are updated to refresh completion list with new properties. Here a demo:

Quarkus could configure their properties with an application.yaml file in the future, that's why I create this issue to think how we could implement the same features with yaml.

After discussing about this topic with @fbricon , we have 2 choices to do that:

• implement our own yaml scanner in Java, in other words translate your yaml scanner/document written in TypeScript to Java (I already did that with lsp4xml where I translate the HTML scanner from vscode-html-languageservice to Java). @apupier told me that they have implemented their own parser in camel language server but there are some limitation, so I think it should be better to translate your scanner to Java.
• communicate with your YAML language server to add custom completion, validation, hover with an extension point kind which communicate with custom LSP services (to support any language like Java).

I would like to know if the second point could be in your scope or not and if it could become a priority otherwise I think translating your YAML scanner in Java could be a good idea too. @martinlippert, @apupier what do you think about that?

[JPinkney]

I wonder if the registerContributor API would work for you here.

Basically this API allows clients to specify a function that runs to check if the current file should be associated with a schema. If it should be associated with a schema then you return the schema contents to the yaml language server and then it's used for completion, validation, etc.

Heres where its used in tests: https://github.com/redhat-developer/vscode-yaml/blob/master/test/schemaProvider.test.ts#L17

But the gist of it is you register your contribution like:

client.registerContributor(SCHEMA, onRequestSchemaURI, onRequestSchemaContent);

and you provide a function that is called when an autocompletion/validation/hover request is made:

function onRequestSchemaURI(resource: string): string | undefined {
	if (resource === ('application.yaml')) {
		return `${SCHEMA}://schema/application`;
	}
	return undefined;
}

Then if the current file name matches application.yaml then another function is run that tries to get the associated schema:

function onRequestSchemaContent(schemaUri: string): string | undefined {
	const parsedUri = Uri.parse(schemaUri);
	if (parsedUri.scheme !== SCHEMA) {
		return undefined;
	}
	if (!parsedUri.path || !parsedUri.path.startsWith('/')) {
		return undefined;
	}

	return schemaJSON;
}

where schemaJSON (the returned value if there is a match) is:

const schemaJSON = JSON.stringify({
	type: "object",
	properties: {
		version: {
			type: "string",
			description: "A stringy string string",
			enum: [
				"test"
			]
		}
	}
});

Then the yaml-language-server uses this schema for autocompletion, hover, validation, etc.

The https://github.com/Azure/vscode-kubernetes-tools extension for vscode is using this for their yaml support.

[angelozerr]

Thanks so much @JPinkney for your help. I need to investigate more if we can use your suggestion in our Quarkus LS context. A thing which is very important is to reset the cache of the grammar, because when classpath changed, our Quarkus properties must change too.

[angelozerr]

@JPinkney once onRequestSchemaContent is called for a given uri, is the schema is cached on yaml language server side, and after that onRequestSchemaContent will not be called the second time?

If it is cached is there a way to reset this cache?

[JPinkney]

@angelozerr I wasn't the one who originally wrote it @andxu was. @andxu I believe there isn't any caching right? It should just pick up whatever schema the client sends back everytime?

If by chance there is caching, I could probably add a way to update that cache so that when something changes in quarkus (for example) you can send a request to the yaml language server to update the cache

[martinlippert]

@angelozerr Hey Angelo, thanks for pointing me at this. I think we should invite @kdvolder to this, too.

I am not exactly sure if I understand this correctly here. The support that we have for our Spring config files that are YAML based is implemented as part of the Spring Boot Language Server. As far as I can see, the existing YAML language server here is implemented in typescript. Are you thinking about re-implementing that in Java?

In addition to that the usual question comes up next: do you think about others being able to re-use this as a library to implement their own (independent) language server or do you think about one YAML language server running and somehow being enhanced by other (language servers)?

Sorry for my confusion here... ;-)

[angelozerr]

Are you thinking about re-implementing that in Java?

I would like to avoid doing that because it's a so an hard task (developing a tolerant scanner, parser is an hard thing that I have done for XML Language Server with LSP4XML). My idea is to gives the capability to extend the YAML Language Server for any features like completion, hover, etc like I have done that in LSP4XML by using Java SPI. See the idea at https://github.com/redhat-developer/vscode-xml#custom-xml-extensions

I think more and more that we must dedicate features to a given language server instead of trying to develop her own parser (we do that because we are in different technology) than the provided language server:

• YAML language server exists to parse very well YAML, manage error, etc, why developing an another language server (we do that because we are in different technology). I find it doesn't follow the idea: sharing features.
• to avoid double, triple, etc parsing of the language. Ex: in your case with Spring Boot, you parse yaml content to manage your case and the vscode-yaml extension parse the yaml content too.

Extension means:

• extend existing LSP features like completion, hover, etc for the language. With LSP4XML you can manage extension by developping a Java SPI IXMLExtension.
• provide new LSP services for the language server like JDT LS does with the delegate command handler mechanism. I would like to manage that in LSP4XML too. The idea is to extend the LSP4XML language server to add your own services (ex : you need to manage a search engine with XPath for searching XML files from your workspace and call a vscode workspace command to execute it).

By providing new LSP services for the language service will give the capability to consume the service (written in Java technolgy in case of LSP4XML) by an another technology like typescript by using language client. It's the idea that I try to promote in microsoft/vscode-languageserver-node#526 with JDT LS case, but my idea is to do that with any language server: using language client (written in any technology) to consume a X language server (like XML, YAML, Java, written in any technology) for standard LSP services or custom LSP services.

In addition to that the usual question comes up next: do you think about others being able to re-use this as a library to implement their own (independent) language server

No I would like to avoid doing that. My idea is have a commons LSP language client which consumes the X Language server and other LSP language server uses their own LSP language client to communicate the the commons LSP language client by using LSP standard services or custom LSP services:

• in vscode:
  • vscode-xml for XML Language Server.
  • vscode-yaml for YAML Language Server.
• in Eclipse IDE: wilwebdeveloper which embed LSP4XML and YAML Language Server.

After that a X Language Server could consume the XML Language Server,YAML Languagse server by using her own language client.

Takes a sample with Spring Beans search engine. You develop a LSP4XML Java SPI IXMExtension which uses LSP4XML features like XML scanner and register it as LSP custom services spring/search/beans (not possible for the moment, but I will work on this issue).

The Spring Language Server consume the LSP spring/search/beans by delegating the search to the vscode/LSP4E Spring language client which will call the vscode-xml / LSP4E language client which will delegate this request to the XML Language Server.

or do you think about one YAML language server running and somehow being enhanced by other (language servers)?

Yes it's my idea, see below my explanation.

[martinlippert]

I am still not sure I get this. Let me try to explain the piece that I don't fully understand yet... :-)

Lets assume you have a YAML language server that is written in Typescript. It parses YAML and provides basic content-assist, symbols, etc. In addition to that I have a Spring Boot language server that is written in Java. It would love to extend the existing YAML language server for specific files.

Both language servers run as separate processes and are implemented in different languages. I cam imagine that the YAML language server and the Spring Boot language server "talk" to each other via commands (or something like that), basically exchanging some sort of data in JSON. I can also imagine that you could make this work for enhancing the YAML language server, so that this language server offers, lets say, something like an extension API in the form of a JSON protocol, so that those two language servers can really "talk" to each other.

The piece that I find hard to imagine is: the Spring Boot language server needs quite detailed information about the YAML file and its logical structure to provide the specific content-assist. Therefore the language server usually parses the YAML file internally and calculates the content-assist based on that YAML structure (+ additional, e.g. Spring-specific project information). If you want to avoid the parsing the Spring Boot language server does and leave that to the general YAML language server, the Spring Boot language server would need to receive that specific data about the YAML structure and content from the YAML language server. Is that what you imagine?

It would (more or less) mean that you would define data structures for those languages (YAML, in this example) and that you would need to implement that data structure in every language server that uses the API (the data structure becomes part of the API) would need to implement that data structure, map it to internal representations, work with it, etc. While that is certainly possible, I am not sure if that outweighs the double parsing in every case.

Another example is the Java language server that contains a parser for Java source files, of course. In order to avoid double parsing in the Spring Boot language server, it would mean to serialize, deserialize, and re-implement the massive JDT AST apis, since that is what the Spring Boot language server would need to do its job. Doing that directly and specifically in the Spring Boot language server is much more efficient that communicating Java ASTs forth and back.

What do you think? Does it explain what I don't understand yet?

[slrtbtfs]

I face a similar problem with the PromQL language server.

What I currently do:

• If a YAML file is opened, both the PromQL and the YAML language server get started and connect to the language client.
• The PromQL language server parses the YAML and provides completion/hover/etc. wherever it finds a PromQL expression, otherwise it stays out of the way.
• This means there is some split responsibility:
  • The yaml server is responsible for the structure of the yaml and errors concerning that structure.
  • The PromQL language server provides additional assistance, whenever there happens to be a PromQL query at the position where the language client asks for completion/metadata.

Even though the two language servers don't communicate at all, this works surprisingly well.

EDIT: lots of typos

[angelozerr]

I am still not sure I get this. Let me try to explain the piece that I don't fully understand yet... :-)

yes sure :)

Lets assume you have a YAML language server that is written in Typescript. It parses YAML and provides basic content-assist, symbols, etc. In addition to that I have a Spring Boot language server that is written in Java. It would love to extend the existing YAML language server for specific files.

Exactly!

Both language servers run as separate processes and are implemented in different languages. I cam imagine that the YAML language server and the Spring Boot language server "talk" to each other via commands (or something like that), basically exchanging some sort of data in JSON. I can also imagine that you could make this work for enhancing the YAML language server, so that this language server offers, lets say, something like an extension API in the form of a JSON protocol, so that those two language servers can really "talk" to each other.

Yes it's was my idea, but instead of using Command, I would prefer using LSP services (like standard LSP services liek completion, hover, etc)

The piece that I find hard to imagine is: the Spring Boot language server needs quite detailed information about the YAML file and its logical structure to provide the specific content-assist. Therefore the language server usually parses the YAML file internally and calculates the content-assist based on that YAML structure (+ additional, e.g. Spring-specific project information). If you want to avoid the parsing the Spring Boot language server does and leave that to the general YAML language server, the Spring Boot language server would need to receive that specific data about the YAML structure and content from the YAML language server. Is that what you imagine?

Yes totally. My idea is that YAML language server provides a completionParticipant extension that you can implement in TypeScript which gives the capability to implement methods like onName, onValue to manage custom completion on name (not necessary from a YAML Schema) or on value.

It's exactly that we have for XML with LSP4XML (it provides completion, hover, etc participant that you can implement in Java and you register them with SPI).

For Spring Tools case, the completionParticipant implementation for completion value, must collect the property name, build an array with parent names and uses the languageClient to call a custom request like 'sts/beans', the languageClient (from YAML language server) must delegate his request to the Spring Tools language client (it's the reason why I create Handle communication between 2 language servers -> language client registry issue).

It would (more or less) mean that you would define data structures for those languages (YAML, in this example) and that you would need to implement that data structure in every language server that uses the API (the data structure becomes part of the API) would need to implement that data structure, map it to internal representations, work with it, etc. While that is certainly possible, I am not sure if that outweighs the double parsing in every case.

The data structures for me it's just an array with name and ancestor.

Another example is the Java language server that contains a parser for Java source files, of course. In order to avoid double parsing in the Spring Boot language server, it would mean to serialize, deserialize, and re-implement the massive JDT AST apis, since that is what the Spring Boot language server would need to do its job. Doing that directly and specifically in the Spring Boot language server is much more efficient that communicating Java ASTs forth and back.

Sending AST is not a thing that I have in my mind. If you need an AST, do that in the JDT (with delegate command handler or custom LSP services) and the service/command handler will receive simple parameters and returns JSON stream (like LSP completion, hover, etc).

[martinlippert]

The piece that I find hard to imagine is: the Spring Boot language server needs quite detailed information about the YAML file and its logical structure to provide the specific content-assist. Therefore the language server usually parses the YAML file internally and calculates the content-assist based on that YAML structure (+ additional, e.g. Spring-specific project information). If you want to avoid the parsing the Spring Boot language server does and leave that to the general YAML language server, the Spring Boot language server would need to receive that specific data about the YAML structure and content from the YAML language server. Is that what you imagine?

Yes totally. My idea is that YAML language server provides a completionParticipant extension that you can implement in TypeScript which gives the capability to implement methods like onName, onValue to manage custom completion on name (not necessary from a YAML Schema) or on value.

It's exactly that we have for XML with LSP4XML (it provides completion, hover, etc participant that you can implement in Java and you register them with SPI).

For Spring Tools case, the completionParticipant implementation for completion value, must collect the property name, build an array with parent names and uses the languageClient to call a custom request like 'sts/beans', the languageClient (from YAML language server) must delegate his request to the Spring Tools language client (it's the reason why I create Handle communication between 2 language servers -> language client registry issue).

Yeah, this helps, I think I understand your proposal in more depth now. Thanks for the additional details. Sounds quite good in general, I think.

The only "detail" that I am not getting yet is how my completionParticipant (that I probably implement in the language of the language server that I would like to extend) gets contributed to the language server it tries to extend. Somehow I would need to contribute my completionParticipant (then written in TypeScript) to the existing YAML language server (for example). Do you have an idea about that?

I know that JDT.LS offers something like this by loading additional bundles (contributed by other extensions) into the same OSGi runtime than the JDT language server is running inside. Do you envision something like that as a general mechanism for every language server?

I have a very concrete example for that, since we are looking at extending the XML support with some Spring-specific XSD namespace lookup mechanism. Therefore I also filed eclipse/lemminx#596.

Would be great to hear your thoughts on that.

[angelozerr]

Would be great to hear your thoughts on that.

Ok @martinlippert let's stop to pollute this issue with lsp4xml and continue our conversation in eclipse/lemminx#596.

@JPinkney I think YAML language server should do like LSP4XML and provides a participant per services (completionParticipant, hoverParticipant, renameParticipant, etc).

It will fix I think for instance #206 and when I will implement communication between LSP4XML and Spring Tools LS for completion for instance we could do the same thing with YAML.

[JPinkney]

@angelozerr If I'm not mistaken everything in this issue is done now? Can we close?

[angelozerr]

@angelozerr If I'm not mistaken everything in this issue is done now? Can we close?

For Quarkus LS to manage YAML completion in application.yaml we use your suggestion, it works good but there are some limitations.

My initial idea was to do like LemMinx the XML Language Server, I mean provide the capability to register participant c(completion participant, hover participant, etc to manage completion like you wish (without generating a JSON Schema file).

An example of limitation is it will impossible to support #212 with JSON Schema.

But if you wish you can close it.

[JPinkney]

I think we can do it the way Gorkem suggested, just extend the JSON Schema Spec to include a new field that stores a link. Then when you ctrl+click on name then go to the link in that field

[safareli]

I think we can do it the way Gorkem suggested, just extend the JSON Schema Spec to include a new field that stores a link. Then when you ctrl+click on name then go to the link in that field

Is this supported?