Semantic tokens features trackList

[soulomoon]

Semantic highlighting would give us a great experience on reading the code. It's been a shame that haskell does not have it for all these times. But now we are going to make it happen for haskell in HLS. (Thank @michaelpj for his pull request that support it in lsp package), first version of semantic tokens have been made in this pull request. (Thank @michaelpj @wz1000 for all the help and reviews. )
But since the amount of works it need , not everything in the LSP semantic tokens specification is supported yet, we would do them step by step.
Here is the track list of features.

• provide configuration to map between the semantic tokens type and lsp standard tokens. add config for semantic-tokens-plugin for mapping from hs token type to LSP default token type #3940
• support module name and split out module name in qualified name #3957
• provide configurability to the priorities of hs semantic token types Semantic tokens make tokens types priority configurable #4042
• Support semantic tokens type: done in Implement semantic tokens plugin to support semantic highlighting(textDocument/semanticTokens/full) #3892
  • class and class method type family name (data family)
  • data constructor name (not distinguishing record and normal data, and GADT)
  • type constructor name (GADT)
  • record field name
  • type synonym
  • pattern synonym
  • functions
  • none-function variables
  • infix operator Semantic Tokens: Distinguish (possibly-partially-applied) infix operators from normal functions/variables #3969
• Support PluginMethodHandler
  • textDocument/semanticTokens/full. done in Implement semantic tokens plugin to support semantic highlighting(textDocument/semanticTokens/full) #3892
  • textDocument/semanticTokens/full/delta #4072
  • workspace/semanticTokens/refresh
• Support modifiers:
  • declaration (as in class declearations, type definition and type family)
  • definition (as in class instance declaration, left hand side value binding, and type family instance)
  • modification (as in rec field update)
  • recursion analyze in let binding Mark recursive references in let bindings #2902
• Support unicode, done in Implement semantic tokens plugin to support semantic highlighting(textDocument/semanticTokens/full) #3892
• Support imported name, done in Implement semantic tokens plugin to support semantic highlighting(textDocument/semanticTokens/full) #3892
• ClientCapabilities, multilineTokenSupport, overlappingTokenSupport
• highlight identifiers in doc, but not the first line of nested comment due to a ghc bug. https://gitlab.haskell.org/ghc/ghc/-/issues/24378
• Bug fixing & minor improvement:
  • Semantic tokens forall function hide under type syn is not recognized #3966
  • Semantic Tokens treats a standalone deriving strategy as type parameter #3983

[Bodigrim]

Thanks for working on this feature! I gave it a quick try in SublimeText on https://github.com/Bodigrim/cabal-add/blob/a83c827e45eeaed722d63f08a4c1bfd183770e97/src/Distribution/Client/Add.hs#L189:

Is it something off with my configuration? Why String is in blue, but ByteString is white?

[michaelpj]

The current implementation highlights type synonyms differently to datatypes. So String and FilePath are different because they are type synonyms. Maybe this is confusing! (The fact that you asked suggests that it is...)

It's a bit funny that you get the same colour for parseCabalFile and MonadError also. Generally it seems a bit hit-and-miss whether any given colour scheme actually distinguishes everything that's in the spec...

[Bodigrim]

Maybe this is confusing! (The fact that you asked suggests that it is...)

Yeah, that's very confusing, especially given how ubiquitous String and FilePath are. I think it would be better not to distinguish.

[soulomoon]

Thank you for testing it out. @Bodigrim
We may need a bit of settings for some color schemes to get the colors right. The mapping from hs semantic token type to the standard semantic token type in lsp is here.

https://github.com/soulomoon/haskell-language-server/blob/master/plugins/hls-semantic-tokens-plugin/src/Ide/Plugin/SemanticTokens/Mappings.hs

say if we want the color scheme to not distinguish between a normal datatype with a type synonyms.
we can always set the Type( type synonym) and Enum(a normal datatype) to have the same color.

=====================================================
To be precise, we are not actually doing the semantic highlighting, we are giving semantic types to the tokens that enables the semantic highlighting feature for haskell in the editors. Maybe it worth a note some where to help user to configure this if the color scheme does not look good. Expecially with the fact that there is a layer of mappings between hs semantic to lsp semantic. And the color scheme is various.

=====================================================

It's a bit funny that you get the same colour for parseCabalFile and MonadError also. Generally it seems a bit hit-and-miss whether any given colour scheme actually distinguishes everything that's in the spec...

Also I was thinking🤔， maybe we can also make the toLspTokenType configurable to provide more flexibility on this. In this way, we can provide as detailed as much for the semantic token type, but also enable the user to configure their flavor if the default mapping does not suit their need.

[soulomoon]

Maybe this is confusing! (The fact that you asked suggests that it is...)

Yeah, that's very confusing, especially given how ubiquitous String and FilePath are. I think it would be better not to distinguish

I do not have a SublimeText subscription to test things out, but most color scheme in vscode do not distinguish them by default.
Here is the the default mappings look like in vscode with the scheme one dark pro.

But type synonym might be constraint too. This is one of the reason why we don't classify type synonym to have the same token type as normal data type.

[michaelpj]

(Possibly we should split this off into another issue)

Worth pointing out that type synonyms are highlighted as "type" and in this case the difference is because datatypes get a special highlighting.

maybe we can also make the toLspTokenType configurable to provide more flexibility on this

Generally we try pretty hard to avoid configuration. It's usually much better if we can find something that's universally acceptable.

I guess I could maybe be convinced to have settings for each thing where we have a choice of a specific or generic token type, e.g. "special highlighting for type synonyms: y/n", and if it's off then we use the generic one (e.g. "just a type"). We could let people pick exactly the LSP semantic token type for each thing but that's probably overkill.

We do already have a configuration switch here: it's whether or not you have semantic tokens on or off. If you turn semantic tokens on, you should probably expect it to make some fine distinctions that aren't obvious to you at first sight. So a different question is "how do we make this discoverable?". Perhaps in this instance having the options would serve a double function: it tells you what kinds of things we distinguish, and then you can fiddle with it to see what it does.

Having written all that I guess I've convinced myself that we should have options. Especially since a lot of the things semantic tokens lets you do are quite opinionated and specific, which some people will inevitably dislike (e.g. I bet people will complain about the different highlighting of things of function type).

[michaelpj]

I split off another issue for discussing the configuration issue: #3937

[michaelpj]

@soulomoon a couple of minor things that we still need to do:

• Add you to CODEOWNERS for the plugin (I've invited you to the repository)
• Update the "Features" section of the documentation: https://github.com/haskell/haskell-language-server/blob/master/docs/features.md
  • New section for semantic tokens
  • New entry in the top table linking to it
  • Remove it from "unimplemented features"
• Update the "plugin support" section of the docs: https://github.com/haskell/haskell-language-server/blob/master/docs/features.md
  • I think it can be Tier 2
• Add it to the test job: https://github.com/haskell/haskell-language-server/blob/master/.github/workflows/test.yml

[soulomoon]

@michaelpj thanks for sort it out. do it in #3938

[michaelpj]

A possible concern: there is a multilineTokenSupport client capability. We probably need to respect that as well as overlappingTokenSupport. I think the current implementation might return multi-line tokens if something goes over several lines.

I think we could maybe fix this universally in lsp: the functions that compute semantic tokens could also take the ClientCapabilities, and then do something like:

• If multilineTokenSupport is false, then split any multi-line tokens into one token per line
• if overlappingTokenSupport is false, then split any overlapped tokens into [<part before overlap>, <part after overlap>]

[michaelpj]

I actaully don't know if we'll emit multi-line tokens today already, do you know @soulomoon ? I guess we probably don't, since all the stuff we recognize today can't be split over multiple lines anyway.

[soulomoon]

I actaully don't know if we'll emit multi-line tokens today already, do you know @soulomoon ? I guess we probably don't, since all the stuff we recognize today can't be split over multiple lines anyway.

no we do not support them yet.

[michaelpj]

I made an issue for doing something helpful in lsp: haskell/lsp#545

[MangoIV]

pretty cool! do you think it would be somehow possible to highlight parameters to a function? I could not figure out how that would work, i.e. that in

f = \x -> x y z

x gets a highlight group and that reoccurs in the body where x is in scope.

[MangoIV]

I've really enjoyed the possibility of distinguishing class and type/ enum.

something that I'm wondering is why you chose to implement class and not interface, when I'm using the same colour schemes across languages, I think semantically it would make more sense if haskell classes would not be coloured like java classes but rather like java interfaces or abstract classes or whatever.

[michaelpj]

pretty cool! do you think it would be somehow possible to highlight parameters to a function?

There is a paramter token type. I'm unsure if that's supposed to be used for the declaration of a parameter or also its uses, since it means we can't mark it as a plan "variable". But I think it would be reasonable to use it for such things. We also have a bit of confusion because we use the function token type for variables of function type, so then we start to have difficulties: what token type do we give for a variable that is both a parameter and a function?

something that I'm wondering is why you chose to implement class and not interface

I think either is plausible, I think @soulomoon just picked one. I guess on reflection interface is maybe a bit more accurate.

[soulomoon]

Since we do have class and not to match class with class is rather confusing.

We won't have a perfect match on Haskell token vs default token. So we actually make it configurable. So you can pick the matches that suit your color scheme. @MangoIV

[MangoIV]

surely, yes, but I wonder what would happen if someone would program in both haskell and an object oriented language :D (I know even then you can configure it but perhaps we should optimize for the stock experience :P)

[soulomoon]

Yes, since haskell do have class term, it is a trade-off between prioritizing haskell term vs prioritizing object oriented language term. As a language server for haskell, I think the choice would be obvious? Although for the ones without default token, we are compromising by projecting them.

For the paramter, I suppose it might a good idea to have it. But, yes it is not easy to classify them in hieAst, so it is not implemented at the first place.

[rumkeller]

@michaelpj

There is a paramter token type. I'm unsure if that's supposed to be used for the declaration of a parameter or also its uses

This seems to suggest that it's meant for both.

what token type do we give for a variable that is both a parameter and a function?

Here is an example of how the text editor deals with multiple types for the same token. So perhaps it's o.k. to emit both variable and parameter types for the same token?

[soulomoon]

So perhaps it's o.k. to emit both variable and parameter types for the same token

@rumkeller AFAIK, VSCode would consider them as overlapping token which they do not support.

[rumkeller]

You're right -- it seems there can indeed be only one token type (though several modifiers). Apologies for the confusion.

So one has to choose between variable, function, and parameter. FWIW, here is what clangd does:

• Parameters are always marked as such, even if they are function pointers.
• It adds functionScope and globalScope modifiers. functionScope is for both parameters and local variables.

[michaelpj]

It adds functionScope and globalScope modifiers. functionScope is for both parameters and local variables.

These are non-specified modifiers, right? Do any clients check for them? (Generally it's a bit sad that we don't have an easy way to agree on extensions to the list of modifiers...)

[rumkeller]

These are non-specified modifiers, right?

Indeed they're not mentioned in the spec.

Do any clients check for them?

Neovim exposes the modifiers verbatim to the color scheme -- quite easy to adjust, though the default scheme appears to not be aware of them.