add doc and ci test for semantic tokens (#3938)

* add doc and ci test for semantic tokens

* fix, semantic tokens test files, case sensitivity issue

.github/workflows/test.yml

@@ -240,6 +240,9 @@ jobs:
         name: Test hls-overloaded-record-dot-plugin test suite
         run: cabal test hls-overloaded-record-dot-plugin --test-options="$TEST_OPTS" || LSP_TEST_LOG_COLOR=0 LSP_TEST_LOG_MESSAGES=true LSP_TEST_LOG_STDERR=true cabal test hls-overloaded-record-dot-plugin --test-options="$TEST_OPTS"
 
+      - if: matrix.test
+        name: Test hls-semantic-tokens-plugin test suite
+        run: cabal test hls-semantic-tokens-plugin --test-options="$TEST_OPTS" || cabal test hls-semantic-tokens-plugin --test-options="$TEST_OPTS"
 
 
   test_post_job:

docs/features.md

@@ -20,6 +20,7 @@ Many of these are standard LSP features, but a lot of special features are provi
 | [Code lenses](#code-lenses)                         | `textDocument/codeLens`                                                                           |
 | [Selection range](#selection-range)                 | `textDocument/selectionRange`                                                                     |
 | [Rename](#rename)                                   | `textDocument/rename`                                                                             |
+| [Semantic tokens](#semantic-tokens)                 | `textDocument/semanticTokens/full`                                                                |
 
 The individual sections below also identify which [HLS plugin](./what-is-hls.md#hls-plugins) is responsible for providing the given functionality, which is useful if you want to raise an issue report or contribute!
 Additionally, not all plugins are supported on all versions of GHC, see the [plugin support page](./support/plugin-support.md) for details.
@@ -380,7 +381,15 @@ Known limitations:
 
 - Cross-module renaming requires all components to be indexed, which sometimes causes [partial renames in multi-component projects](https://github.com/haskell/haskell-language-server/issues/2193).
 
-### Rewrite to overloaded record syntax
+## Semantic tokens
+
+Provided by: `hls-semantic-tokens-plugin`
+
+Provides semantic tokens for each token to support semantic highlighting.
+
+![Semantic Tokens Demo](https://private-user-images.githubusercontent.com/14073857/290981908-9619fae2-cb92-4d4e-b8f8-6507851ba9f3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDQ4MjgwODYsIm5iZiI6MTcwNDgyNzc4NiwicGF0aCI6Ii8xNDA3Mzg1Ny8yOTA5ODE5MDgtOTYxOWZhZTItY2I5Mi00ZDRlLWI4ZjgtNjUwNzg1MWJhOWYzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAxMDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMTA5VDE5MTYyNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBjOTUxNTM0ZDcyNmFmZjEyN2JlNzkwNWNjZjA4NTAzNDVkMzdlNmMxNDNiMzgxNGMzMTQ1NDRiMzUxZjM5OWQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.n-CL6e2R0TWHpmzVo1i60QEDczTEJ-8zvQWxjaBsnks)
+
+## Rewrite to overloaded record syntax
 
 Provided by: `hls-overloaded-record-dot-plugin`
 
@@ -389,7 +398,8 @@ Code action kind: `refactor.rewrite`
 Rewrites record selectors to use overloaded dot syntax
 
 ![Explicit Wildcard Demo](../plugins/hls-overloaded-record-dot-plugin/example.gif)
-## Missing features
+
+### Missing features
 
 The following features are supported by the LSP specification but not implemented in HLS.
 Contributions welcome!
@@ -399,7 +409,6 @@ Contributions welcome!
 | Signature help         | Unimplemented     | `textDocument/signatureHelp`                   |
 | Jump to declaration    | Unclear if useful | `textDocument/declaration`                     |
 | Jump to implementation | Unclear if useful | `textDocument/implementation`                  |
-| Semantic tokens        | Unimplemented     | `textDocument/semanticTokens`                  |
 | Linked editing         | Unimplemented     | `textDocument/linkedEditingRange`              |
 | Document links         | Unimplemented     | `textDocument/documentLink`                    |
 | Document color         | Unclear if useful | `textDocument/documentColor`                   |

docs/support/plugin-support.md

@@ -61,6 +61,7 @@ For example, a plugin to provide a formatter which has itself been abandoned has
 | `hls-rename-plugin`                 | 2    | 9.8                      |
 | `hls-stylish-haskell-plugin`        | 2    | 9.8                      |
 | `hls-overloaded-record-dot-plugin`  | 2    |                          |
+| `hls-semantic-tokens-plugin`        | 2    |                          |
 | `hls-floskell-plugin`               | 3    | 9.8                      |
 | `hls-stan-plugin`                   | 3    | 9.2.(4-8)                |
 | `hls-retrie-plugin`                 | 3    | 9.8                      |

plugins/hls-semantic-tokens-plugin/test/testdata/TDataFamily.hs

@@ -0,0 +1,11 @@
+{-# LANGUAGE TypeFamilies #-}
+module TDatafamily where
+
+-- Declare a list-like data family
+data family XList a
+
+-- Declare a list-like instance for Char
+data instance XList Char = XCons !Char !(XList Char) | XNil
+
+-- Declare a number-like instance for ()
+data instance XList () = XListUnit !Int

plugins/hls-semantic-tokens-plugin/test/testdata/TDataType.hs

@@ -0,0 +1,3 @@
+module TDataType where
+
+data Foo = Foo Int deriving (Eq)

plugins/hls-semantic-tokens-plugin/test/testdata/TPatternSyn.hs

@@ -0,0 +1,7 @@
+{-# LANGUAGE PatternSynonyms #-}
+module TPatternSyn where
+
+
+pattern Foo = 1
+
+