Async support for custom TokensProvider

[PranayAgarwal]

The setTokensProvider function currently has the signature

export function setTokensProvider(languageId: string, provider: TokensProvider | EncodedTokensProvider | Thenable<TokensProvider | EncodedTokensProvider>): IDisposable;

The Thenable<TokensProvider | EncodedTokensProvider> part doesn't really make sense (unless there is a use case for it I am missing). Instead of this, it would be very helpful to have a tokenize(line: string, state: IState): Thenable<ILineTokens> variant which can defer tokenizing to another process/network/something else (same as all other providers).

[alexdima]

The Thenable<TokensProvider | EncodedTokensProvider> part doesn't really make sense (unless there is a use case for it I am missing).

All the languages in monaco-languages register a promise in the similarly typed setMonarchTokensProvider. The promise is helpful because it allows the editor to know that there will be a tokenizer for a certain language and the editor can listen for that promise when it encounters embedded modes or when tokenizing hovers, or when the colorize API is used, etc. Basically, when tokenizing html, it will initially tokenize <script> blocks and <style> blocks in black & white and then install listeners for when the tokenizers for JS and CSS are loaded and at that point it will retokenize the blocks.

Instead of this, it would be very helpful to have a tokenize(line: string, state: IState): Thenable<ILineTokens> variant which can defer tokenizing to another process/network/something else (same as all other providers).

Yes, that would be good, but unfortunately there are many assumptions in the editor that tokenization can be done synchronously. I wrote a bit about this in the past at microsoft/vscode#77140