Move tokenization to a web worker

[hediet]

Also see #77140

Plan of attack:

• Adopt vscode-textmate refactorings in vscode #167288
• Make sure rule ids are stable by initializing injection rules eagerly
• Tokenize in the webworker
• Send over states by diffing against the state of the previous line. Send the delta as pop(n), push(state) instructions.

Sync tokenization:

Code_-_OSS_RQiEvFdPla.mp4

Async tokenization:

Code_-_OSS_owfdHXRTpY.mp4

[IllusionMH]

At first I was thinking it's about .ts extension that is treated as Video Transport Stream (TS) 😆

[RedCMD]

lines inside the viewport are retokenized everytime the viewport changes (scrolling etc)
this causes many lag spikes when scrolling over massively long lines that don't tokenize fully

this is different to when it was synchronous, as it would try to tokenize the line once and not try again until the user types

[hediet]

this is different to when it was synchronous, as it would try to tokenize the line once and not try again until the user types

I don't think this is true, it wouldn't tokenize lines only once when it used heuristically computed states to tokenize them.

[RedCMD]

me just scrolling normally at the same speed in both videos
notice the lag spikes with async on and the flickering of colours at the bottom section of the document

test file is the builtin c extension c.tmLanguage.json file
it is a single line of 72138 characters long
vscode manages to tokenize roughly 40,000 chars before giving up
roughly 45,500 chars when async is enabled

Async.Off-1.mp4

Async.On-1.mp4

I don't think this is true, it wouldn't tokenize lines only once when it used heuristically computed states to tokenize them.

I don't understand
is when multiple lines are present in the viewport?