Explore deleting the Oniguruma dependency

[fabiospampinato]

I've just stumbled across this project, which aims to convert Ruby regexes to JS regexes. /cc @jaynetics, in case you are interested in this.

Not everything is supported at the moment, but a lot is, so I think there might be a chance that the missing features are not used in existing TM grammars, or if they are used in a few spots maybe the affected grammars could be tweaked slightly to not use them.

If TM grammars could be evaluated without Oniguruma that would allow VS Code to drop a fairly large dependency, and possibly also speed syntax highlighting up significantly, as regexes would be executed natively rather than with a userland regex engine.

Potentially if deleting our dependency on Oniguruma entirely is impractical it may still be interesting to explore taking an hybrid approach, converting what can be converted faithfully at runtime and executing those natively, and deferring execution to Oniguruma only for the trickiest regexes (if any).

[jaynetics]

@fabiospampinato thanks for the mention!

As an avid vscode user, i'd be happy to help!

If I understand correctly, you're suggesting to add transpilation-like capabilities to VSCode that are applied to TM grammars at runtime, as opposed to providing pre-transpiled grammars in some way?

If so, this transpilation would need to be coded in TypeScript - is that right?

That would be a big project then. js_regex was a big project, and was only possible to its current extent because there was a pre-existing parser for Onigmo regexps in Ruby (b.t.w., Ruby and thus js_regex are based on Onigmo, a fork of Onigiruma, and the syntaxes of Onigmo and Oniguruma have diverged a bit).

[fabiospampinato]

If I understand correctly, you're suggesting to add transpilation-like capabilities to VSCode that are applied to TM grammars at runtime, as opposed to providing pre-transpiled grammars in some way?

I think there would be use cases for both, potentially, but transpiling at runtime would be preferable.

If so, this transpilation would need to be coded in TypeScript - is that right?

TS or something that compiles to JS or WASM I guess.

That would be a big project then. js_regex was a big project, and was only possible to its current extent because there was a pre-existing parser for Onigmo regexps in Ruby (b.t.w., Ruby and thus js_regex are based on Onigmo, a fork of Onigiruma, and the syntaxes of Onigmo and Oniguruma have diverged a bit).

If it would allow for deleting Oniguruma it'd still be worth it imo 🤔 There seem to be some projects for compiling Ruby to JS, maybe those are good enough and could spare us from porting js_regex.

[alexdima]

For syntax coloring, we will continue investigating Tree Sitter (see #161479). But for the foreseeable future we will support TM grammars and these grammars use the oniguruma regex syntax. To not break anyone, we will therefore continue shipping the oniguruma regex library.

[fabiospampinato]

Could it still make sense to explore converting some popular TM grammars to use native JS regexes? Like I'm thinking if converting the JS/TS/Markdown/CSS/HTML/etc. grammars at build time to not use Oniguruma could meaningfully improve the experience for a large portion of users it might still be worth it?

Either way I'm interested in exploring this myself, eventually, I'll report interesting findings back here if it could be useful 👍

[alexdima]

I'm open to ideas, please let me know if you come up with something. In the meantime, I'm going to close this issue since we're cleaning up issues this milestone.

But please consider that we have tried to be good OSS citizens and so far avoided adding VS Code specific concepts to TM grammars. So all TM grammars in use in VS Code can easily run in other editors. For example, we really needed some concepts in grammars, but we've taken care to add them from the outside despite things being not so nice e.g..