feat: Use Oniguruma-To-ES in the JS engine (#828) #832
Conversation
✅ Deploy Preview for shiki-matsu ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
✅ Deploy Preview for shiki-next ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #832 +/- ##
==========================================
+ Coverage 94.86% 94.91% +0.04%
==========================================
Files 78 77 -1
Lines 6608 6583 -25
Branches 1319 1314 -5
==========================================
- Hits 6269 6248 -21
+ Misses 330 326 -4
Partials 9 9 ☔ View full report in Codecov by Sentry. |
|
@antfu Would it be possible for you to modify or take over this PR to fix the tests?
|
| // Later we might consider to bundle them. | ||
| 'oniguruma-to-js', | ||
| 'regex', | ||
| 'oniguruma-to-es', |
There was a problem hiding this comment.
Not sure whether oniguruma-to-es's dependencies should also be externalized. I removed the comment about considering bundling because I'd much prefer not to do that if it means these projects will lose much of their download stats (which helps add to their credibility which is really valuable for now, especially while they're new projects).
There was a problem hiding this comment.
Totally! If oniguruma-to-es gets externalized, then its dependencies will be as well.
|
What do you suggest for the better name of |
|
Interestingly the regex behave different on macOS and Windows/Linux 😇 |
I would probably just remove the option (but keep its handling). I'd probably also remove the option to provide a different JS regex constructor, since that might be unrealistic for people to do. But I might not be thinking about it the same way as you. If you don't remove the option, maybe keep the same name at least for now (I can't think of a better one). I'm thinking of bringing back some Aside: vscode-textmate also does special/hacky things with numbered backreferences in
What?? 😖 In which JS environment? With Safari? I don't have access to macOS at the moment, but if you can identify specific regexes that are behaving differently, maybe I can identify JS engine bugs and work around them. |
I think it would still be valuable, people could copy our implementation of using
Node.js, see the CI. I am using macOS, and I updated the snapshot in a commit, which result the snapshot mismatch on Linux and Windows. If you update it, it would be another way around. I haven't dig into which regex is causing that yet tho |
That's all reasonable. I won't push on this.
IMO this is the strongest use case, but we can make Shiki's JS engine support this out of the box by accepting a
Whoa. That's really tricky if the regex engine in the same version of Node.js is behaving differently on macOS and Linux/Windows. But yeah, if you're able to identify particular regexes causing it, I'll do everything I can to try to work around it. |
|
BTW, I've got a breakthrough coming in the next version (in the works). The number of supported grammars will go way up. 😊 |
| * | ||
| * @default 'auto' | ||
| */ | ||
| target?: 'ES2024' | 'ES2025' | 'ES2018' | 'auto' |
There was a problem hiding this comment.
I implemented a auto target here, do you think it's worth to bringing it to oniguruma-to-es?
There was a problem hiding this comment.
Yes; great idea. And after I move target: 'auto' into oniguruma-to-es, it will also be able to use target 'ES2025' when supported, making results even better.
A recent package update broke the tool so we are adding a lock file to ensure we have a reliable execution. For details the newer version of https://www.npmjs.com/package/shiki?activeTab=versions version 1.23.0 introduced a new dependency in shikijs/shiki#832 which no longer compiles. ``` node_modules/oniguruma-to-es/types/parse.d.ts(61,49): error TS2339: Property 'orphan' does not exist on type '{}'. node_modules/oniguruma-to-es/types/parse.d.ts(96,31): error TS2339: Property 'atomic' does not exist on type '{}'. node_modules/oniguruma-to-es/types/parse.d.ts(96,39): error TS2339: Property 'flags' does not exist on type '{}'. ``` Until we figure out what the best way to fix those issues we are pinning to the older version using a lock file.
Description
See #828. This changes the JS engine to use Oniguruma-To-ES for much improved emulation (beyond what the numbers say) and sets a new foundation to continue improving grammar support in the JS engine.
Additional context
I don't really understand what's going on with the
packages/engine-javascript/testtests (which I didn't modify in this PR except for deletingutils.test.tssince it was testing a now-deleted file), so it's possible some stuff in there should also be deleted or modified after this lands.pnpm testpasses.The
simulationoption inpackages/engine-javascript/src/index.tsis probably not appropriately named anymore now that most of its behavior has been removed. I've updated the comment for it but didn't change the name (will leave for @antfu).