Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: improve snapshot testing for js engine #866

Merged
merged 9 commits into from
Dec 17, 2024
Merged

test: improve snapshot testing for js engine #866

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
6b20f86
draft: investigating source of mismatches in js engine
slevithan Dec 16, 2024
59791fc
Merge branch 'main' into pr/slevithan/866
antfu Dec 17, 2024
9e00aa3
chore: update
antfu Dec 17, 2024
6b835d5
chore: update
antfu Dec 17, 2024
d027d75
chore: update
antfu Dec 17, 2024
1254a9c
chore: update data format
antfu Dec 17, 2024
27a7b0b
chore: update
antfu Dec 17, 2024
0be0cde
chore: update
antfu Dec 17, 2024
f1505c4
chore: update deps
antfu Dec 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 19 additions & 16 deletions docs/references/engine-js-compat.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,19 @@

Compatibility reference of all built-in grammars with the [JavaScript RegExp engine](/guide/regex-engines#javascript-regexp-engine-experimental).

> Generated on Wednesday, November 27, 2024
> Generated on Monday, December 16, 2024
>
> Version `1.23.1`
> Version `1.24.2`
>
> Runtime: Node.js v22.11.0

## Report Summary

| | Count |
| :-------------- | --------------------------: |
| Total Languages | 215 |
| Supported | [197](#supported-languages) |
| Mismatched | [12](#mismatched-languages) |
| Total Languages | 218 |
| Supported | [199](#supported-languages) |
| Mismatched | [13](#mismatched-languages) |
| Unsupported | [6](#unsupported-languages) |

## Supported Languages
Expand Down Expand Up @@ -43,9 +43,10 @@ In some edge cases, it's not guaranteed that the highlighting will be 100% the s
| bibtex | ✅ OK | 19 | - | |
| bicep | ✅ OK | 27 | - | |
| blade | ✅ OK | 1126 | - | |
| bsl | ✅ OK | 96 | - | |
| c | ✅ OK | 177 | - | |
| cadence | ✅ OK | 71 | - | |
| cairo | ✅ OK | 80 | - | |
| cairo | ✅ OK | 236 | - | |
| clarity | ✅ OK | 43 | - | |
| clj | ✅ OK | 38 | - | |
| clojure | ✅ OK | 38 | - | |
Expand All @@ -55,14 +56,14 @@ In some edge cases, it's not guaranteed that the highlighting will be 100% the s
| codeql | ✅ OK | 151 | - | |
| coffee | ✅ OK | 469 | - | |
| common-lisp | ✅ OK | 60 | - | |
| coq | ✅ OK | 25 | - | |
| coq | ✅ OK | 26 | - | |
| crystal | ✅ OK | 1067 | - | |
| css | ✅ OK | 141 | - | |
| csv | ✅ OK | 1 | - | |
| cue | ✅ OK | 85 | - | |
| cypher | ✅ OK | 39 | - | |
| d | ✅ OK | 270 | - | |
| dart | ✅ OK | 72 | - | |
| dart | ✅ OK | 76 | - | |
| dax | ✅ OK | 23 | - | |
| desktop | ✅ OK | 16 | - | |
| diff | ✅ OK | 16 | - | |
Expand All @@ -82,8 +83,8 @@ In some edge cases, it's not guaranteed that the highlighting will be 100% the s
| fortran-free-form | ✅ OK | 328 | - | |
| fsharp | ✅ OK | 239 | - | |
| fsl | ✅ OK | 30 | - | |
| gdresource | ✅ OK | 157 | - | |
| gdscript | ✅ OK | 93 | - | |
| gdresource | ✅ OK | 159 | - | |
| gdscript | ✅ OK | 95 | - | |
| gdshader | ✅ OK | 39 | - | |
| genie | ✅ OK | 20 | - | |
| gherkin | ✅ OK | 19 | - | |
Expand All @@ -92,7 +93,7 @@ In some edge cases, it's not guaranteed that the highlighting will be 100% the s
| glimmer-ts | ✅ OK | 676 | - | |
| glsl | ✅ OK | 186 | - | |
| gnuplot | ✅ OK | 82 | - | |
| go | ✅ OK | 123 | - | |
| go | ✅ OK | 125 | - | |
| graphql | ✅ OK | 448 | - | |
| groovy | ✅ OK | 133 | - | |
| haml | ✅ OK | 562 | - | |
Expand Down Expand Up @@ -122,17 +123,16 @@ In some edge cases, it's not guaranteed that the highlighting will be 100% the s
| latex | ✅ OK | 283 | - | |
| lean | ✅ OK | 32 | - | |
| less | ✅ OK | 280 | - | |
| liquid | ✅ OK | 684 | - | |
| liquid | ✅ OK | 689 | - | |
| log | ✅ OK | 30 | - | |
| logo | ✅ OK | 9 | - | |
| lua | ✅ OK | 278 | - | |
| luau | ✅ OK | 88 | - | |
| make | ✅ OK | 51 | - | |
| marko | ✅ OK | 926 | - | |
| matlab | ✅ OK | 88 | - | |
| mdc | ✅ OK | 783 | - | |
| mdx | ✅ OK | 197 | - | |
| mermaid | ✅ OK | 129 | - | |
| mermaid | ✅ OK | 134 | - | |
| mipsasm | ✅ OK | 17 | - | |
| mojo | ✅ OK | 213 | - | |
| move | ✅ OK | 120 | - | |
Expand All @@ -159,7 +159,7 @@ In some edge cases, it's not guaranteed that the highlighting will be 100% the s
| qmldir | ✅ OK | 7 | - | |
| qss | ✅ OK | 31 | - | |
| r | ✅ OK | 71 | - | |
| racket | ✅ OK | 69 | - | |
| racket | ✅ OK | 69 | - | 8 |
| raku | ✅ OK | 52 | - | |
| reg | ✅ OK | 9 | - | |
| regexp | ✅ OK | 34 | - | |
Expand All @@ -172,6 +172,7 @@ In some edge cases, it's not guaranteed that the highlighting will be 100% the s
| scala | ✅ OK | 117 | - | |
| scheme | ✅ OK | 34 | - | |
| scss | ✅ OK | 234 | - | |
| sdbl | ✅ OK | 23 | - | |
| shaderlab | ✅ OK | 87 | - | |
| shellscript | ✅ OK | 148 | - | |
| shellsession | ✅ OK | 150 | - | |
Expand All @@ -187,9 +188,10 @@ In some edge cases, it's not guaranteed that the highlighting will be 100% the s
| svelte | ✅ OK | 637 | - | |
| system-verilog | ✅ OK | 102 | - | |
| systemd | ✅ OK | 32 | - | |
| talonscript | ✅ OK | 44 | - | |
| tasl | ✅ OK | 23 | - | |
| tcl | ✅ OK | 34 | - | |
| templ | ✅ OK | 673 | - | |
| templ | ✅ OK | 675 | - | |
| terraform | ✅ OK | 68 | - | |
| tex | ✅ OK | 106 | - | |
| toml | ✅ OK | 44 | - | |
Expand Down Expand Up @@ -242,6 +244,7 @@ Languages that do not throw with the JavaScript RegExp engine, but will produce
| kotlin | [🚧 Mismatch](https://textmate-grammars-themes.netlify.app/?grammar=kotlin) | 58 | - | 1953 |
| kusto | [🚧 Mismatch](https://textmate-grammars-themes.netlify.app/?grammar=kusto) | 60 | - | 40 |
| markdown | [🚧 Mismatch](https://textmate-grammars-themes.netlify.app/?grammar=markdown) | 118 | - | 915 |
| mdc | [🚧 Mismatch](https://textmate-grammars-themes.netlify.app/?grammar=mdc) | 783 | - | 702 |
| nginx | [🚧 Mismatch](https://textmate-grammars-themes.netlify.app/?grammar=nginx) | 378 | - | 4 |
| php | [🚧 Mismatch](https://textmate-grammars-themes.netlify.app/?grammar=php) | 1131 | - | 605 |
| po | [🚧 Mismatch](https://textmate-grammars-themes.netlify.app/?grammar=po) | 23 | - | 423 |
Expand Down
1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
"mdast-util-gfm": "catalog:",
"mdast-util-to-hast": "catalog:",
"ofetch": "catalog:",
"ohash": "catalog:",
"picocolors": "catalog:",
"pnpm": "catalog:",
"prettier": "catalog:",
Expand Down
10 changes: 7 additions & 3 deletions packages/engine-javascript/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,11 @@ export function defaultJavaScriptRegexConstructor(pattern: string, options?: Oni
hasIndices: true,
rules: {
allowOrphanBackrefs: true,
// Dropping unhandled `\G` anchors without erroring allows more grammars to pass but also
// allows some false positive and negative matches. It's mostly because this is true that
// the JS engine is marked as experimental
allowUnhandledGAnchors: true,
// Improves search performance for generated regexes
asciiWordBoundaries: true,
},
...options,
Expand Down Expand Up @@ -106,7 +110,7 @@ export class JavaScriptScanner implements PatternScanner {
})
}

findNextMatchSync(string: string | RegexEngineString, startPosition: number): IOnigMatch | null {
findNextMatchSync(string: string | RegexEngineString, startPosition: number, _options: number): IOnigMatch | null {
const str = typeof string === 'string'
? string
: string.content
Expand All @@ -118,15 +122,15 @@ export class JavaScriptScanner implements PatternScanner {
captureIndices: match.indices!.map((indice) => {
if (indice == null) {
return {
end: MAX,
start: MAX,
end: MAX,
length: 0,
}
}
return {
start: indice[0] + offset,
length: indice[1] - indice[0],
end: indice[1] + offset,
length: indice[1] - indice[0],
}
}),
}
Expand Down
Loading
Loading