Unsupported regex in md-math-block.tmLanguage.json

[jensli]

This regex in

tm4e/org.eclipse.tm4e.language_pack/markdown-math/md-math-block.tmLanguage.json

Line 16 in ac34c5f

"begin": "(?<=^\\s*)(\\${2})(?![^$]*\\${2})",

...is not supported by Joni.

The result if that when the line gets selected in Preferences > TextMate > Grammars the following exception is logged:

SEVERE: org.eclipse.tm4e.core.TMException: Parsing regex pattern "(?<=^\s*)(\${2})(?![^$]*\${2})" failed with org.joni.exception.SyntaxException: invalid pattern in look-behind

Stack trace:

	at org.eclipse.tm4e.core.internal.oniguruma.OnigRegExp.<init>(OnigRegExp.java:60)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
	at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
	at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
	at org.eclipse.tm4e.core.internal.oniguruma.OnigSearcher.<init>(OnigSearcher.java:33)
	at org.eclipse.tm4e.core.internal.oniguruma.OnigScanner.<init>(OnigScanner.java:32)
	at org.eclipse.tm4e.core.internal.rule.CompiledRule.<init>(CompiledRule.java:37)
	at org.eclipse.tm4e.core.internal.rule.RegExpSourceList.compile(RegExpSourceList.java:77)
	at org.eclipse.tm4e.core.internal.rule.RegExpSourceList.compileAG(RegExpSourceList.java:84)
	at org.eclipse.tm4e.core.internal.rule.IncludeOnlyRule.compileAG(IncludeOnlyRule.java:57)
	at org.eclipse.tm4e.core.internal.grammar.LineTokenizer.matchRule(LineTokenizer.java:314)
	at org.eclipse.tm4e.core.internal.grammar.LineTokenizer.matchRuleOrInjections(LineTokenizer.java:328)
	at org.eclipse.tm4e.core.internal.grammar.LineTokenizer.scanNext(LineTokenizer.java:137)
	at org.eclipse.tm4e.core.internal.grammar.LineTokenizer.scan(LineTokenizer.java:128)
	at org.eclipse.tm4e.core.internal.grammar.LineTokenizer.tokenizeString(LineTokenizer.java:564)
	at org.eclipse.tm4e.core.internal.grammar.Grammar._tokenize(Grammar.java:341)
	at org.eclipse.tm4e.core.internal.grammar.Grammar.tokenizeLine(Grammar.java:258)
	at org.eclipse.tm4e.core.model.TMTokenization.tokenize(TMTokenization.java:72)
	at org.eclipse.tm4e.core.model.TMModel$TokenizerThread.updateTokensOfLine(TMModel.java:169)
	at org.eclipse.tm4e.core.model.TMModel$TokenizerThread.lambda$0(TMModel.java:125)
	at org.eclipse.tm4e.core.model.TMModel.buildAndEmitEvent(TMModel.java:269)
	at org.eclipse.tm4e.core.model.TMModel$TokenizerThread.revalidateTokens(TMModel.java:121)
	at org.eclipse.tm4e.core.model.TMModel$TokenizerThread.run(TMModel.java:101)
Caused by: org.joni.exception.SyntaxException: invalid pattern in look-behind
	at org.joni.ScannerSupport.newSyntaxException(ScannerSupport.java:163)
	at org.joni.Analyser.setupLookBehind(Analyser.java:1404)
	at org.joni.Analyser.setupTree(Analyser.java:2004)
	at org.joni.Analyser.setupTree(Analyser.java:1825)
	at org.joni.Analyser.compile(Analyser.java:110)
	at org.joni.Regex.<init>(Regex.java:155)
	at org.joni.Regex.<init>(Regex.java:134)
	at org.eclipse.tm4e.core.internal.oniguruma.OnigRegExp.<init>(OnigRegExp.java:58)
	... 28 more

[jensli]

The solution is probably to remove the "Markdown Math" grammars in plugin.xml:

tm4e/org.eclipse.tm4e.language_pack/plugin.xml

Lines 405 to 411 in ac34c5f

	<!-- markdown-math -->
	<extension point="org.eclipse.tm4e.registry.grammars">
	<grammar scopeName="lngpck.text.html.markdown.math" path="markdown-math/md-math.tmLanguage.json"/>
	<scopeNameContentTypeBinding scopeName="lngpck.text.html.markdown.math" contentTypeId="lng.markdown-math"/>
	<grammar scopeName="lngpck.markdown.math.block" path="markdown-math/md-math-block.tmLanguage.json"/>
	<grammar scopeName="lngpck.markdown.math.inline" path="markdown-math/md-math-inline.tmLanguage.json"/>
	</extension>

[jensli]

Also, TMModel should probably log the whole exception, not just the message:

tm4e/org.eclipse.tm4e.core/src/main/java/org/eclipse/tm4e/core/model/TMModel.java

Lines 170 to 173 in ac34c5f

	} catch (final Exception ex) {
	LOGGER.log(ERROR, ex.toString());
	return UpdateTokensOfLineResult.UPDATE_FAILED;
	}

[jensli]

Also, I think tm4e should handle unsupported regexes and other problems with the grammar files more gracefully. Instead of just logging the error an error dialog should be displayed that explains the problem.

[mickaelistria]

all the proposals would be welcome as pull requests ;)

[sebthom]

@jensli Maybe this should be reported in the upstream repo instead: https://github.com/microsoft/vscode/blob/main/extensions/markdown-math/syntaxes/md-math-block.tmLanguage.json#L16

If joni does not support this, then this is probably not a valid Oniguruma style regex, which is required by textmate syntax. https://macromates.com/manual/en/regular_expressions#syntax_oniguruma

[jensli]

@sebthom Good idea. I will check.

Also: I plan to provide a PR disabling the invalid grammar.

Also: I'll have a look and try to improve error handling in general. Since all loading is lazy it is hard to find a good place to report problems. Also, the problem with this grammar is re-encountered and reported many times a second in the look by the update thread. I'll try to do something about that.

[angelozerr]

If joni supports it it could be interesting to see if vscode supports. As tm4e is a port of vscode-textmate perhaps we should upgrade it again.

[sebthom]

@angelozerr we are using the latest joni release and are in sync with latest vscode-textmate changes.

[jensli]

Oniguruma also rejects the regex, with exactly then same error message:

(?<=^\\s*)(\\${2})(?![^$]*\\${2})
=> jdoodle.rb:1: invalid pattern in look-behind: /(?<=^\\s*)(\\${2})(?![^$]*\\${2})/

Tested at jdoodle.com using Ruby 3.0.2.

I will file a bug upstream.

Do anyone know what regex implementation that Visual Studio Code uses?

[sebthom]

Based on https://github.com/microsoft/vscode-textmate#using I would say they use https://github.com/microsoft/vscode-oniguruma