anchor operation seems sparadic past first repetition on a line.

[msftrncs]

Regarding microsoft/vscode#57831, using VS Code 1.27.2,

syntax (JSON) as simplified:

{
	"name": "PowerShell",
	"scopeName": "source.powershell",
	"patterns": [
		{
			"include": "#variable"
		},
		{
			"include": "#operators"
		}
	],
	"repository": {
		"operators": {
			"patterns": [
				{
					"match": "(?:\\G|(?<!\\p{L}|!))-(?i:is(?:not)?|as)(?!\\p{L})",
					"name": "keyword.operator.comparison.powershell"
				},
				{
					"match": "(?:\\G|(?<!\\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\\p{L})",
					"name": "keyword.operator.comparison.powershell"
				},
				{
					"match": "(?:\\G|(?<!\\p{L}|!))-(?i:join|split)(?!\\p{L})|--|\\+\\+",
					"name": "keyword.operator.unary.powershell"
				},
				{
					"match": "(?:\\G|(?<!\\p{L}|!))-(?i:and|or|not|xor)(?!\\p{L})|!",
					"name": "keyword.operator.logical.powershell"
				},
				{
					"match": "(?:\\G|(?<!\\p{L}|!))-(?i:band|bor|bnot|bxor|shl|shr)(?!\\p{L})",
					"name": "keyword.operator.bitwise.powershell"
				},
				{
					"match": "(?:\\G|(?<!\\p{L}|!))-(?i:f)(?!\\p{L})",
					"name": "keyword.operator.string-format.powershell"
				},
				{
					"match": "[+%*/-]?=|[+/*%]|-(?!\\p{L}|-%)",
					"name": "keyword.operator.assignment.powershell"
				},
				{
					"match": "\\|\\||&&|;",
					"name": "punctuation.terminator.statement.powershell"
				},
				{
					"match": "&|(?<!\\p{L}|!)(?<!\\.)\\.(?=\\s)|,|\\|",
					"name": "keyword.operator.other.powershell"
				},
				{
					"comment": "This is very imprecise, is there a syntax for 'must come after...' ",
					"match": "(?<!^)\\s*\\.\\.\\s*(?=\\-?\\d|\\(|\\$)",
					"name": "keyword.operator.range.powershell"
				}
			]
		},
		"variable": {
			"patterns": [
				{
					"captures": {
						"0": {
							"name": "variable.other.readwrite.powershell"
						},
						"1": {
							"name": "punctuation.definition.variable.powershell"
						}
					},
					"begin": "(\\$)(?:\\p{L}|\\d|[_?])+",
					"patterns": [
						{
							"include": "#operators"
						}
					],
					"name": "meta.test",
					"end": "(?!\\G)",
					"applyEndPatternLast": 1
				}
			]
		}
	}
}

applyEndPatternLast was an experiment. Had no affect as either 1 or "1" or not present.

A test set as

# Operators
if (10 -cgt 100) { }
$a -is $b
$b -contains $c
$x -notcontains $c
$a -match $b
$a-notmatch $b
$x -like $c
100-and 0
$a-ceq 4 -and $a-ine $d-or
$c -is [Type]
$c -isnot [Type]
$c -as [Type]
$k = $y -bor $k
$x = $y -and $x-and $z
$x = $y-and $x-and $z-split 
$z = -bnot $x
$k = $y -xor $b
$k = $y -bxor $b
$a -icontains $c-or
$a-ccontains $c
$a-iNotContains $c
$a-cNotContains $c
$a-cmatch $c
$x-iMatch $c
$x -iNotMatch $c
$a -iLike $b
$b -cLike $c
"hey" -cgt "Hey"
"Hey" -igt "hey"
"hey" -cge "Hey"
"Hey" -ige "hey"
"HEY" -clt "hey"
"HEY" -ilt "hey"
"HEY" -cle "hey"
"HEY" -ile "hey"

A variable that starts the line can always match the operator immediately behind it, but after that, I cannot seem to find a reason for why it sometimes matches, sometimes doesn't. It specifically deals with the operators that, if they were immediately following a command name, would be possibly an extension of the command name, so a lookbehind is used to verify the last digit isn't alpha, when in reality, what needs to be checked is the last scope, so I am trying to use the anchor \G (as an alternate), as all these operators can immediately follow a variable. I have been looking for a way to support the operators following items where the '-' is a separator, so that a space is not required (since PowerShell doesn't require a space). I have more ideas how to handle this, but I really think there is something going wrong with the rule parsing, probably around the anchor. I'll probably try separating the operators from the lookbehind, so that in this case, I can just recursively call the base operators, but please keep in mind I am just editing the original VS Code language file attempting to make it work better.

Unexpectedly the scope 'meta.test' is showing up on the spaces after the variable matches, and on the failed operator matches, the 'meta.test' scope shows up on the '-' only. However, if I change the end match to (?=.|$) then that doesn't happen, but everything is still exactly the same for the matching of the operators. Interesting note, when it normally works, the operator immediately following the variable will be included in 'meta.test' as expected, but when a part that isn't working, is edited (remove spaces from places on the same line) and suddenly starts working, the operator matches that suddenly start working are NOT in the 'meta.test' scope as it would have been expected!

[alexdima]

Have you tried cloning this repository, running npm install and then npm run inspect with your grammar and your test file in this repository?

It is a good tool that explains how states are pushed and popped and why.

Here is the output for your test case:

OUTPUT

> vscode-textmate@4.0.1 inspect /Users/alex/src/vscode-textmate
> node scripts/inspect.js "testGrammar.json" "testFile.txt"

LOADING GRAMMAR: testGrammar.json


===========================================
TOKENIZING LINE 1: |# Operators|

@@scanNext 0: |# Operators\n|
  no more matches.
  token: |# Operators\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 2: |if (10 -cgt 100) { }|

@@scanNext 0: |if (10 -cgt 100) { }\n|
matched: 7 / 11
  token: |if (10 |
      * source.powershell
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-cgt|
      * source.powershell
      * keyword.operator.comparison.powershell

@@scanNext 11: | 100) { }\n|
  no more matches.
  token: | 100) { }\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 3: |$a -is $b|

@@scanNext 0: |$a -is $b\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |a|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | -is $b\n|
matched: 3 / 6
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#9 @ testGrammar.json:15 - (?:\G|(?<!\p{L}|!))-(?i:is(?:not)?|as)(?!\p{L})
  token: |-is|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 6: | $b\n|
matched: 6 / 6
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 6: | $b\n|
matched: 7 / 9
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |b|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 9: |\n|
matched: 10 / 10
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 10: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 4: |$b -contains $c|

@@scanNext 0: |$b -contains $c\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |b|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | -contains $c\n|
matched: 3 / 12
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-contains|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 12: | $c\n|
matched: 12 / 12
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 12: | $c\n|
matched: 13 / 15
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |c|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 15: |\n|
matched: 16 / 16
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 16: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 5: |$x -notcontains $c|

@@scanNext 0: |$x -notcontains $c\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |x|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | -notcontains $c\n|
matched: 3 / 15
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-notcontains|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 15: | $c\n|
matched: 15 / 15
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 15: | $c\n|
matched: 16 / 18
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |c|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 18: |\n|
matched: 19 / 19
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 19: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 6: |$a -match $b|

@@scanNext 0: |$a -match $b\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |a|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | -match $b\n|
matched: 3 / 9
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-match|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 9: | $b\n|
matched: 9 / 9
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 9: | $b\n|
matched: 10 / 12
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |b|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 12: |\n|
matched: 13 / 13
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 13: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 7: |$a-notmatch $b|

@@scanNext 0: |$a-notmatch $b\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |a|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: |-notmatch $b\n|
matched: 2 / 11
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-notmatch|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 11: | $b\n|
matched: 11 / 11
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 11: | $b\n|
matched: 12 / 14
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |b|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 14: |\n|
matched: 15 / 15
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 15: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 8: |$x -like $c|

@@scanNext 0: |$x -like $c\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |x|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | -like $c\n|
matched: 3 / 8
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-like|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 8: | $c\n|
matched: 8 / 8
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 8: | $c\n|
matched: 9 / 11
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |c|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 11: |\n|
matched: 12 / 12
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 12: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 9: |100-and 0|

@@scanNext 0: |100-and 0\n|
matched: 3 / 7
  token: |100|
      * source.powershell
  matched MatchRule#12 @ testGrammar.json:27 - (?:\G|(?<!\p{L}|!))-(?i:and|or|not|xor)(?!\p{L})|!
  token: |-and|
      * source.powershell
      * keyword.operator.logical.powershell

@@scanNext 7: | 0\n|
  no more matches.
  token: | 0\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 10: |$a-ceq 4 -and $a-ine $d-or|

@@scanNext 0: |$a-ceq 4 -and $a-ine $d-or\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |a|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: |-ceq 4 -and $a-ine $d-or\n|
matched: 2 / 6
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-ceq|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 6: | 4 -and $a-ine $d-or\n|
matched: 6 / 6
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 6: | 4 -and $a-ine $d-or\n|
matched: 9 / 13
  token: | 4 |
      * source.powershell
  matched MatchRule#12 @ testGrammar.json:27 - (?:\G|(?<!\p{L}|!))-(?i:and|or|not|xor)(?!\p{L})|!
  token: |-and|
      * source.powershell
      * keyword.operator.logical.powershell

@@scanNext 13: | $a-ine $d-or\n|
matched: 14 / 16
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |a|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 16: |-ine $d-or\n|
matched: 16 / 20
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-ine|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 20: | $d-or\n|
matched: 20 / 20
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 20: | $d-or\n|
matched: 21 / 23
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |d|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 23: |-or\n|
matched: 23 / 26
  matched MatchRule#12 @ testGrammar.json:27 - (?:\G|(?<!\p{L}|!))-(?i:and|or|not|xor)(?!\p{L})|!
  token: |-or|
      * source.powershell
      * meta.test
      * keyword.operator.logical.powershell

@@scanNext 26: |\n|
matched: 26 / 26
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 26: |\n|
  no more matches.
  token: |\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 11: |$c -is [Type]|

@@scanNext 0: |$c -is [Type]\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |c|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | -is [Type]\n|
matched: 3 / 6
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#9 @ testGrammar.json:15 - (?:\G|(?<!\p{L}|!))-(?i:is(?:not)?|as)(?!\p{L})
  token: |-is|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 6: | [Type]\n|
matched: 6 / 6
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 6: | [Type]\n|
  no more matches.
  token: | [Type]\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 12: |$c -isnot [Type]|

@@scanNext 0: |$c -isnot [Type]\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |c|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | -isnot [Type]\n|
matched: 3 / 9
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#9 @ testGrammar.json:15 - (?:\G|(?<!\p{L}|!))-(?i:is(?:not)?|as)(?!\p{L})
  token: |-isnot|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 9: | [Type]\n|
matched: 9 / 9
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 9: | [Type]\n|
  no more matches.
  token: | [Type]\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 13: |$c -as [Type]|

@@scanNext 0: |$c -as [Type]\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |c|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | -as [Type]\n|
matched: 3 / 6
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#9 @ testGrammar.json:15 - (?:\G|(?<!\p{L}|!))-(?i:is(?:not)?|as)(?!\p{L})
  token: |-as|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 6: | [Type]\n|
matched: 6 / 6
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 6: | [Type]\n|
  no more matches.
  token: | [Type]\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 14: |$k = $y -bor $k|

@@scanNext 0: |$k = $y -bor $k\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |k|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | = $y -bor $k\n|
matched: 3 / 4
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#15 @ testGrammar.json:39 - [+%*/-]?=|[+/*%]|-(?!\p{L}|-%)
  token: |=|
      * source.powershell
      * meta.test
      * keyword.operator.assignment.powershell

@@scanNext 4: | $y -bor $k\n|
matched: 4 / 4
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 4: | $y -bor $k\n|
matched: 5 / 7
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |y|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 7: | -bor $k\n|
matched: 8 / 12
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#13 @ testGrammar.json:31 - (?:\G|(?<!\p{L}|!))-(?i:band|bor|bnot|bxor|shl|shr)(?!\p{L})
  token: |-bor|
      * source.powershell
      * meta.test
      * keyword.operator.bitwise.powershell

@@scanNext 12: | $k\n|
matched: 12 / 12
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 12: | $k\n|
matched: 13 / 15
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |k|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 15: |\n|
matched: 16 / 16
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 16: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 15: |$x = $y -and $x-and $z|

@@scanNext 0: |$x = $y -and $x-and $z\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |x|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | = $y -and $x-and $z\n|
matched: 3 / 4
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#15 @ testGrammar.json:39 - [+%*/-]?=|[+/*%]|-(?!\p{L}|-%)
  token: |=|
      * source.powershell
      * meta.test
      * keyword.operator.assignment.powershell

@@scanNext 4: | $y -and $x-and $z\n|
matched: 4 / 4
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 4: | $y -and $x-and $z\n|
matched: 5 / 7
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |y|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 7: | -and $x-and $z\n|
matched: 8 / 12
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#12 @ testGrammar.json:27 - (?:\G|(?<!\p{L}|!))-(?i:and|or|not|xor)(?!\p{L})|!
  token: |-and|
      * source.powershell
      * meta.test
      * keyword.operator.logical.powershell

@@scanNext 12: | $x-and $z\n|
matched: 12 / 12
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 12: | $x-and $z\n|
matched: 13 / 15
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |x|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 15: |-and $z\n|
matched: 15 / 19
  matched MatchRule#12 @ testGrammar.json:27 - (?:\G|(?<!\p{L}|!))-(?i:and|or|not|xor)(?!\p{L})|!
  token: |-and|
      * source.powershell
      * meta.test
      * keyword.operator.logical.powershell

@@scanNext 19: | $z\n|
matched: 19 / 19
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 19: | $z\n|
matched: 20 / 22
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |z|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 22: |\n|
matched: 23 / 23
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 23: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 16: |$x = $y-and $x-and $z-split|

@@scanNext 0: |$x = $y-and $x-and $z-split\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |x|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | = $y-and $x-and $z-split\n|
matched: 3 / 4
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#15 @ testGrammar.json:39 - [+%*/-]?=|[+/*%]|-(?!\p{L}|-%)
  token: |=|
      * source.powershell
      * meta.test
      * keyword.operator.assignment.powershell

@@scanNext 4: | $y-and $x-and $z-split\n|
matched: 4 / 4
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 4: | $y-and $x-and $z-split\n|
matched: 5 / 7
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |y|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 7: |-and $x-and $z-split\n|
matched: 8 / 8
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |-|
      * source.powershell
      * meta.test

@@scanNext 8: |and $x-and $z-split\n|
matched: 12 / 14
  token: |and |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |x|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 14: |-and $z-split\n|
matched: 15 / 15
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |-|
      * source.powershell
      * meta.test

@@scanNext 15: |and $z-split\n|
matched: 19 / 21
  token: |and |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |z|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 21: |-split\n|
matched: 22 / 22
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |-|
      * source.powershell
      * meta.test

@@scanNext 22: |split\n|
  no more matches.
  token: |split\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 17: |$z = -bnot $x|

@@scanNext 0: |$z = -bnot $x\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |z|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | = -bnot $x\n|
matched: 3 / 4
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#15 @ testGrammar.json:39 - [+%*/-]?=|[+/*%]|-(?!\p{L}|-%)
  token: |=|
      * source.powershell
      * meta.test
      * keyword.operator.assignment.powershell

@@scanNext 4: | -bnot $x\n|
matched: 4 / 4
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 4: | -bnot $x\n|
matched: 5 / 10
  token: | |
      * source.powershell
  matched MatchRule#13 @ testGrammar.json:31 - (?:\G|(?<!\p{L}|!))-(?i:band|bor|bnot|bxor|shl|shr)(?!\p{L})
  token: |-bnot|
      * source.powershell
      * keyword.operator.bitwise.powershell

@@scanNext 10: | $x\n|
matched: 11 / 13
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |x|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 13: |\n|
matched: 14 / 14
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 14: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 18: |$k = $y -xor $b|

@@scanNext 0: |$k = $y -xor $b\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |k|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | = $y -xor $b\n|
matched: 3 / 4
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#15 @ testGrammar.json:39 - [+%*/-]?=|[+/*%]|-(?!\p{L}|-%)
  token: |=|
      * source.powershell
      * meta.test
      * keyword.operator.assignment.powershell

@@scanNext 4: | $y -xor $b\n|
matched: 4 / 4
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 4: | $y -xor $b\n|
matched: 5 / 7
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |y|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 7: | -xor $b\n|
matched: 8 / 12
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#12 @ testGrammar.json:27 - (?:\G|(?<!\p{L}|!))-(?i:and|or|not|xor)(?!\p{L})|!
  token: |-xor|
      * source.powershell
      * meta.test
      * keyword.operator.logical.powershell

@@scanNext 12: | $b\n|
matched: 12 / 12
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 12: | $b\n|
matched: 13 / 15
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |b|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 15: |\n|
matched: 16 / 16
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 16: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 19: |$k = $y -bxor $b|

@@scanNext 0: |$k = $y -bxor $b\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |k|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | = $y -bxor $b\n|
matched: 3 / 4
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#15 @ testGrammar.json:39 - [+%*/-]?=|[+/*%]|-(?!\p{L}|-%)
  token: |=|
      * source.powershell
      * meta.test
      * keyword.operator.assignment.powershell

@@scanNext 4: | $y -bxor $b\n|
matched: 4 / 4
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 4: | $y -bxor $b\n|
matched: 5 / 7
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |y|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 7: | -bxor $b\n|
matched: 8 / 13
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#13 @ testGrammar.json:31 - (?:\G|(?<!\p{L}|!))-(?i:band|bor|bnot|bxor|shl|shr)(?!\p{L})
  token: |-bxor|
      * source.powershell
      * meta.test
      * keyword.operator.bitwise.powershell

@@scanNext 13: | $b\n|
matched: 13 / 13
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 13: | $b\n|
matched: 14 / 16
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |b|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 16: |\n|
matched: 17 / 17
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 17: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 20: |$a -icontains $c-or|

@@scanNext 0: |$a -icontains $c-or\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |a|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | -icontains $c-or\n|
matched: 3 / 13
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-icontains|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 13: | $c-or\n|
matched: 13 / 13
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 13: | $c-or\n|
matched: 14 / 16
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |c|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 16: |-or\n|
matched: 17 / 17
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |-|
      * source.powershell
      * meta.test

@@scanNext 17: |or\n|
  no more matches.
  token: |or\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 21: |$a-ccontains $c|

@@scanNext 0: |$a-ccontains $c\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |a|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: |-ccontains $c\n|
matched: 2 / 12
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-ccontains|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 12: | $c\n|
matched: 12 / 12
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 12: | $c\n|
matched: 13 / 15
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |c|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 15: |\n|
matched: 16 / 16
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 16: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 22: |$a-iNotContains $c|

@@scanNext 0: |$a-iNotContains $c\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |a|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: |-iNotContains $c\n|
matched: 2 / 15
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-iNotContains|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 15: | $c\n|
matched: 15 / 15
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 15: | $c\n|
matched: 16 / 18
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |c|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 18: |\n|
matched: 19 / 19
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 19: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 23: |$a-cNotContains $c|

@@scanNext 0: |$a-cNotContains $c\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |a|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: |-cNotContains $c\n|
matched: 2 / 15
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-cNotContains|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 15: | $c\n|
matched: 15 / 15
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 15: | $c\n|
matched: 16 / 18
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |c|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 18: |\n|
matched: 19 / 19
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 19: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 24: |$a-cmatch $c|

@@scanNext 0: |$a-cmatch $c\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |a|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: |-cmatch $c\n|
matched: 2 / 9
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-cmatch|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 9: | $c\n|
matched: 9 / 9
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 9: | $c\n|
matched: 10 / 12
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |c|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 12: |\n|
matched: 13 / 13
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 13: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 25: |$x-iMatch $c|

@@scanNext 0: |$x-iMatch $c\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |x|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: |-iMatch $c\n|
matched: 2 / 9
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-iMatch|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 9: | $c\n|
matched: 9 / 9
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 9: | $c\n|
matched: 10 / 12
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |c|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 12: |\n|
matched: 13 / 13
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 13: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 26: |$x -iNotMatch $c|

@@scanNext 0: |$x -iNotMatch $c\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |x|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | -iNotMatch $c\n|
matched: 3 / 13
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-iNotMatch|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 13: | $c\n|
matched: 13 / 13
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 13: | $c\n|
matched: 14 / 16
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |c|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 16: |\n|
matched: 17 / 17
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 17: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 27: |$a -iLike $b|

@@scanNext 0: |$a -iLike $b\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |a|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | -iLike $b\n|
matched: 3 / 9
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-iLike|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 9: | $b\n|
matched: 9 / 9
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 9: | $b\n|
matched: 10 / 12
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |b|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 12: |\n|
matched: 13 / 13
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 13: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 28: |$b -cLike $c|

@@scanNext 0: |$b -cLike $c\n|
matched: 0 / 2
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |b|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 2: | -cLike $c\n|
matched: 3 / 9
  token: | |
      * source.powershell
      * meta.test
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-cLike|
      * source.powershell
      * meta.test
      * keyword.operator.comparison.powershell

@@scanNext 9: | $c\n|
matched: 9 / 9
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)

@@scanNext 9: | $c\n|
matched: 10 / 12
  token: | |
      * source.powershell
  pushing BeginEndRule#3 @ testGrammar.json:60 - (\$)(?:\p{L}|\d|[_?])+
  token: |$|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell
      * punctuation.definition.variable.powershell
  token: |c|
      * source.powershell
      * meta.test
      * variable.other.readwrite.powershell

@@scanNext 12: |\n|
matched: 13 / 13
  popping BeginEndRule#3 @ testGrammar.json:60 - (?!\G)
  token: |\n|
      * source.powershell
      * meta.test

@@scanNext 13: |n|
  no more matches.

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 29: |"hey" -cgt "Hey"|

@@scanNext 0: |"hey" -cgt "Hey"\n|
matched: 6 / 10
  token: |"hey" |
      * source.powershell
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-cgt|
      * source.powershell
      * keyword.operator.comparison.powershell

@@scanNext 10: | "Hey"\n|
  no more matches.
  token: | "Hey"\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 30: |"Hey" -igt "hey"|

@@scanNext 0: |"Hey" -igt "hey"\n|
matched: 6 / 10
  token: |"Hey" |
      * source.powershell
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-igt|
      * source.powershell
      * keyword.operator.comparison.powershell

@@scanNext 10: | "hey"\n|
  no more matches.
  token: | "hey"\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 31: |"hey" -cge "Hey"|

@@scanNext 0: |"hey" -cge "Hey"\n|
matched: 6 / 10
  token: |"hey" |
      * source.powershell
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-cge|
      * source.powershell
      * keyword.operator.comparison.powershell

@@scanNext 10: | "Hey"\n|
  no more matches.
  token: | "Hey"\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 32: |"Hey" -ige "hey"|

@@scanNext 0: |"Hey" -ige "hey"\n|
matched: 6 / 10
  token: |"Hey" |
      * source.powershell
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-ige|
      * source.powershell
      * keyword.operator.comparison.powershell

@@scanNext 10: | "hey"\n|
  no more matches.
  token: | "hey"\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 33: |"HEY" -clt "hey"|

@@scanNext 0: |"HEY" -clt "hey"\n|
matched: 6 / 10
  token: |"HEY" |
      * source.powershell
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-clt|
      * source.powershell
      * keyword.operator.comparison.powershell

@@scanNext 10: | "hey"\n|
  no more matches.
  token: | "hey"\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 34: |"HEY" -ilt "hey"|

@@scanNext 0: |"HEY" -ilt "hey"\n|
matched: 6 / 10
  token: |"HEY" |
      * source.powershell
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-ilt|
      * source.powershell
      * keyword.operator.comparison.powershell

@@scanNext 10: | "hey"\n|
  no more matches.
  token: | "hey"\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 35: |"HEY" -cle "hey"|

@@scanNext 0: |"HEY" -cle "hey"\n|
matched: 6 / 10
  token: |"HEY" |
      * source.powershell
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-cle|
      * source.powershell
      * keyword.operator.comparison.powershell

@@scanNext 10: | "hey"\n|
  no more matches.
  token: | "hey"\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined


===========================================
TOKENIZING LINE 36: |"HEY" -ile "hey"|

@@scanNext 0: |"HEY" -ile "hey"\n|
matched: 6 / 10
  token: |"HEY" |
      * source.powershell
  matched MatchRule#10 @ testGrammar.json:19 - (?:\G|(?<!\p{L}|!))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L})
  token: |-ile|
      * source.powershell
      * keyword.operator.comparison.powershell

@@scanNext 10: | "hey"\n|
  no more matches.
  token: | "hey"\n|
      * source.powershell

@@LINE END RULE STACK CONTAINS 1 RULES:
  * no rule description found for rule id: undefined

[msftrncs]

The output fails to tell me 'why'

So, I forked, downloaded, and attempted to build, fixed prerequisites to the prerequisites, then fixed issues with the prerequisites, finally got a build, and finally got an output. Added to the '@@ScanNext' logging, the anchor position, and confirmed that the anchor is where I think it is. So all of this does not explain why the matching against the anchor isn't working.

However, I removed the comment marks in front of the debug: (grammer.ts: matchRule)

		console.log('  scanning for');
		console.log(debugCompiledRuleToString(ruleScanner));

and this shows me something interesting. For some reason, in the matches, the \G is being changed to \￿ . (this looks very similar to what I think is the UTF-8 BOM?) Now why is that happening? In the places where the match of the anchor works as expected, the rule isn't corrupted, but in the ones where it fails the rule appears corrupted. However, to be clear, the end rule is also corrupted, yet it states it matches in those cases.

Also, notice on the end of many lines, that the last 2 tokens processed are |\n| and then |n|, but it doesn't happen on every line. That seems like something weird going on.

[alexdima]

The rule is not corrupted. When \G should not match we replace it with '\uFFFF' which should not match any valid Unicode character.

IMHO the explanation I have provided here for \G is complete.

Also, notice on the end of many lines, that the last 2 tokens processed are |\n| and then |n|, but it doesn't happen on every line. That seems like something weird going on.

I will investigate if you can demonstrate a clear issue: a grammar, a body of text, and tokenization results in TextMate (expected) that are different than the ones in vscode-textmate (actual)?

[msftrncs]

Please excuse me for an error I made. While the 'change of the G (or A) to a uFFFF' is pretty clear in the code, for some reason when I was looking at the log, I swear I saw the G translation at points where it shouldn't have occurred, but looking again (and not from a new run either, its still the same run) that's not true.

SO that means my real issue still stands. \G isn't reliably working!
Take for example:

@@scanNext 14,{14}: |-and $z-split \n|
  scanning for
   - 9: (?:\G|(?<![\p{L}!]))-(?i:is(?:not)?|as)(?!\p{L})
   - 10: (?:(?:\G|(?<![\p{L}!]))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L}))
   - 11: (?:(?:\G|(?<![\p{L}!]))-(?i:join|split)(?!\p{L})|--|\+\+)
   - 12: (?:(?:\G|(?<![\p{L}!]))-(?i:and|or|not|xor)(?!\p{L})|!)
   - 13: (?:(?:\G|(?<![\p{L}!]))-(?i:band|bor|bnot|bxor|shl|shr)(?!\p{L}))
   - 14: (?:(?:\G|(?<![\p{L}!]))-(?i:f)(?!\p{L}))
   - 15: [+%*/-]?=|[+/*%]|-(?!\p{L}|-%)
   - 16: \|\||&&|;
   - 17: &|(?<![\p{L}!])(?<!\.)\.(?=\s)|,|\|
   - 18: (?<!^)\s*\.\.\s*(?=\-?\d|\(|\$)
   - -1: (?<!\G)
matched: 15 / 15
  popping BeginEndRule#3 @ powershell.tmLanguage.json:60 - (?<!\G)
  token: |-|
      * source.powershell
      * meta.test

This particular rule is failing on two counts.

1. Rule # 12 should have matched.
2. Rule # 15 could have matched. (EDIT, no it couldn't, this is incorrect, sorry, letter character follows, making the trailing lookahead fail, a recent change I made in order to prevent this rule from potentially causing the others to not get the '-')
3. both would have matched at position 14, the anchor position, ,{14}

Instead, rule -1 matched at position 15. Something is wrong.

Here is nearly exactly the same match scenario, one line up, and it worked correctly.

@@scanNext 15,{15}: |-and $z\n|
  scanning for
   - 9: (?:\G|(?<![\p{L}!]))-(?i:is(?:not)?|as)(?!\p{L})
   - 10: (?:(?:\G|(?<![\p{L}!]))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L}))
   - 11: (?:(?:\G|(?<![\p{L}!]))-(?i:join|split)(?!\p{L})|--|\+\+)
   - 12: (?:(?:\G|(?<![\p{L}!]))-(?i:and|or|not|xor)(?!\p{L})|!)
   - 13: (?:(?:\G|(?<![\p{L}!]))-(?i:band|bor|bnot|bxor|shl|shr)(?!\p{L}))
   - 14: (?:(?:\G|(?<![\p{L}!]))-(?i:f)(?!\p{L}))
   - 15: [+%*/-]?=|[+/*%]|-(?!\p{L}|-%)
   - 16: \|\||&&|;
   - 17: &|(?<![\p{L}!])(?<!\.)\.(?=\s)|,|\|
   - 18: (?<!^)\s*\.\.\s*(?=\-?\d|\(|\$)
   - -1: (?<!\G)
matched: 15 / 19
  matched MatchRule#12 @ powershell.tmLanguage.json:27 - (?:(?:\G|(?<![\p{L}!]))-(?i:and|or|not|xor)(?!\p{L})|!)
  token: |-and|
      * source.powershell
      * meta.test
      * keyword.operator.logical.powershell

In both cases, the anchor should have matched and the negative lookbehind should have failed, so both matches should have occurred based solely on the anchor.

[msftrncs]

Regarding the extra |n| sometimes appearing on the end of a lines match log, I think its caused by doing the replace before the substr in the logging for scanNext:

console.log(`@@scanNext ${linePos},{${anchorPosition}}: |${lineText.content.replace(/\n$/, '\\n').substr(linePos)}|`);

The replacement changes the length of the string so that linePos isn't really valid, at least past that '\n'. These only occur when the last token on a line is matched (probably most specifically by a beginEndRule), but doesn't include the newline (\n), A rule pop occurs at after the newline, and then one more match is attempted with an empty remaining string.

[alexdima]

I confirm, there is a bug. Thank you for following through and for your patience!

It is shown here indeed:

@@scanNext 14,{14}: |-and $z-split \n|
  scanning for
   - 9: (?:\G|(?<![\p{L}!]))-(?i:is(?:not)?|as)(?!\p{L})
   - 10: (?:(?:\G|(?<![\p{L}!]))-(?i:[ic]?(?:eq|ne|[gl][te]|(?:not)?(?:like|match|contains|in)|replace))(?!\p{L}))
   - 11: (?:(?:\G|(?<![\p{L}!]))-(?i:join|split)(?!\p{L})|--|\+\+)
   - 12: (?:(?:\G|(?<![\p{L}!]))-(?i:and|or|not|xor)(?!\p{L})|!)
   - 13: (?:(?:\G|(?<![\p{L}!]))-(?i:band|bor|bnot|bxor|shl|shr)(?!\p{L}))
   - 14: (?:(?:\G|(?<![\p{L}!]))-(?i:f)(?!\p{L}))
   - 15: [+%*/-]?=|[+/*%]|-(?!\p{L}|-%)
   - 16: \|\||&&|;
   - 17: &|(?<![\p{L}!])(?<!\.)\.(?=\s)|,|\|
   - 18: (?<!^)\s*\.\.\s*(?=\-?\d|\(|\$)
   - -1: (?<!\G)
matched: 15 / 15
  popping BeginEndRule#3 @ powershell.tmLanguage.json:60 - (?<!\G)
  token: |-|
      * source.powershell
      * meta.test

Rule # 12 should have matched.

That is correct. The root cause is caching in node-oniguruma. The regex at #12 is once before evaluated on this line and then the fact that it doesn't match is recorded and for that line it is always considered to not match.

i.e. the root cause is that node-oniguruma should not cache results for regular expressions searches if the regular expression contains \G.

[alexdima]

@msftrncs I've created atom/node-oniguruma#87 and a PR to fix it atom/node-oniguruma#88

Once node-oniguruma fixes the issue and publishes a new version, we can adopt that in VS Code and \G should work reliably again.

Thank you!

[alexdima]

This has been fixed upstream and adopted in vscode via microsoft/vscode@f1f688c

Tomorrow's insiders build will have the fix.

[msftrncs]

@alexandrudima , confirming that I have tested an insiders build and my concept for matching PowerShell operators immediately after variables (relying on the \G) is now working. Thank-you.