Fix gen lexer word boundary, case insensitive, and literal matching cases (plus conformance tests) #274
Conversation
Also add conformance tests for these cases
|
Upon further inspection, the previous iteration of the word boundary conformance test was failing because case insensitive matching wasn't implemented. Updated to fix case insensitive matching and add new tests for this case. And then changed the word boundary test to not rely on case insensitive matching. |
Also add conformance test for this case
If \b is used at start of pattern, it could match right before the current position, rather than right after. Check both cases.
|
Identified and fixed another edge case where literal matches were not working if they were at exactly the end of the string. Also identified and fixed the issue with matching |
klondikedragon
changed the title
\b
| if n == 1 { | ||
| fmt.Fprintf(w, "if p < len(s) && s[p] == %q {\n", re.Rune[0]) | ||
| if re.Flags&syntax.FoldCase != 0 { | ||
| fmt.Fprintf(w, "if p+%d <= len(s) && strings.EqualFold(s[p:p+%d], %q) {\n", n, n, string(re.Rune)) |
| @@ -18,6 +18,9 @@ var conformanceLexer = lexer.MustStateful(lexer.Rules{ | |||
| "Root": { | |||
| {"String", `"`, lexer.Push("String")}, | |||
| // {"Heredoc", `<<(\w+)`, lexer.Push("Heredoc")}, | |||
| {"LiteralTest", `LITTEST:`, lexer.Push("LiteralTest")}, | |||
There was a problem hiding this comment.
Yeah nice, I like this pattern.
There was a problem hiding this comment.
Would you mind converting the existing tests to use this pattern too?
| {"LITKeyword", "SELECT"}, | ||
| }}, | ||
| {"LiteralMixed", `LITTEST:hello ONE test LIKE world`, []token{ | ||
| {"LiteralTest", "LITTEST:"}, |
There was a problem hiding this comment.
Maybe we could make the test harness strip these automatically? ie. the first token iff it ends with TEST:
| @@ -117,7 +200,7 @@ func TestLexerConformanceGenerated(t *testing.T) { | |||
| args := []string{"test", "-run", "TestLexerConformanceGeneratedInternal", "-tags", "generated"} | |||
| // Propagate test flags. | |||
| flag.CommandLine.VisitAll(func(f *flag.Flag) { | |||
| if f.Value.String() != f.DefValue { | |||
| if f.Value.String() != f.DefValue && f.Name != "test.run" { | |||
|
@alecthomas - I did some cleanup, see what you think. Thanks! |
|
Awesome, thanks :) |
@alecthomas - I wanted to get your feedback early on the approach to add a specialized test to the conformance test suite. It uses a special token (e.g.,
WBTEST:) to push into a new lexer state that then might have different rules vs the rest of the grammar. This would allow for testing different areas of the lexer without having to add more and more rules to the Root state (which might start interacting with each other over time as the number of conformance test cases grow and the rules get more complex).Opening as a draft to get your feedback on the approach. This demonstrates how the generated lexer currently doesn't support the word boundary
\bregex properly. (Will next see what might be needed to fix this in the lexer generation.)This also tweaks
TestLexerConformanceGeneratedso that if you run this test function from VSCode IDE it won't just hang forever (the command that eventually gets run before this change wasgo test -run TestLexerConformanceGeneratedInternal -tags generated '-test.run=^TestLexerConformanceGenerated$', which confuses go and causes it to hang forever until Ctrl+C is pressed).