Identify newline separated segments

[sungshik]

This PR adds improved support for segment-based "gobbling up" tokens in begin/end patterns. To motivate, suppose we have the following grammar:

lexical String = @category="string" "\"" Char* "\"";

lexical Char
    = [a-z]
    | "\\" [\"]
    ;

For instance, "asdf" and "foo\"bar" should be fully highlighted, but "foo"bar" should not be.

Before this PR, the following TextMate rule is generated (simplified):

{
  "begin": "\""
  "end": "\""
  "name": "string"
  "patterns": [
    { "match": "[a-z]" },
    { "match": "[\\]" },
    { "match": "[\"]" }
  ]
}

That is, roughly, for each symbol s in Char, a nested pattern is included to gobble up content between begin and end that matches individually s. However, gobbling up individual symbols is too fine-grained. For instance, with this rule, "foo\"bar" will not be fully highlighted: only "foo\" will be highlighted (because the \ is not considered in combination with the second ").

Instead of gobbling up individual symbols, the tokenizer should gobble up segments of symbols that belong together. That is what this PR adds.

After this PR, the following TextMate rule is generated (simplified):

{
  "begin": "\""
  "end": "\""
  "name": "string"
  "patterns": [
    { "match": "[a-z]" },
    { "match": "[\\][\"]" }
  ]
}

[sungshik]

A few additional comments

[DavyLandman]

This looks a lot better indeed 👍 also the textmate regexes look more like you would expect them too.

[sungshik]

Thank you for the quick review, @DavyLandman, 🙂