Support generics over multiple lines

[frigus02]

Fixes #20

This may be harder to do that I had hoped:

As shown in the screenshot the textmate scopes don't seem to understand generics over multiple lines. That means we can't piggyback on those regexes and have to write our own. The reason for why they're still highlighted properly is semantic highlighting. Without that it looks like this:

[karlhorky]

Ah, this seems like it would be a bit challenging, since type parameters can also receive type parameters!

I guess there was microsoft/TypeScript-TmLanguage#452, but it seems like the eventual commit microsoft/TypeScript-TmLanguage@632a9db didn't resolve the issue that is causing these difficulties here...

Btw, I've asked about multi-line generic type parameter support with the lit-html extension over here too: mjbvz/vscode-lit-html#96

[karlhorky]

I'm guessing the tokens that you're looking for are these ones:

punctuation.definition.typeparameters.begin.ts
meta.type.parameters.ts
string.template.ts

I showed this overlay by using the Developer: Inspect Editor Tokens and Scopes command (aka the Scope Inspector)

[karlhorky]

Oh it looks like this may be a limitation of the TextMate tmLanguage single-line lookup: microsoft/TypeScript-TmLanguage#761

[frigus02]

Yeah, the type parameter scopes are missing once the type parameters are spread across multiple lines. This may be possible to support with multiple regular expressions, but I don't feel like writing those myself. The way to go is probably to hook into the semantic highlighting if that's supported.

[karlhorky]

Oh ok, wasn't aware that the semantic highlighting information was available to extensions / TS language service plugins. That would be the best, for sure!

If that is complex, then writing regex is something that I have done a lot, and may be able to help there.

[frigus02]

Oh ok, wasn't aware that the semantic highlighting information was available to extensions / TS language service plugins. That would be the best, for sure!

Sorry, I wasn't clear. I don't know if extensions can hook into semantic highlighting. If they can, that may be the way to go.

If that is complex, then writing regex is something that I have done a lot, and may be able to help there.

The textmate grammar regex do somehow support function parameters over multiple lines. This makes me think that in theory they should be able to support type parameters over multiple lines, too. Those regexes are massive, though, e.g.:

vscode-sql-tagged-template-literals/extension-syntax-only/syntaxes/grammar.json

Line 8 in 5488239

"begin": "(?:([_$[:alpha:]][_$[:alnum:]]*)(?:\\s*(\\??\\.)\\s*(\\#?[_$[:alpha:]][_$[:alnum:]]*))*\\s*(\\??\\.))?\\s*(#?(?i)sql|sqlFragment(?-i))\\s*(?=(?:(\\?\\.\\s*)|(\\!))?((<\\s*(((keyof|infer|awaited|typeof|readonly)\\s+)|(([_$[:alpha:]][_$[:alnum:]]*|(\\{([^\\{\\}]|(\\{([^\\{\\}]|\\{[^\\{\\}]*\\})*\\}))*\\})|(\\(([^\\(\\)]|(\\(([^\\(\\)]|\\([^\\(\\)]*\\))*\\)))*\\))|(\\[([^\\[\\]]|(\\[([^\\[\\]]|\\[[^\\[\\]]*\\])*\\]))*\\])|(\\'([^\\'\\\\]|\\\\.)*\\')|(\\\"([^\\\"\\\\]|\\\\.)*\\\")|(\\`([^\\`\\\\]|\\\\.)*\\`))(?=\\s*([\\<\\>\\,\\.\\[]|=>|&(?!&)|\\|(?!\\|)))))([^<>\\(]|(\\(([^\\(\\)]|(\\(([^\\(\\)]|\\([^\\(\\)]*\\))*\\)))*\\))|(?<==)\\>|\\<\\s*(((keyof|infer|awaited|typeof|readonly)\\s+)|(([_$[:alpha:]][_$[:alnum:]]*|(\\{([^\\{\\}]|(\\{([^\\{\\}]|\\{[^\\{\\}]*\\})*\\}))*\\})|(\\(([^\\(\\)]|(\\(([^\\(\\)]|\\([^\\(\\)]*\\))*\\)))*\\))|(\\[([^\\[\\]]|(\\[([^\\[\\]]|\\[[^\\[\\]]*\\])*\\]))*\\])|(\\'([^\\'\\\\]|\\\\.)*\\')|(\\\"([^\\\"\\\\]|\\\\.)*\\\")|(\\`([^\\`\\\\]|\\\\.)*\\`))(?=\\s*([\\<\\>\\,\\.\\[]|=>|&(?!&)|\\|(?!\\|)))))(([^<>\\(]|(\\(([^\\(\\)]|(\\(([^\\(\\)]|\\([^\\(\\)]*\\))*\\)))*\\))|(?<==)\\>|\\<\\s*(((keyof|infer|awaited|typeof|readonly)\\s+)|(([_$[:alpha:]][_$[:alnum:]]*|(\\{([^\\{\\}]|(\\{([^\\{\\}]|\\{[^\\{\\}]*\\})*\\}))*\\})|(\\(([^\\(\\)]|(\\(([^\\(\\)]|\\([^\\(\\)]*\\))*\\)))*\\))|(\\[([^\\[\\]]|(\\[([^\\[\\]]|\\[[^\\[\\]]*\\])*\\]))*\\])|(\\'([^\\'\\\\]|\\\\.)*\\')|(\\\"([^\\\"\\\\]|\\\\.)*\\\")|(\\`([^\\`\\\\]|\\\\.)*\\`))(?=\\s*([\\<\\>\\,\\.\\[]|=>|&(?!&)|\\|(?!\\|)))))([^<>\\(]|(\\(([^\\(\\)]|(\\(([^\\(\\)]|\\([^\\(\\)]*\\))*\\)))*\\))|(?<==)\\>)*(?<!=)\\>))*(?<!=)\\>)*(?<!=)>\\s*)?\\())",

I haven't written this myself. I copied it from the TypeScript grammar.

Since the regex is so massive, I've even written regex-help.mjs to add/remove newlines/indentation to a regex to make it easier to read. It's been a while, so I don't fully remember. I think I copied the regular expression to a file, the did something like:

# copy regex to file regex.txt
$ node ./regex-help.mjs prettify regex.txt >pretty.txt
# edit pretty.txt
$ node ./regex-help.mjs minify pretty.txt >new_regex.txt
# copy new regex and add to grammar

The bits that I've learned about textmate grammar, I've tried to document here: https://github.com/frigus02/vscode-sql-tagged-template-literals#interesting-bits-about-the-grammar

If you can come up with a regex that supports multi line type parameters I'd be happy to have a look!

[tantaman]

Any progress on this one?

I ran into the same problem when implementing this project https://github.com/vlcn-io/typed-sql which generates TypeScript types from the query string in the template literal.

E.g.,

cities-example.mov

[tantaman]

hmm, the function call case you linked (

vscode-sql-tagged-template-literals/extension-syntax-only/syntaxes/grammar.json

Line 8 in 5488239

"begin": "(?:([_$[:alpha:]][_$[:alnum:]]*)(?:\\s*(\\??\\.)\\s*(\\#?[_$[:alpha:]][_$[:alnum:]]*))*\\s*(\\??\\.))?\\s*(#?(?i)sql|sqlFragment(?-i))\\s*(?=(?:(\\?\\.\\s*)|(\\!))?((<\\s*(((keyof|infer|awaited|typeof|readonly)\\s+)|(([_$[:alpha:]][_$[:alnum:]]*|(\\{([^\\{\\}]|(\\{([^\\{\\}]|\\{[^\\{\\}]*\\})*\\}))*\\})|(\\(([^\\(\\)]|(\\(([^\\(\\)]|\\([^\\(\\)]*\\))*\\)))*\\))|(\\[([^\\[\\]]|(\\[([^\\[\\]]|\\[[^\\[\\]]*\\])*\\]))*\\])|(\\'([^\\'\\\\]|\\\\.)*\\')|(\\\"([^\\\"\\\\]|\\\\.)*\\\")|(\\`([^\\`\\\\]|\\\\.)*\\`))(?=\\s*([\\<\\>\\,\\.\\[]|=>|&(?!&)|\\|(?!\\|)))))([^<>\\(]|(\\(([^\\(\\)]|(\\(([^\\(\\)]|\\([^\\(\\)]*\\))*\\)))*\\))|(?<==)\\>|\\<\\s*(((keyof|infer|awaited|typeof|readonly)\\s+)|(([_$[:alpha:]][_$[:alnum:]]*|(\\{([^\\{\\}]|(\\{([^\\{\\}]|\\{[^\\{\\}]*\\})*\\}))*\\})|(\\(([^\\(\\)]|(\\(([^\\(\\)]|\\([^\\(\\)]*\\))*\\)))*\\))|(\\[([^\\[\\]]|(\\[([^\\[\\]]|\\[[^\\[\\]]*\\])*\\]))*\\])|(\\'([^\\'\\\\]|\\\\.)*\\')|(\\\"([^\\\"\\\\]|\\\\.)*\\\")|(\\`([^\\`\\\\]|\\\\.)*\\`))(?=\\s*([\\<\\>\\,\\.\\[]|=>|&(?!&)|\\|(?!\\|)))))(([^<>\\(]|(\\(([^\\(\\)]|(\\(([^\\(\\)]|\\([^\\(\\)]*\\))*\\)))*\\))|(?<==)\\>|\\<\\s*(((keyof|infer|awaited|typeof|readonly)\\s+)|(([_$[:alpha:]][_$[:alnum:]]*|(\\{([^\\{\\}]|(\\{([^\\{\\}]|\\{[^\\{\\}]*\\})*\\}))*\\})|(\\(([^\\(\\)]|(\\(([^\\(\\)]|\\([^\\(\\)]*\\))*\\)))*\\))|(\\[([^\\[\\]]|(\\[([^\\[\\]]|\\[[^\\[\\]]*\\])*\\]))*\\])|(\\'([^\\'\\\\]|\\\\.)*\\')|(\\\"([^\\\"\\\\]|\\\\.)*\\\")|(\\`([^\\`\\\\]|\\\\.)*\\`))(?=\\s*([\\<\\>\\,\\.\\[]|=>|&(?!&)|\\|(?!\\|)))))([^<>\\(]|(\\(([^\\(\\)]|(\\(([^\\(\\)]|\\([^\\(\\)]*\\))*\\)))*\\))|(?<==)\\>)*(?<!=)\\>))*(?<!=)\\>)*(?<!=)>\\s*)?\\())",

) is a good hint as it highlights correctly after line breaks.

Maybe this won't be too hard to get working based on that example.

@frigus02 - is there a good way to iterate on these regexes as I change them? I have an idea on how to make it work but I'm not sure of a good testing flow other than:

1. bundle the code into an extension
2. install the extension
3. reload vscode
4. test

which is really laborious

[tantaman]

is there a good way to iterate on these regexes as I change them? I have an idea on how to make it work but I'm not sure of a good testing flow other than:

Ok, ChatGPT helped me out on this one :)

Basically just edit the extension within the install location and do cmd + r to reload the vscode window after changes.

---

And, voila, I have something that might just work:

I'll submit a PR after more testing.

[tantaman]

This is my first time doing tmlanguage definitions but it seems like your regexes can be vastly simplified.

I opened a PR with my changes. The test.ts file looks good even with the vast simplifications to the grammar definition.

#28