Add more scopes to the tree-sitter grammar

[Ben3eeE]

🚨 WIP 🚨

TODO:

• Investigate open issues Add more scopes to the tree-sitter grammar #303 (comment)
• Scope punctuation in C

Refs atom/atom#18347 (comment)

Depends on: atom/atom#18382

Fixes: #297
Fixes: #296
Fixes: #295
Fixes: #294
Fixes: #293
Fixes: #291

/cc: @maxbrunsfeld

[thomasjo]

@Ben3eeE Haven't had time to check this myself yet, but does this cause any false positives with C++ template expressions? E.g. template <typename T>, template <std::vector<T>>, and so forth? For those expressions the brackets should not be marked as operators.

[Ben3eeE]

@thomasjo Thanks for pointing that out. It's entirely possible that it will conflict with template expressions. I haven't tested this myself yet actually. Just editing through the web editor 🙂

I will take a look at this as soon as I can. Are there any other examples you can think of?

[thomasjo]

@Ben3eeE Can't think of anything else right now, but I'll have a "quick" scan through the language specification later. Will let you know if I find anything exotic 😁

[Ben3eeE]

@thomasjo After atom/atom#18382 is merged this PR as it is now will work and not conflict with template expressions.

TextMate still scopes > and < in template expressions as keyword.operator.comparison so we might want to just leave it scoped incorrectly to not break highlighting of < and > in themes.

@maxbrunsfeld Do you think we should scope > and < as keyword.operator in template expressions to keep the highlighting the same as textmate?

[maxbrunsfeld]

Do you think we should scope > and < as keyword.operator in template expressions to keep the highlighting the same as textmate?

I think we shouldn't; that seems like a bug in the TextMate parser. I'd prefer to either leave them as unscoped, or scope them in a way that's similar to the (, ), [, and ] characters.

[thomasjo]

I strongly agree with @maxbrunsfeld. That’s a bug in the TextMate grammar.

[Ben3eeE]

Seems like we don't have any punctuation scopes in the tree-sitter grammar at all. I will sync it up with textmate and correct this/other bugs 👍

Thanks so much for taking a look at this PR @thomasjo it made us notice the bug in atom core and fix up the scopes more 💯

[Ben3eeE]

We have a bunch of opened issues from @chbk regarding punctuation scopes but all are with the textmate grammar. I think it will make sense to fix these for the tree-sitter grammar.

[winstliu]

Another agreement with @maxbrunsfeld. We shouldn't let old bugs with the TextMate grammars affect Tree-sitter.

[Ben3eeE]

@maxbrunsfeld I took a look at the open issues we have here to see if I could improve further. Some of these parse incorrectly with tree-sitter. I didn't check if any of these are valid reports. Just blindly assuming that they are.

From a slack chat with @thomasjo:

• and, or and not is not parsed. They are text variants of the "normal" logical operators
• ->* seems to be parsed by tree-sitter-cpp but it's not working? I am seeing it parsed as ->, *, ERROR
• Operator function scope incorrectly #240

From open issues:

• #error directive doesn't handle ' or " correctly #233
• C++17 nested namespaces don't have the correct colorization #260
• Wrong code colors with specific template and macro code #267
• Trigraphs not supported #273
• C++: Single open brace in #define #280
• hexadecimal floating point numbers #153
• alignof(i) not colored as a keyword #246
  We could solve alignof(i) not colored as a keyword #246 with a regular expression. We could parse this in tree-sitter like sizeof. Searching around for this I found https://en.cppreference.com/w/cpp/language/alignof which has the example code:

struct alignas(64) Empty64 {};

alignas(64) is parsed as a type_identifier

Issues we can solve that are also an issue in tree-sitter:
#177 e1362f4

[maxbrunsfeld]

I think let's get this into today's hotfix as-is. We can circle back and fix the parsing issues that you mentioned in another pass.