Fix objective-C and objective-C++ to not inherit

[jeff-hykin]

Other languages (Objective-C, Objective-C++) include the C and C++ syntax

What is the issue?

These other languages assume the C and C++ syntax use this thing $base that says "re-include every single pattern, and (if inside another language) only re-include all of that other languages patterns". Objective-C++ makes the assumption that basically every C++ namespace/struct/function/class/if/else etc uses $base inside of it.

// Objective-C++ code
int main() {
   // (this is the inside of a C++ pattern)
   // all Objective C++ patterns are allowed in here
}

Why is this a bad assumption?

1. C and C++ (or any other language that uses $base) cannot work inside of markdown. One example of this is Markdown syntax colour issue while using /* .. */ atom/language-c#146 Here's why:

This is a markdown file 
Let me give an example of C++ code
```cpp
class A {
   // because of $base, all the markup patterns are allowed here
   // (and C++ patterns are not allowed)
**bold** // marked as bold
int a = 10; // not highlighted at all
}
```

This is unfixable because of the Objective-C++ and Objective-C dependency.

2. Importing C++ is a bad assumption for Objective-C++ in particular because there are C++ constructs that are not allowed in Objective-C++ (Objective-C++ is not a superset of C++). Including the entire C++ language as the highest-priority group is requiring that the C++ syntax be incomplete for the Objective-C++ syntax to be correct.

3. $base is harmful even within C++

int main() {
  // all C++ patterns are allowed here
  // (not all C++ patterns should be allowed here)
  int main() {} // not a valid C++ function definition, but it gets highlighted because of `$base`
}

Because of the Objective C++ dependency, we are knowingly having to include incorrect patterns within C++, which is stopping us from adding support for major features like highlighting of variables and types.

What is the solution?

It is easy to make the Objective-C++ syntax not depend on C++, by importing the valid pieces of C++ syntax into Objective-C++. This would mean C++ improvements would never affect Objective-C++, and it means we can finally start using C++ inside of markup.

[jeff-hykin]

Once this is done we can clean up/remove the cpp_base pattern.

[matter123]

For reference:

• the current markdown grammar is: https://github.com/Microsoft/vscode-markdown-tm-grammar
• the current Objective-C grammar is https://github.com/atom/language-objective-c

[matter123]

The objective-c grammar includes source.c 3 times

• source.c#string_escaped_char
• source.c#string_placeholder
• and second from last the entire source.c grammar

The objective-C++ grammar includes source.cpp then source.objc https://github.com/atom/language-objective-c/blob/master/grammars/objective-c%2B%2B.cson

[jeff-hykin]

Wow I think that means the C language was being double included on Objective-C++ (once inside C++ and again inside Objective-C)

[matter123]

having both source.c and source.c.extend where source.c uses $self and source.c.extend uses $base, would allow for both uses right?

This would require modifying the objective-c includes.

[jeff-hykin]

I was thinking about doing something like that, but I think it doesn't really fix the case when the language spec diverges, or when there are generally complicated ways of matching things in Textmate.

For example Objective C could break the assumptions we made about [[attribute]], and our changes could break their syntax.

This is from Quora:

[jeff-hykin]

Although I guess it is true that Objective-C is a strict superset of C (unlike C++)
https://stackoverflow.com/questions/19366134/what-does-objective-c-is-a-superset-of-c-more-strictly-than-c-mean-exactly

I still think there is times where the TextMate syntax of one will screw up the TextMate syntax of the other because of priority matching.

[matter123]

yeah, it's a strict superset so source.objc extending source.c.extend. There is surprisingly relatively little information on Objective-C++ I am not sure if is also a strict superset, but if it is I imagine that dcl.attr.grammar/7 still applies

Two consecutive left square bracket tokens shall appear only when introducing an attribute-specifier or within the balanced-token-seq of an attribute-argument-clause.

[matter123]

well, https://github.com/GrinCash/Grinc-core/blob/1377979453ba84082f70f9c128be38e57b65a909/depends/work/build/i686-w64-mingw32/qt/5.9.7-f2560c1efa6/qtbase/src/testlib/qxctestlogger.mm#L135 shows that Objective-C++ is not a strict superset of C++.

[matter123]

So assuming that Objective-C++ continues to extend c++ [[attributes]] at least need to have their scope limited, alternatively the extended syntax could exclude attributes

[jeff-hykin]

I had enough time today to redo a bit of the fundamentals of the Grammar object. I changed the way it saves patterns/ranges so that they can be shared more easily now.

I also changed the $base. You can now use :$base and :$self, but I also added :$initial_context. When saving a grammar to a file now, you can give it a inherit_or_embedded: argument. If you give it :inherit (which is the default) it will convert all the :$initial_context's into $base, if you give it :embedded it will convert all of them to $self.

I left in the $base and $self encase there is some cases that we don't want it to convert.

Sadly didn't have enough time to look into the Objective C repo, but this should get us a step closer.

[matter123]

Objective C and C++ don't inherit now, right? So it should be safe to set the default for :$initial_context to be $self?

[jeff-hykin]

Sorta, I'm going to see if VS Code will pull Objective-C and Objective-C++.

If they only pull C++ and C then we're going to have a problem. I'm going to talk to Alex tomorrow