Improve compatibility with default/other syntax themes

[burodepeper]

• Switch default scope to source.text.md
• Add a markup-and-down.less resource file to enable quick and easy patching of syntax-themes
  • Set up framework for generic markup styles
  • Expand framework with custom markdown styles
  • Test resource file via minimal-syntax, and add scopes wherever necessary
• Test default syntax themes with and without patch
• BUG: markdown-preview doesn't work with the changed base scope; perhaps this can better be solved in that package. (see Idea: improve/change grammar matching atom/markdown-preview#384)
• Create patches for default syntax themes
  • Atom Dark
  • Atom Light
  • base16 Tomorrow Dark
  • base16 Tomorrow Light
  • One Dark
  • One Light
  • Solarized Dark
  • Solarized Light
• Create patches for popular syntax themes
  • Monokai (Move markup styles out of the gfm scope kevinsawicki/monokai#68)

[plttn]

Since this actually handles reference links nicely, I'm working on patching my theme/port of Tubster, but the .less you've provided to aid in patching doesn't seem to line up with what the current scopes are for things.

I find it helps to actually include the new markdown.less file in base.

[plttn]

And I've got my theme up and running with language-markdown, but thanks to an renaming issue, it won't be available as an update until it gets fixed.

[burodepeper]

Hi @plttn,

The .less that's provided in this branch will indeed only be relevant from v1.0.0, but that might take a while. The biggest issue is that the default themes provided by Atom are all different in structure, and have their markup classes often scoped to gfm for instance. I intend to create PRs for each of these, but they can't break anything, while I want to support more generic styling in them at the same time. If these PRs go through, I'll have to wait until they are included in Atom before I can safely publish v1.0.0 because there's a big chance it will break syntax highlighting. I might finish and release v1.0.0 earlier, but it's a tedious job to check support with syntax-themes. I'm in a bit of a conundrum.

Cool that you're updating tubster to work with language-markdown though. When I have had enough caffeine (just woken up), I'll have a look at your code to see if there's a quick fix possible (which I presume there is). I'll respond in your issue which I hope you'll rename to because language-markdown isn't the bit that's broken... ; )

[nylki]

Hi @burodepeper and @plttn,
Just wanted to quickly drop by to tell you I'll soon have some time to see what changes are necessary
for my theme pen-paper-coffee to best make use of language-markdown.
keep up the good work! :)

[plttn]

I can now officially say my theme pubster-syntax just hit v1, with the headlining feature of v1 being language-markdown support.

Thanks to @burodepeper for making a IMO much better markdown grammar than the core one.

[burodepeper]

Hi @MaximSokolov (and others),

I found a PR of yours a while ago (you recently linked me to this but I remember a much longer discussion) about standardizing scope-names. I found it an interesting topic/issue, wanted to get involved with it initially, but got side-tracked. While working on language-markdown, and this PR in particular, I've come across the problem once again. With help from @simurai (hi there!), I believe that most markup-related scope-names have been moved out off the .gfm scope in the default syntax-themes. I recently noticed that most of them (and probably/logically for every syntax-theme based on them as well) lack support for markup related things at all, or merely provide very limited support. I was wondering (hoping actually) if you (and @simurai and hopefully others) are interested in collaborating in improving this in general, perhaps as a stepping stone for introducing a wider-spread standard for scope-names in syntax-themes in general.

My efforts on this front have been a little stale the last couple of weeks. I got demotivated by the potential scale of trying to really fix the issue at hand, but I believe that a few like-minded people could actually get some stuff realized!

If any of you are interested, I'd like to suggest coming up with a roadmap to tackle the issue first only for markup-related scope names before escalating it to an Atom-wide issue/solution. My efforts so far have resulted in this less-template (section 2 is most relevant). From that I suggest the following plan:

1. Discuss and draft a specification/documentation for markup-related scope-names. This would result in something similar to the template I linked to above.
2. Apply this template to all (eight?) default syntax-themes, without breaking any existing styles (if possible). If possible, use the same structure for all themes.
3. Additionally create a default 'empty' syntax-theme template, which can be extended in the future with all other scope-name conventions etc.

Parallel to this, I will be adapting language-markdown to whatever we're doing, because I try to implement a fallback mechanism when certain markup-elements have a semantical overlap with syntax-elements (a horizontal-rule for instance can be considered a constant of sorts). I have some additional ideas on how scope-names and syntax-theming in general tie together, but I've typed enough for now ; )

I hope I've sparked some interest, and if similar efforts are happening elsewhere, links please!

[simurai]

@MaximSokolov suggested to add an official syntax naming convention to Atom: atom/atom#8430 and the WIP: atom/flight-manual.atom.io#114. I think it's a great idea, but since there are already hundreds of syntax themes and language packages out there, not easy to change too much. So that PR is more about thinking how it could work for Atom 2.0.

@burodepeper Something I was wondering. How much different is this PR from the language-gfm package? Would it work if themes just add the .md, selector at the top?

+ .md,
.gfm {
  // styles
}

[burodepeper]

@simurai I had already noticed that both those issues you've linked haven't been updated for months. Mentioning you and Maxim here is an (possibly futile/unneccessary) attempt to jump-start it again.

In general, I think the this PR is not about the difference between this package and language-gfm, but rather that the markup styles should be available for any "language" that adds a form of markup. language-hyperlink adds markup (an underline class) for instance, I'm sure there are others. For example, I thought of creating an atomdoc-language kind of thing, that parses comments (which are otherwise "untreated") in any/all language(s). This would use generic markup styles. The .md and .gfm scopes should be used to distinguish, not to constrain.

Perhaps most importantly, I'm trying to figure this out without breaking any backwards compatibility. Removing the .gfm class for instance, didn't break anything. I understand the "Atom 2.0" argument, but at this moment I am very limited in creating any markup related things that I want supported by syntax-themes, unless I use horribly wrong scopes. That is what I meant by the fallback that I'm adding as well. Most elements in the dev-scopes branch have double scopes: one that is purely based on markup, and one that is based on the default scopes that I hope exist in most syntax-themes.

I don't mind doing some heavy lifting, and seeing as there are a little over a thousand syntax-themes currently, I'd even say that it is realistic to suggest changes or perhaps even create PRs for the most-used themes. I'd even dare to say that it is realistic to do this not just limited to markup-related scopes. It needs some coordination, but it's do-able, right? We're talking about creating a template, and a way to distribute it. Or am I completely overlooking something? Again, I don't mind doing some heavy lifting, but it would be nice to have some support beforehand, a few extra braincells to think about things that ought to stand the test of time (with @MaximSokolov's work as a good start), and eventually some help of the Atom team when it comes to communication/distribution.

[simurai]

Sorry, yeah right.. .md and .gfm are more specific language packages. .markup could be used much more broadly for many languages/packages. 👍

the fallback that I'm adding as well.

With "fallback", do you mean this? It's pretty close to the template currently used for the syntax theme generator and I think also TextMate converter. So the basics should be ok in most themes.

But I just looked at a random (newest) syntax theme and it's scoped to .gfm https://github.com/qolop/shamrock-syntax/blob/master/index.less#L52-L89. Probably a fork of another existing theme. So making a PR like this:

- .gfm {
  .markup {
    // styles
  }
- }

wouldn't be too hard and the theme author will most likely happily accept it. But if we define a completely new template and try to convince theme authors to use it, I can imagine it's a much harder sell. Also, some themes are probably abandoned and the theme author wouldn't care about getting PRs.

Not saying we couldn't improve the current convention already now, and it would be opt-in for those themes that want to use it... just that trying to enforce it, would really need some heavy lifting and hope theme authors play along.

[simurai]

Sorry if off topic, but somehow I still think that moving the (some of the) styling into language packages might be a good idea. So..

1. Themes would define color variables, like https://github.com/burodepeper/language-markdown/blob/dev-scopes/resources/markup-and-down.less#L1-L17
2. Themes style only the basics. Maybe just each .variable { color: @variable }. but not more.
3. Languages packages use the color variables to add more syntax highlighting that is specific to the language, like: https://github.com/burodepeper/language-markdown/blob/dev-scopes/resources/markup-and-down.less#L21-L223
4. If a theme really wants to, it can override the language package.

Benefits:

• Language packages can make changes any time and don't have to wait for another major version with a new naming convention.
• Language packages can highlight just a little or as much as they want. There might be language-basic and language-advanced. So people can choose what they prefer, instead of typically only have a single language package that follows a predefined convention.
• Language package authors probably know better what should be syntax highlighted than theme authors. Theme authors don't know every language, but a language package author knows a language very well.

Actually, the https://atom.io/packages/angularjs already tried to do it here, but there aren't any fitting variables to use, see this issue.

The hard part will be to define the variables that make sense for most language packages.

[burodepeper]

The fallback is in the actual parsing/scopes assigned. This for example is created in side a .markup.link (somewhere else) but is also designated as a string as backup. It's not the best example, but it will give you an idea.

---

As an approach of moving styles out of the .gfm scope, I would suggest submitting issues to syntax-themes at first. The diff as you suggest it is a small change, and if it comes with a link/reference to a clear explanation as what needs to happen and why (possibly copy-pasted into the issue report), I'm sure most theme authors will be happy to oblige.

I agree that suggesting/enforcing a change of structure isn't the way to go, but I think it would be good for the default themes to implement it, so everything that is forked off of them can take advantage of it. The template you reference is a good start, but I have a feeling that a lot of new themes aren't forked off of that one. If we think of this as a disease we want to spread, we need multiple patient zeroes ; )

Seeing as you are more involved with Atom, do you think it's possible to compile a list of most used syntax-themes by actual usage? It would be good to know the scale of the problem, and how long its tail is (as in "The Long Tail").

---

Yup, my approach with markup-and-down.less is indeed an intermediary form of what you suggest. But as much as I'd like to discuss that, I think a different place would better suit the discussion.

[burodepeper]

The hard part will be to define the variables that make sense for most language packages.

My first instincts were that this would indeed be the hard part, but now I think of it again, I think it is actually quite simple. It seems hard, because there's is no apparent overlap between for example html, javascript and css. But if you group languages by type, things become more apparent. Think of the following groups for example:

• actual programming languages (c, javascript, ruby, python)
• markup related languages (html, markdown)
• styling related languages (css)
• data-containers (json, yaml)

The concept of a class is not generic of its own accord, but it is generic (to a significant extent) within its group. I think technically, and I'm thinking waaay into the future here, this would mean subdividing the base-class source.

Let's get back on-topic ; )

[burodepeper]

Speaking of test-cases: I created this PR (kevinsawicki/monokai#68) for the Monokai theme by @kevinsawicki, but it hasn't gotten any attention yet so far. I'm sure Kevin has other things on his mind, and I don't blame him in the slightest (although this is intended as a slight nudge ;), but this could be a good indicator of what we would be working 'against'. On the other side of the spectrum, I would imagine that authors of lesser used themes would be extra motivated by outside interest.

[burodepeper]

I'm closing/abandoning this attempt. See #52 for some additional details.