Implement markdown extra features

[dmmalam]

Thanks for the excellent md parser!

There are a bunch of various extensions to markdown
eg
http://freewisdom.org/projects/python-markdown/Extra
http://michelf.com/projects/php-markdown/extra/
http://maruku.rubyforge.org/proposal.html

While not all are exactly critical, things like super/subscript, tables, definition lists, abbreviations can go a long way to make writing md documents easier.

Also simple transformations (like texttile http://rpc.textpattern.com/help/?item=intro) can make the text much more readable
For example
... -> …
' ' -> ‘ ’
" " -> “ ”
-- -> –
(c) -> ©
(r) -> ®
and so on .

Ideally the goal should be to never write pure html for any typographical feature.

[chjj]

I was considering adding smartypants transformations as an option, since a lot of markdown parsers seem to have it. I was also toying around with adding footnote syntax, something Gruber himself considered.

The reason I added GFM features by default is because I think they're really smart and well thought-out (everything except for the line breaks).

I really want to keep marked small. I would really like to keep any extra features in a kind of addon that wouldn't be part of marked by default.

[chrisnicola]

Adding my vote for this. At least as an option.

[gjtorikian]

I most definitely respect @chjj's choice here to keep marked small. It's much better than the urge to say "yes I'll do this."...and then never do it.

I, too, however, would like to see more Markdown extension features, particularly those of Maruku. To that extent I've created a new repo to port these into Node.js. It's empty now but I'll start porting to "standard" Markdown features soon.

[gjtorikian]

Actually, I might just keep an eye on Robotskirt, which has ported sundown features instead.

[chrisnicola]

Unfortunately I'm looking for something that will actually work on the client so Robotskirt isn't of much use to me.

[chjj]

There is a reason I'm leaving this issue open. I'm not saying it will never happen. I'm saying when it does, it won't be part of marked by default, but it will definitely be maintained in the same repo. It will be part of marked, just an optional part.

This ties into the goal of making marked more hookable from the outside. This has been on the todo list for too long. It's really hard to design an elegant and sensible interface for that. It's really deceptively simple. I have a couple local branches now, containing different approaches to making marked extensible.

[gjtorikian]

@chjj I've started my own fork to hook into the core Marked lib. I think having a plugin system is neat, but just being able to extend it at all is a great asset.

I'd be curious to know what you had in mind. Mine is obviously vary naïve.

[chjj]

It's really tough because the precedence of each rule matters, and there's no good way to maintain order of properties in an object for example. So here's what would probably have to happen: an exposed array containing each rule name and it's corresponding regex. The loop in the lexer iterates over the array trying to match each rule until it finds one, eats the matching part of the string, and passes control to the rule's function. Once the function has returned, continue to the next iteration of the main loop.

Anyway, from the outside it might look like:

var marked = require('marked');

marked.rules.push(['dd', /^hello world/]);
marked.lexer.handlers['dd'] = function(cap, tokens) {
  // ... do things here
  tokens.push({
    type: 'my_token'
  });
  // could maybe potentially return `text` here if it needs to be changed
};

marked.parser.handlers['dd'] = function(token, next) {
  // parse tokens
  return str;
};

There are other ways, but there's a couple things that are annoying: the string part - no pointers in JS, a function can only return one value, etc. So we could to refactor marked to make it oo, then people can just stick functions on the prototype and easily change the input text: this.text = this.text.substring(...). This gets kind of awkward when we have to recurse during blockquotes and lists.

All of this is kind of ugly in my opinion, and it remains to be seen whether it will have a negative effect on performance. Before v8 prototypes were optimized even more, marked was 6-7 times faster than markdown-js. Part of the reason marked is so fast is that it doesn't have an object that needs to be instantiated, the lexers barely call any functions (everything is just in one big loop), there's no closures, no nested functions, etc.

I've tried implementing this a couple times now. I may have tried every way there is, so the only thing left to do is decide which is the best.

[NuckChorris]

I'd like to suggest the addition of tables at the very least, because they seem like the most well-thought out extension besides that GFM adds. Not sure on subscripts/superscripts, I'm definitely on the fence over that. The smartypants stuff would definitely be nice to have as an option like GFM, though.

[TooTallNate]

+1 for tables support

[hawkrives]

Personally, if someone wanted to add the MultiMarkdown feature set as a plugin, well... that'd be great. Tables, footnotes; you could leave out the LaTeX processing, though.

[bootstraponline]

It'd be great if marked supported GitHub Flavored Markdown.

[instanceofme]

@bootstraponline it does, just set gfmoption to true. There are some issues though, notably with GFM line breaks.

[bootstraponline]

Marked does not fully support gfm.

• No support for tables GFM Tables #50
• Syntax highlighting using ```language is broken GFM code with syntax highlighting #58

There are a bunch of other features not supported as well. Lack of tables is the main missing feature.

Here are gfm test files which demonstrate the lack of support for GitHub Flavored Markdown in marked.

• https://github.com/suan/github-flavored-markdown-test/blob/master/README.md
• http://github.github.com/github-flavored-markdown/sample_content.html

[jonahkagan]

Another vote for more features. For me, specifically footnotes. Any progress on deciding how to move forward with this?

[mbacou]

+1 for tables
+1 for ToC

[tlvince]

+1 for SmartyPants/Pandoc-like smart punctuation

[gavinmcfarland]

+1

What would one need to do to add support for their own syntax rules?

I saw this post regarding a Drupal Markdown filter and was curious.

[romainPellerin]

Did marked now supports TOC like GMF toc?
If not, is there a way to do that kind of things with current marked grammar ?

[hplphotos]

I've recently discovered Adam Pritchard's outstanding Markdown Here. I understand that MDH uses the Marked library. MDH lets me write email in Markdown inside the gmail compose window, which saves the copy/paste step compared to writing the email in an external Markdown editor.

My emails (and other text) tends to use a lot of footnotes. It would be great if the Marked library supported footnotes in a way similar to MultiMarkdown.

[Mithgol]

Note: footnote syntax in MultiMarkdown is identical to that of PHP Markdown Extra.

[prashaantt]

+1 for PHP-Markdown-style footnotes.¹ This is literally the last feature that would make marked perfect for my present use case. I have otherwise been able to achieve almost a complete feature parity between redcarpet (server-side) and marked, and would prefer to use marked all out.

Footnotes

1. Please consider supporting it. ↩

[Ognian]

+1 and is there any progress on this?

[aruizca]

+1 It would be awesome if marked supported markdown-extra

[cabrinoob]

+1 for this feature. You can also take a look at https://stackedit.io/ which is an online editor of md files which supports extended Markdown. It could be a very nice feature to add to this lib.

[aruizca]

@cabrinoob Thanks for the tip. Awesome Markdown editor!!! I wonder what parser uses.

[puzrin]

We implemented those extensions in markdown-it package.

• subscript/supersript
• definition lists
• footnotes
• abbreviations
• advanced typography replacements

Also due pluggable architecture, it's not a problem to write extensions for diagrams, todos, toc and so on.

[SenshiSentou]

Just wanted to ask if custom extensions have since been implemented. I would specifically like to add sub- and superscript support to my project.

[joshbruce]

@SenshiSentou: I cannot answer this directly as I'm fairly new around here. See #956

Having said that, I don't know enough about markdown-it to determine if it has the same vision; or, if marked has (or could have) a fundamentally different approach to achieving that mission.

@UziTech: Thoughts??

[UziTech]

At this point I feel like anyone who wants these extensions should move to markdown-it and this package should be only about GFM and CommonMark and bug fixes.

If this is maintained further it would probably be best to create an extension system instead of adding all of these features to marked directly.

[UziTech]

Marked does provide access to the lexer and parser for anyone who wants to create a wrapper with any of these options.

[joshbruce]

@UziTech: Just to make sure I'm picking up what you're putting down:

GFM = GitHub Flavored Markdown
CommonMark = http://commonmark.org an attempt at standardizing the syntax, but guarding extension based on demand...some components of MultiMarkdown, for example.

Is that about right?

If so, I concur with this assessment. Maybe markdown-it would actually use Marked as its parser/compiler...offering all the bells and whistles.

[UziTech]

Ya basically at this time we should focus on getting the number of issues down and making sure marked works well for all the packages that already depend on it and put off adding any new functionality.

[joshbruce]

Closing see previous conversation.

[Crystal-RainSlide]

Now this may be implemented as a extension (template) of Marked.

There is xiaoliwang/markedjs-extra for some reference, but it's pretty old.