Is Parser.js doing too much?

[calculuschild]

At a certain point, splitting the functionality of Marked.js across multiple documents becomes harder and harder to track and maintain. Especially when the flow of source -> token -> html gets spread across several sections of code. Just looking over parser.js, it looks like mainly it's just a large switch statement that concatenates the output HTML string by calling renderers in the right order. That's fine.

Only, there's also large amounts of processing in parser.js for Table and List tokens which seems out of place. Would it make sense to move that processing into some larger renderers? Instead of the Table renderer just wrapping the Header and Body from parser.js in <table> tags? I suspect the speed would remain similar, while also opening up more of that background processing on the Renderer.js side for those who want to extend Tables, for instance.

It seems like the core functions should be delineated as follows:

• Rules.js -> Regex to detect tokens in the source
• Tokenizer.js -> Test rules and process matches into Token objects (including sub-tokens)
• Lexer.js -> Determine order of Tokenizer execution, and collect Tokens into an ordered list
• Parser.js -> Determine top-level order of Renderer execution, and concatenate HTML into a single output string
• Renderer.js -> Process Token objects into HTML (including its own sub-tokens)

[UziTech]

Are you proposing that we remove the tablerow and tablecell renderers and just do it all in the table renderer?

[calculuschild]

Well, supposedly the user doesn't need to change any part of marked, but that's the point of supporting extensions. So they can change things.

[UziTech]

What is there to customize in the table parser? It essentially just loops through the child tokens and parses each of them.

[calculuschild]

So, this might not be the best example, but there are some Markdown variants that support column span/row span by changing up the syntax. Getting this to work with Marked.js might require looking further back in the Tokenizer or use walkTokens or something, but I could see and have seen related syntax where the contents of a cell (maybe the header has some symbol) will determine how the rest of the table is laid out. Having access to the sub-tokens of a Table token (the contents of a TableCell maybe) would be required to determine the presence of specified parameters (column width, etc.) in the resulting Table. What I'm trying to say is I could see cases where full knowledge of the Table token tree would be helpful in determining the render process.

[calculuschild]

But mostly, use cases aside, its just that having half of the logic separated out into a different file but for only a couple of cases feels weird to me. I am of course the newcomer here, so you guys have the final say on anything. Just pointing out some weirdness that maybe I'm not understanding fully.

[UziTech]

It may feel weird but there doesn't seem to be a better way to do it without requiring the user to handle the parsing logic in a render function.