fix: speed up parsing long lists

[calculuschild]

Marked version:

v3.0.0 +

Description

• Fixes Performance of parse() is 15 times worse in releases after 2.1.3 #2297. The rework of Lists in v3.0.0 accidentally introduced a O(n^2) slowdown issue for long documents.

The List tokenizer was using a RegEx to capture the potential next list Item, and then split that captured text line-by-line to determine if it had proper indentation, etc. and whether each line should be part of the current list Item.

Problem is, the captured text was literally the entire document, so for every potential list item, we were capturing the entire document and then splitting the document into lines. For longer documents, this meant spending the majority of time just splitting the document into lines over and over.

Here's the offending Regex, and notice that (?:\\n[^\\n]*)* matches everything to the end of the document:

marked/src/Tokenizer.js

Line 193 in d098d55

const itemRegex = new RegExp(`^( {0,3}${bull})((?: [^\\n]*| *)(?:\\n[^\\n]*)*(?:\\n|$))`);

This PR changes that so we only capture the first line (with it's bullet point), and once we verify that it is a candidate for starting a new list Item, we just traverse the SRC one line at a time. No more mass line-splitting when we really only need to look one line at a time anyway.

Offending line-splitting line where 90% of processing time was spent:

marked/src/Tokenizer.js

Line 205 in d098d55

lines = cap[2].split('\n');

Needs a bit of cleanup, but it's passing tests.

Contributor

• Test(s) exist to ensure functionality and minimize regression (if no tests added, list tests covering this PR); or,
• no tests required for this PR.
• If submitting new feature, it has been documented in the appropriate places.

Committer

In most cases, this should be a different person than the contributor.

• CI is green (no forced merge required).
• Squash and Merge PR following conventional commit guidelines.

[vercel]

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/markedjs/markedjs/5cTrJbPLiAZvhvjqo23kEq9L2pBJ
✅ Preview: https://markedjs-git-fork-calculuschild-fixon2scalingforlists-markedjs.vercel.app

[UziTech]

Why was this test removed? Is it failing with this PR?

[UziTech]

nvm I see this was an accidental addition.

[calculuschild]

That was a test I added playing around trying to debug my program and accidentally committed it to the PR.

[UziTech]

I can confirm this speeds up parsing long lists.

This test takes about 2 seconds on my machine on master and about 100 ms with this PR.

[calculuschild]

Hooray!

[calculuschild]

I want to go over this code once more before it gets merged. I have a nagging sense that there's still some debris left behind I should clean up but exactly where is eluding me at the moment.

[calculuschild]

@UziTech Cleaned up the logic how I wanted. Passes the specs but is failing the SNYK security test here and I'm not sure why.

Otherwise, this is now ready to merged.

Edit: Ah, there was a merge conflict somewhere setting it off. All fixed now!

[github-actions]

🎉 This PR is included in version 4.0.6 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀