I've accidentally pasted large block of text to my notes and I cannot launch Joplin now

[piotrek-k]

Environment

Joplin version: 2.4.9
Platform: Linux
OS specifics: Ubuntu 21.04

Steps to reproduce

1. Create note
2. Paste large block of text to "code block"

I'm using markdown editor and pasted large block of text (JSON containing encoded image) between two sequences of ``` (code block). Joplin immadiately became unresponsive, finally I've killed it. I cannot open Joplin now, window appears, but I can only see upper menu and white screen (see screenshot). Also, I'm experiencing high CPU usage. After a few minutes Jopin crashes.

Any tips how to recover from this state?

[laurent22]

If you go to Help => Toggle Safe Mode does it work?

Also please could you check if there's any error in the console? https://joplinapp.org/debugging

[piotrek-k]

Hi, thanks for quick response. It turns out I needed to wait for a few minutes before anything happened after clicking "Toggle safe mode". After several attempts, it was finally able to enable it. It indeed helped and I've managed to remove note which recovered Joplin to usable state. Thank you again.

Nevertheless, some feature need to be implemented which will prevent such things in the future. Maybe prompt with warning that user is pasting big block of text?

[laurent22]

Could you provide jex of the note?

[piotrek-k]

I've removed it, but I can send you text that I've accidentally pasted:
joplin_pasted_text.txt

[laurent22]

Thanks, so you've just put three backticks, then pasted this text, and it happened?

[piotrek-k]

Yes, exactly, I've just did it again for the sake of science and it broke again. What might be important is that I'm using markdown editor with split screen (raw markdown on the left, preview on the right)

EDIT: also, I can't see any errors. It just freezes

[CalebJohn]

I spent some time tonight digging into the performance issues present here. And unfortunately it seems like the performance problems are present in both the viewer and the editor.

Editor

Line wrapping is really, really slow for long lines (like in the attached)

• I think a possible fix for this will be to have an option to enable/disable line wrap. We can have safe mode enforce line wrap = disabled. For users on low end systems they can just disable it to get extra performance (also some users prefer not to have it).

The syntax highlighter uses regexes in 2 places that both individually contribute to slowing this down.
regex to find katex markers
regex to detect tables

• I'm hesitant to spend much time optimizing these, the current version of CodeMirror uses a regex based highlighting engine/ Upgrading to CodeMirror 6 will require this to be re-written in a more performant manner anyways.
• We might be able to disable highlighting for long lines, which would effectively fix this.
• We can also disable all syntax highlighting in safe mode, which would allow users to recover a frozen instance.

Viewer

The code block highlighting appears to be the primary culprit here.
There performance problems come from 2 places
regex to trim trailing \n

• This can certainly be handled in more efficiently

highlight.js.

• We might want to consider limiting what gets passed to highlight.js based on input length. I couldn't find any concrete performance metrics from highlight.js, but it doesn't seem unreasonable to disable it for very large notes, Joplin is not a code viewer after all.
• We should at least consider disabling it in safe mode for the same reasons as above.

[laurent22]

Thanks for looking into it Caleb. For previous performance issues or freezes, we simply upgraded highlight.js and it usually worked but in this case it sounds like it's a complex issue with many causes.

We could indeed disable highlighting for large documents (after all, even VSCode does this). Would you say we should check the document size or the line length? (or both?)

As for safe mode, normally all Markdown rendering and plugins are disabled (it just displays a plain text area), unless there have been some regression recently. If it's not doing this anymore it should be fixed.

[CalebJohn]

Would you say we should check the document size or the line length? (or both?)

I think document size for highlight.js is sufficient. Well actually just the code block size is easiest. But for the markdown it rendering itself, I think document size is fine. As far as I can tell, markdown-it handles that 330,000 character line just fine.

For the renderer i think we can disable it for document sizes > than a couple MBs (we can see what VSCode does). Highlight.js will need to be disabled for much smaller blocks, probably a few kB. I couldn't find any form numbers about highlight.js performance, but i know it's tracked so I'll look again.

As for the CodeMirror editor, it doesn't look like anything changes with safe mode. I'd like to switch to the PlainEditor (if it still works), otherwise just disabling all the options should be sufficient.
We can also put some line length checks in place to prevent performance issues for very long lines. Document checks won't be necessary for the CodeMirror highlighting because it operates line by line and doesn't consider lines outside the viewport (I need to double check this, it might not always be true).

[CalebJohn]

It turns out that highlight.js doesn't do as much performance testing as I had hoped, but that's fine because I did some unscientific testing of my own. I also noticed the severe performance issue is related to using the highlight.js language detection. On my machine the example note from this thread takes ~0.25s to render if lang=json is provided, otherwise it takes ~30s.

Knowing that, I tested 5 conditions with autodetect language.

1. a 65 KB python source file (this is one of the highlight.js perf tests): ......1.33s
2. The same python file but duplicated to be about 320 KB: ..... 3.35s
3. The JSON file attached to this thread, but the long line is split into ~4500 shorter lines (330 KB): ..... 9.91s
4. The JSON file from this thread with no changes (330 KB): ..... 28.89s
5. Same json with ~300,000 characters removed from the long line (35 KB): ..... 2.85s
6. The same as 5 but split into 450 lines (39KB): ..... 1.13s
7. A plain text file with 322KB worth of the letter "a": ..... I terminated after close to 30 minutes
8. A plain text file with 3KB worth of the lettter "a": ..... 1.14s

My guess for the reason python is so much faster is that the auto-detect algorithm is able to short circuit the python code, but can't in the case of JSON (either because it's too vague, or because of the binary blob).

My takeaway here is that we should probably be considering both line length and document length for highlight.js. However, it takes time to calculate line length, so we can just use document (code block) length as a proxy. We also need to be much stricter on sizes when using the language auto-detection.

I'd say the following makes sense as rough guides
With language supplied: 512 KB (worst case takes < 0.5s on my machine)

Without language: 1 KB (~0.5s worst case on my machine, this is actually the average performance case, this seems to be approximately the overhead of the auto-detection algorithm)

Another option for speeding up the auto-detect is to limit the number of checked languages. I tried the first 4 examples from above with autodetect limited to the topLanguages

I tested the first 4 examples again with just the top languages

1. 0.48s
2. 0.72s
3. 1.54s
4. 4.16s

So it looks like we can get quite a speedup by limiting the languages that can be detected. I don't think that's a particularly good solution though because users are likely to be mad.

---

*disclaimer: All the above run times are from a single test each (although they were all roughly duplicated many times). Some of the run times are quite long and I didn't want to spend my entire evening getting good data. Take all these numbers with a pinch of salt.

[laurent22]

Thanks @CalebJohn, I think your recommendations make sense so I've ended up implementing them in these two commits:

• aa3cbbd
• 0b01b5b

The first one restricts HLJS rendering depending on the size of the text and language, and the other enables safe mode for Code Mirror too by disabling all options.

I guess there might still be cases where the editor or viewer will freeze but the above should at least improve the situation a bit.