Update to Mistune v3

[marmitar]

Mistune v3 is already in Pre-Release on PyPi, with a few incompatibilities. Not a lot of changes on our side, besides moving regexes to new variables and updating methods to the new API.

I haven't tested on older Python versions (< 3.10) yet, just waiting for a proper release.

Highlights in Mistune Code

• Dropped CI tests for Python 2.7 and 3.6 (see commit)
• New base Parser that doesn't use the hidden re.Scanner (see commits), we can even used named groups now!
• Benchmarking code to avoid perfomance regressions (see benchmark files)

[kloczek]

Any plans to backport that PR for 6.5.x?

[marmitar]

Any plans to backport that PR for 6.5.x?

You should try discussing that in #1809. Anyway, Mistune 2.0 took a very long release cycle:

• Version 2.0.0 Alpha 1: Dec 07, 2019
• Version 2.0.0 RC1: Feb 18, 2021
• Version 2.0.0: Dec 05, 2021

So maybe when version 3.0 lands it won't make sense to use nbconvert 6.5 anymore.

[blink1073]

Any plans to backport that PR for 6.5.x?

No, while we (mainly @TiagodePAlves) did our best to preserve backward compatibility, there are just too many differences between the versions, which is what is prompting us to move to 7.0 in the first place.

[kloczek]

Just tested this PR and looks like it is not possible to apply it on top of 6.5.3

+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-nbconvert-Update-to-Mistune-v3.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
2 out of 4 hunks FAILED -- saving rejects to file nbconvert/filters/markdown_mistune.py.rej
1 out of 1 hunk FAILED -- saving rejects to file pyproject.toml.rej
6 out of 6 hunks FAILED -- saving rejects to file nbconvert/filters/markdown_mistune.py.rej
1 out of 1 hunk FAILED -- saving rejects to file nbconvert/filters/markdown_mistune.py.rej
2 out of 3 hunks FAILED -- saving rejects to file nbconvert/filters/markdown_mistune.py.rej

[marmitar]

You'll probably need #1764 and #1774 too.

[lepture]

Also, there is a built-in math plugin in mistune v3. Can you review it? If there should be something required to change, please let me know.

[jarrodmillman]

https://pypi.org/project/mistune/3.0.0/ is out now!

[marmitar]

Just updated to Mistune v3. There are a lot of changes for markdown_mistune.py now, but they are mostly related to the type hints. The main changes in functionality are in the MathBlockParser and MathInlineParser classes to align with v3.

Now, the check_links job is not liking the changes. It can't import name 'BlockState' from 'mistune', so I suspect it may be using an older version of Mistune. Don't know what to do here.

[blink1073]

Check links is failing because pytest-check-links is installing the released version of nbconvert. How much work would it be to support both mistune 2 and 3?

[marmitar]

Oh, makes sense. I think it should be okay to support both, I just hadn't considered it.

[marmitar]

Ok, I had to keep two versios of MathBlockParser and MathInlineParser, one for v2 and one for v3, but the rest had minimal impact. I ran the full test suite in Python 3.7 and Python 3.11 for Mistune 2.0.5 and 3.0.1 and everything passed successfully.

[blink1073]

Nice work, thank you! I'll make a new minor release early next week.