🏇 PRF: prefer (and encourage) lxml

[bollwyvl]

• for Slow page builds in 6.0 when generating navigation bar #381

[bollwyvl]

Aside from various screwups on my end, it looks like bs4-on-lxml returns different dom...

[drammock]

might need this too?

Suggested change

	except ImportError:
	except (ImportError, ModuleNotFoundError):

[bollwyvl]

nawp, think we're good:

ModuleNotFoundError.__mro__
(ModuleNotFoundError, ImportError, Exception, BaseException, object)

[bollwyvl]

So: re: the dom changes:

• lxml will parse whatever, but always gives back a full HTML document
• if the lxml is not used, an invalid <input type="checkbox"><tags><that><aren't><allowed></input> gets created, as can be seen in the fixtures

So actually: I think the right play for this PR is to just always use lxml... and indeed, stop using bs4 altogether, so that xpath, etc. becomes usable. It's 2021, it's not that big a deal to install anymore, even on windows, and if two major downstreams are feeling performance issues, it doesn't seem unreasonable to have a performance-first approach. Of course we should look more places, but it's pretty hard without benchmarks, which are... tricky to add, after the fact.

[jorisvandenbossche]

and indeed, stop using bs4 altogether

You mean not just using the lxml parser through bs4, but actually not using bs4? Note that right now we quite heavily depend on manipulating the HTML using bs4 functionality (although it has decreased a bit after the changes in #346)

[bollwyvl]

heavily depend on manipulating the HTML using bs4 functionality

yeah, lxml can edit stuff, too. one thing i have noted: html actually is quite whitespace aware, so the lxml pretty print is very conservative, and makes testing hard... if proceeding further, would probably keep bs4 for validation purposes.

[bollwyvl]

Anyhow, I'll hold on this, as it would almost certainly need more investigation, and no doubt trigger a 0.7.0...